======================================================= TAC KBP 2016 ENGLISH KB CONSTRUCTION EVALUATION RESULTS ======================================================= Team ID: lilian Organization: Shanghai Lilian Information Technology Ltd. Co. ************************************************************* Run ID: lilian_KB_ENG_1 Did the run access the live Web during the evaluation window: No Did the run extract relations from the Cold Start source corpus: Yes Did the run generate meaningful confidence values: No Entity Discovery Evaluation: ONLY English documents: Prec Recall F1 Metric 0.854 0.558 0.675 strong_mention_match 0.766 0.500 0.605 strong_typed_mention_match 0.000 0.000 0.000 entity_match 0.803 0.347 0.484 b_cubed 0.718 0.469 0.567 mention_ceaf 0.666 0.435 0.526 typed_mention_ceaf Slot Filling Evaluation: Metric RunID Hop Prec Recall F1 SF-ALL-Micro lilian_KB_ENG_1 0 0.1935 0.0225 0.0403 SF-ALL-Micro lilian_KB_ENG_1 1 0.0000 0.0000 0.0000 SF-ALL-Micro lilian_KB_ENG_1 ALL 0.1765 0.0149 0.0275 SF-ALL-Macro lilian_KB_ENG_1 0 0.0423 0.0272 0.0302 SF-ALL-Macro lilian_KB_ENG_1 1 0.0000 0.0000 0.0000 SF-ALL-Macro lilian_KB_ENG_1 ALL 0.0262 0.0168 0.0187 LDC-MAX-ALL-Micro lilian_KB_ENG_1 0 0.1944 0.0229 0.0409 LDC-MAX-ALL-Micro lilian_KB_ENG_1 1 0.0000 0.0000 0.0000 LDC-MAX-ALL-Micro lilian_KB_ENG_1 ALL 0.1772 0.0152 0.0280 LDC-MAX-ALL-Macro lilian_KB_ENG_1 0 0.0425 0.0299 0.0323 LDC-MAX-ALL-Macro lilian_KB_ENG_1 1 0.0000 0.0000 0.0000 LDC-MAX-ALL-Macro lilian_KB_ENG_1 ALL 0.0258 0.0182 0.0196 LDC-MEAN-ALL-Macro lilian_KB_ENG_1 0 0.0350 0.0227 0.0249 LDC-MEAN-ALL-Macro lilian_KB_ENG_1 1 0.0000 0.0000 0.0000 LDC-MEAN-ALL-Macro lilian_KB_ENG_1 ALL 0.0212 0.0138 0.0152 *ALL-Macro Prec, Recall and F1 refer to mean-precision, mean-recall and mean-F1. NIL-DETECTION P/R/F1: 0.2995 0.9350 0.4537 ************************************************************* Run ID: lilian_KB_ENG_2 Did the run access the live Web during the evaluation window: No Did the run extract relations from the Cold Start source corpus: Yes Did the run generate meaningful confidence values: No Entity Discovery Evaluation: ONLY English documents: Prec Recall F1 Metric 0.854 0.558 0.675 strong_mention_match 0.766 0.500 0.605 strong_typed_mention_match 0.000 0.000 0.000 entity_match 0.803 0.347 0.484 b_cubed 0.718 0.469 0.567 mention_ceaf 0.666 0.435 0.526 typed_mention_ceaf Slot Filling Evaluation: Metric RunID Hop Prec Recall F1 SF-ALL-Micro lilian_KB_ENG_2 0 0.2647 0.0225 0.0414 SF-ALL-Micro lilian_KB_ENG_2 1 0.0000 0.0000 0.0000 SF-ALL-Micro lilian_KB_ENG_2 ALL 0.2535 0.0149 0.0281 SF-ALL-Macro lilian_KB_ENG_2 0 0.0423 0.0272 0.0302 SF-ALL-Macro lilian_KB_ENG_2 1 0.0000 0.0000 0.0000 SF-ALL-Macro lilian_KB_ENG_2 ALL 0.0262 0.0168 0.0187 LDC-MAX-ALL-Micro lilian_KB_ENG_2 0 0.2745 0.0229 0.0422 LDC-MAX-ALL-Micro lilian_KB_ENG_2 1 0.0000 0.0000 0.0000 LDC-MAX-ALL-Micro lilian_KB_ENG_2 ALL 0.2692 0.0152 0.0288 LDC-MAX-ALL-Macro lilian_KB_ENG_2 0 0.0425 0.0299 0.0323 LDC-MAX-ALL-Macro lilian_KB_ENG_2 1 0.0000 0.0000 0.0000 LDC-MAX-ALL-Macro lilian_KB_ENG_2 ALL 0.0258 0.0182 0.0196 LDC-MEAN-ALL-Macro lilian_KB_ENG_2 0 0.0350 0.0227 0.0249 LDC-MEAN-ALL-Macro lilian_KB_ENG_2 1 0.0000 0.0000 0.0000 LDC-MEAN-ALL-Macro lilian_KB_ENG_2 ALL 0.0212 0.0138 0.0152 *ALL-Macro Prec, Recall and F1 refer to mean-precision, mean-recall and mean-F1. NIL-DETECTION P/R/F1: 0.3067 0.9675 0.4658 ************************************************************* Run ID: lilian_KB_ENG_3 Did the run access the live Web during the evaluation window: No Did the run extract relations from the Cold Start source corpus: Yes Did the run generate meaningful confidence values: No Entity Discovery Evaluation: ONLY English documents: Prec Recall F1 Metric 0.854 0.558 0.675 strong_mention_match 0.766 0.500 0.605 strong_typed_mention_match 0.000 0.000 0.000 entity_match 0.803 0.347 0.484 b_cubed 0.718 0.469 0.567 mention_ceaf 0.666 0.435 0.526 typed_mention_ceaf Slot Filling Evaluation: Metric RunID Hop Prec Recall F1 SF-ALL-Micro lilian_KB_ENG_3 0 0.2881 0.0212 0.0395 SF-ALL-Micro lilian_KB_ENG_3 1 0.0000 0.0000 0.0000 SF-ALL-Micro lilian_KB_ENG_3 ALL 0.2742 0.0141 0.0268 SF-ALL-Macro lilian_KB_ENG_3 0 0.0412 0.0241 0.0275 SF-ALL-Macro lilian_KB_ENG_3 1 0.0000 0.0000 0.0000 SF-ALL-Macro lilian_KB_ENG_3 ALL 0.0255 0.0150 0.0170 LDC-MAX-ALL-Micro lilian_KB_ENG_3 0 0.3023 0.0212 0.0397 LDC-MAX-ALL-Micro lilian_KB_ENG_3 1 0.0000 0.0000 0.0000 LDC-MAX-ALL-Micro lilian_KB_ENG_3 ALL 0.2889 0.0141 0.0269 LDC-MAX-ALL-Macro lilian_KB_ENG_3 0 0.0409 0.0256 0.0284 LDC-MAX-ALL-Macro lilian_KB_ENG_3 1 0.0000 0.0000 0.0000 LDC-MAX-ALL-Macro lilian_KB_ENG_3 ALL 0.0249 0.0155 0.0173 LDC-MEAN-ALL-Macro lilian_KB_ENG_3 0 0.0356 0.0206 0.0232 LDC-MEAN-ALL-Macro lilian_KB_ENG_3 1 0.0000 0.0000 0.0000 LDC-MEAN-ALL-Macro lilian_KB_ENG_3 ALL 0.0216 0.0125 0.0141 *ALL-Macro Prec, Recall and F1 refer to mean-precision, mean-recall and mean-F1. NIL-DETECTION P/R/F1: 0.3049 0.9593 0.4627