======================================================= TAC KBP 2016 ENGLISH KB CONSTRUCTION EVALUATION RESULTS ======================================================= Team ID: summa Organization: University College London ************************************************************* Run ID: summa_KB_ENG_1 Did the run access the live Web during the evaluation window: No Did the run extract relations from the Cold Start source corpus: Yes Did the run generate meaningful confidence values: No Entity Discovery Evaluation: ONLY English documents: Prec Recall F1 Metric 0.765 0.371 0.500 strong_mention_match 0.722 0.350 0.471 strong_typed_mention_match 0.000 0.000 0.000 entity_match 0.697 0.169 0.272 b_cubed 0.577 0.280 0.377 mention_ceaf 0.553 0.268 0.361 typed_mention_ceaf Slot Filling Evaluation: Metric RunID Hop Prec Recall F1 SF-ALL-Micro summa_KB_ENG_1 0 0.3171 0.0162 0.0309 SF-ALL-Micro summa_KB_ENG_1 1 0.0000 0.0000 0.0000 SF-ALL-Micro summa_KB_ENG_1 ALL 0.2955 0.0108 0.0208 SF-ALL-Macro summa_KB_ENG_1 0 0.0244 0.0122 0.0137 SF-ALL-Macro summa_KB_ENG_1 1 0.0000 0.0000 0.0000 SF-ALL-Macro summa_KB_ENG_1 ALL 0.0151 0.0076 0.0085 LDC-MAX-ALL-Micro summa_KB_ENG_1 0 0.2895 0.0180 0.0338 LDC-MAX-ALL-Micro summa_KB_ENG_1 1 0.0000 0.0000 0.0000 LDC-MAX-ALL-Micro summa_KB_ENG_1 ALL 0.2683 0.0120 0.0229 LDC-MAX-ALL-Macro summa_KB_ENG_1 0 0.0259 0.0127 0.0145 LDC-MAX-ALL-Macro summa_KB_ENG_1 1 0.0000 0.0000 0.0000 LDC-MAX-ALL-Macro summa_KB_ENG_1 ALL 0.0157 0.0077 0.0088 LDC-MEAN-ALL-Macro summa_KB_ENG_1 0 0.0194 0.0083 0.0098 LDC-MEAN-ALL-Macro summa_KB_ENG_1 1 0.0000 0.0000 0.0000 LDC-MEAN-ALL-Macro summa_KB_ENG_1 ALL 0.0118 0.0050 0.0059 *ALL-Macro Prec, Recall and F1 refer to mean-precision, mean-recall and mean-F1. NIL-DETECTION P/R/F1: 0.3138 1.0000 0.4777 ************************************************************* Run ID: summa_KB_ENG_2 Did the run access the live Web during the evaluation window: No Did the run extract relations from the Cold Start source corpus: Yes Did the run generate meaningful confidence values: No Entity Discovery Evaluation: ONLY English documents: Prec Recall F1 Metric 0.749 0.368 0.493 strong_mention_match 0.683 0.336 0.450 strong_typed_mention_match 0.000 0.000 0.000 entity_match 0.713 0.166 0.270 b_cubed 0.560 0.275 0.369 mention_ceaf 0.516 0.254 0.340 typed_mention_ceaf Slot Filling Evaluation: Metric RunID Hop Prec Recall F1 SF-ALL-Micro summa_KB_ENG_2 0 0.4884 0.0262 0.0498 SF-ALL-Micro summa_KB_ENG_2 1 0.2000 0.0049 0.0096 SF-ALL-Micro summa_KB_ENG_2 ALL 0.4340 0.0190 0.0365 SF-ALL-Macro summa_KB_ENG_2 0 0.0287 0.0118 0.0150 SF-ALL-Macro summa_KB_ENG_2 1 0.0100 0.0100 0.0100 SF-ALL-Macro summa_KB_ENG_2 ALL 0.0216 0.0111 0.0131 LDC-MAX-ALL-Micro summa_KB_ENG_2 0 0.4500 0.0294 0.0552 LDC-MAX-ALL-Micro summa_KB_ENG_2 1 0.2000 0.0065 0.0126 LDC-MAX-ALL-Micro summa_KB_ENG_2 ALL 0.4000 0.0217 0.0412 LDC-MAX-ALL-Macro summa_KB_ENG_2 0 0.0362 0.0159 0.0199 LDC-MAX-ALL-Macro summa_KB_ENG_2 1 0.0133 0.0133 0.0133 LDC-MAX-ALL-Macro summa_KB_ENG_2 ALL 0.0272 0.0149 0.0173 LDC-MEAN-ALL-Macro summa_KB_ENG_2 0 0.0293 0.0138 0.0171 LDC-MEAN-ALL-Macro summa_KB_ENG_2 1 0.0133 0.0133 0.0133 LDC-MEAN-ALL-Macro summa_KB_ENG_2 ALL 0.0230 0.0136 0.0156 *ALL-Macro Prec, Recall and F1 refer to mean-precision, mean-recall and mean-F1. NIL-DETECTION P/R/F1: 0.3138 1.0000 0.4777 ************************************************************* Run ID: summa_KB_ENG_5 Did the run access the live Web during the evaluation window: No Did the run extract relations from the Cold Start source corpus: Yes Did the run generate meaningful confidence values: No Entity Discovery Evaluation: ONLY English documents: Prec Recall F1 Metric 0.585 0.279 0.378 strong_mention_match 0.483 0.231 0.312 strong_typed_mention_match 0.000 0.000 0.000 entity_match 0.531 0.094 0.160 b_cubed 0.371 0.177 0.240 mention_ceaf 0.339 0.162 0.219 typed_mention_ceaf Slot Filling Evaluation: Metric RunID Hop Prec Recall F1 SF-ALL-Micro summa_KB_ENG_5 0 0.7500 0.0112 0.0221 SF-ALL-Micro summa_KB_ENG_5 1 0.4000 0.0049 0.0097 SF-ALL-Micro summa_KB_ENG_5 ALL 0.6471 0.0091 0.0179 SF-ALL-Macro summa_KB_ENG_5 0 0.0079 0.0066 0.0070 SF-ALL-Macro summa_KB_ENG_5 1 0.0100 0.0100 0.0100 SF-ALL-Macro summa_KB_ENG_5 ALL 0.0087 0.0079 0.0081 LDC-MAX-ALL-Micro summa_KB_ENG_5 0 0.7500 0.0147 0.0288 LDC-MAX-ALL-Micro summa_KB_ENG_5 1 0.4000 0.0065 0.0128 LDC-MAX-ALL-Micro summa_KB_ENG_5 ALL 0.6471 0.0120 0.0235 LDC-MAX-ALL-Macro summa_KB_ENG_5 0 0.0112 0.0093 0.0098 LDC-MAX-ALL-Macro summa_KB_ENG_5 1 0.0133 0.0133 0.0133 LDC-MAX-ALL-Macro summa_KB_ENG_5 ALL 0.0120 0.0109 0.0112 LDC-MEAN-ALL-Macro summa_KB_ENG_5 0 0.0112 0.0093 0.0098 LDC-MEAN-ALL-Macro summa_KB_ENG_5 1 0.0133 0.0133 0.0133 LDC-MEAN-ALL-Macro summa_KB_ENG_5 ALL 0.0120 0.0109 0.0112 *ALL-Macro Prec, Recall and F1 refer to mean-precision, mean-recall and mean-F1. NIL-DETECTION P/R/F1: 0.3138 1.0000 0.4777