==================================================== TAC KBP 2016 ENGLISH SLOT FILLING EVALUATION RESULTS ==================================================== Team ID: CMUML Organization: Carnegie Mellon University ************************************************************* Run ID: CMUML_SF_ENG_1 Did the run access the live Web during the evaluation window: No Did the run extract relations from the Cold Start source corpus: No Did the run generate meaningful confidence values: Yes Slot Filling Evaluation: Metric RunID Hop Prec Recall F1 SF-ALL-Micro CMUML_SF_ENG_1 0 0.2909 0.0200 0.0374 SF-ALL-Micro CMUML_SF_ENG_1 1 0.0000 0.0000 0.0000 SF-ALL-Micro CMUML_SF_ENG_1 ALL 0.1839 0.0132 0.0247 SF-ALL-Macro CMUML_SF_ENG_1 0 0.0414 0.0289 0.0317 SF-ALL-Macro CMUML_SF_ENG_1 1 0.0000 0.0000 0.0000 SF-ALL-Macro CMUML_SF_ENG_1 ALL 0.0257 0.0179 0.0197 LDC-MAX-ALL-Micro CMUML_SF_ENG_1 0 0.2708 0.0212 0.0394 LDC-MAX-ALL-Micro CMUML_SF_ENG_1 1 0.0000 0.0000 0.0000 LDC-MAX-ALL-Micro CMUML_SF_ENG_1 ALL 0.1831 0.0141 0.0262 LDC-MAX-ALL-Macro CMUML_SF_ENG_1 0 0.0456 0.0322 0.0348 LDC-MAX-ALL-Macro CMUML_SF_ENG_1 1 0.0000 0.0000 0.0000 LDC-MAX-ALL-Macro CMUML_SF_ENG_1 ALL 0.0277 0.0195 0.0211 LDC-MEAN-ALL-Macro CMUML_SF_ENG_1 0 0.0388 0.0257 0.0282 LDC-MEAN-ALL-Macro CMUML_SF_ENG_1 1 0.0000 0.0000 0.0000 LDC-MEAN-ALL-Macro CMUML_SF_ENG_1 ALL 0.0236 0.0156 0.0171 *ALL-Macro Prec, Recall and F1 refer to mean-precision, mean-recall and mean-F1. NIL-DETECTION P/R/F1: 0.3138 1.0000 0.4777 ************************************************************* Run ID: CMUML_SF_ENG_2 Did the run access the live Web during the evaluation window: No Did the run extract relations from the Cold Start source corpus: Yes Did the run generate meaningful confidence values: Yes Slot Filling Evaluation: Metric RunID Hop Prec Recall F1 SF-ALL-Micro CMUML_SF_ENG_2 0 0.1630 0.0187 0.0336 SF-ALL-Micro CMUML_SF_ENG_2 1 0.0000 0.0000 0.0000 SF-ALL-Micro CMUML_SF_ENG_2 ALL 0.1240 0.0124 0.0226 SF-ALL-Macro CMUML_SF_ENG_2 0 0.0340 0.0255 0.0263 SF-ALL-Macro CMUML_SF_ENG_2 1 0.0000 0.0000 0.0000 SF-ALL-Macro CMUML_SF_ENG_2 ALL 0.0211 0.0158 0.0163 LDC-MAX-ALL-Micro CMUML_SF_ENG_2 0 0.1757 0.0212 0.0379 LDC-MAX-ALL-Micro CMUML_SF_ENG_2 1 0.0000 0.0000 0.0000 LDC-MAX-ALL-Micro CMUML_SF_ENG_2 ALL 0.1313 0.0141 0.0255 LDC-MAX-ALL-Macro CMUML_SF_ENG_2 0 0.0416 0.0314 0.0324 LDC-MAX-ALL-Macro CMUML_SF_ENG_2 1 0.0000 0.0000 0.0000 LDC-MAX-ALL-Macro CMUML_SF_ENG_2 ALL 0.0252 0.0191 0.0197 LDC-MEAN-ALL-Macro CMUML_SF_ENG_2 0 0.0329 0.0228 0.0238 LDC-MEAN-ALL-Macro CMUML_SF_ENG_2 1 0.0000 0.0000 0.0000 LDC-MEAN-ALL-Macro CMUML_SF_ENG_2 ALL 0.0200 0.0139 0.0144 *ALL-Macro Prec, Recall and F1 refer to mean-precision, mean-recall and mean-F1. NIL-DETECTION P/R/F1: 0.3138 1.0000 0.4777 ************************************************************* Run ID: CMUML_SF_ENG_3 Did the run access the live Web during the evaluation window: No Did the run extract relations from the Cold Start source corpus: No Did the run generate meaningful confidence values: Yes Slot Filling Evaluation: Metric RunID Hop Prec Recall F1 SF-ALL-Micro CMUML_SF_ENG_3 0 0.0741 0.0025 0.0048 SF-ALL-Micro CMUML_SF_ENG_3 1 0.0000 0.0000 0.0000 SF-ALL-Micro CMUML_SF_ENG_3 ALL 0.0556 0.0017 0.0032 SF-ALL-Macro CMUML_SF_ENG_3 0 0.0038 0.0032 0.0033 SF-ALL-Macro CMUML_SF_ENG_3 1 0.0000 0.0000 0.0000 SF-ALL-Macro CMUML_SF_ENG_3 ALL 0.0024 0.0020 0.0020 LDC-MAX-ALL-Micro CMUML_SF_ENG_3 0 0.0800 0.0033 0.0063 LDC-MAX-ALL-Micro CMUML_SF_ENG_3 1 0.0000 0.0000 0.0000 LDC-MAX-ALL-Micro CMUML_SF_ENG_3 ALL 0.0588 0.0022 0.0042 LDC-MAX-ALL-Macro CMUML_SF_ENG_3 0 0.0054 0.0045 0.0047 LDC-MAX-ALL-Macro CMUML_SF_ENG_3 1 0.0000 0.0000 0.0000 LDC-MAX-ALL-Macro CMUML_SF_ENG_3 ALL 0.0033 0.0027 0.0028 LDC-MEAN-ALL-Macro CMUML_SF_ENG_3 0 0.0032 0.0024 0.0025 LDC-MEAN-ALL-Macro CMUML_SF_ENG_3 1 0.0000 0.0000 0.0000 LDC-MEAN-ALL-Macro CMUML_SF_ENG_3 ALL 0.0020 0.0014 0.0015 *ALL-Macro Prec, Recall and F1 refer to mean-precision, mean-recall and mean-F1. NIL-DETECTION P/R/F1: 0.3138 1.0000 0.4777 ************************************************************* Run ID: CMUML_SF_ENG_4 Did the run access the live Web during the evaluation window: No Did the run extract relations from the Cold Start source corpus: Yes Did the run generate meaningful confidence values: Yes Slot Filling Evaluation: Metric RunID Hop Prec Recall F1 SF-ALL-Micro CMUML_SF_ENG_4 0 0.4643 0.0162 0.0314 SF-ALL-Micro CMUML_SF_ENG_4 1 0.0000 0.0000 0.0000 SF-ALL-Micro CMUML_SF_ENG_4 ALL 0.3333 0.0108 0.0208 SF-ALL-Macro CMUML_SF_ENG_4 0 0.0381 0.0246 0.0275 SF-ALL-Macro CMUML_SF_ENG_4 1 0.0000 0.0000 0.0000 SF-ALL-Macro CMUML_SF_ENG_4 ALL 0.0236 0.0153 0.0171 LDC-MAX-ALL-Micro CMUML_SF_ENG_4 0 0.4762 0.0163 0.0316 LDC-MAX-ALL-Micro CMUML_SF_ENG_4 1 0.0000 0.0000 0.0000 LDC-MAX-ALL-Micro CMUML_SF_ENG_4 ALL 0.3448 0.0109 0.0211 LDC-MAX-ALL-Macro CMUML_SF_ENG_4 0 0.0409 0.0262 0.0289 LDC-MAX-ALL-Macro CMUML_SF_ENG_4 1 0.0000 0.0000 0.0000 LDC-MAX-ALL-Macro CMUML_SF_ENG_4 ALL 0.0249 0.0159 0.0175 LDC-MEAN-ALL-Macro CMUML_SF_ENG_4 0 0.0374 0.0226 0.0253 LDC-MEAN-ALL-Macro CMUML_SF_ENG_4 1 0.0000 0.0000 0.0000 LDC-MEAN-ALL-Macro CMUML_SF_ENG_4 ALL 0.0227 0.0137 0.0154 *ALL-Macro Prec, Recall and F1 refer to mean-precision, mean-recall and mean-F1. NIL-DETECTION P/R/F1: 0.3138 1.0000 0.4777 ************************************************************* Run ID: CMUML_SF_ENG_5 Did the run access the live Web during the evaluation window: No Did the run extract relations from the Cold Start source corpus: Yes Did the run generate meaningful confidence values: Yes Slot Filling Evaluation: Metric RunID Hop Prec Recall F1 SF-ALL-Micro CMUML_SF_ENG_5 0 0.2909 0.0200 0.0374 SF-ALL-Micro CMUML_SF_ENG_5 1 0.0000 0.0000 0.0000 SF-ALL-Micro CMUML_SF_ENG_5 ALL 0.1839 0.0132 0.0247 SF-ALL-Macro CMUML_SF_ENG_5 0 0.0414 0.0289 0.0317 SF-ALL-Macro CMUML_SF_ENG_5 1 0.0000 0.0000 0.0000 SF-ALL-Macro CMUML_SF_ENG_5 ALL 0.0257 0.0179 0.0197 LDC-MAX-ALL-Micro CMUML_SF_ENG_5 0 0.2766 0.0212 0.0395 LDC-MAX-ALL-Micro CMUML_SF_ENG_5 1 0.0000 0.0000 0.0000 LDC-MAX-ALL-Micro CMUML_SF_ENG_5 ALL 0.1733 0.0141 0.0261 LDC-MAX-ALL-Macro CMUML_SF_ENG_5 0 0.0456 0.0322 0.0348 LDC-MAX-ALL-Macro CMUML_SF_ENG_5 1 0.0000 0.0000 0.0000 LDC-MAX-ALL-Macro CMUML_SF_ENG_5 ALL 0.0277 0.0195 0.0211 LDC-MEAN-ALL-Macro CMUML_SF_ENG_5 0 0.0388 0.0257 0.0282 LDC-MEAN-ALL-Macro CMUML_SF_ENG_5 1 0.0000 0.0000 0.0000 LDC-MEAN-ALL-Macro CMUML_SF_ENG_5 ALL 0.0236 0.0156 0.0171 *ALL-Macro Prec, Recall and F1 refer to mean-precision, mean-recall and mean-F1. NIL-DETECTION P/R/F1: 0.3138 1.0000 0.4777