==================================================== TAC KBP 2016 ENGLISH SLOT FILLING EVALUATION RESULTS ==================================================== Team ID: MSR Organization: Microsoft Research ************************************************************* Run ID: MSR_SF_ENG_1 Did the run access the live Web during the evaluation window: No Did the run extract relations from the Cold Start source corpus: Yes Did the run generate meaningful confidence values: No Slot Filling Evaluation: Metric RunID Hop Prec Recall F1 SF-ALL-Micro MSR_SF_ENG_1 0 0.0214 0.0300 0.0250 SF-ALL-Micro MSR_SF_ENG_1 1 0.0000 0.0000 0.0000 SF-ALL-Micro MSR_SF_ENG_1 ALL 0.0104 0.0199 0.0136 SF-ALL-Macro MSR_SF_ENG_1 0 0.0297 0.0388 0.0291 SF-ALL-Macro MSR_SF_ENG_1 1 0.0000 0.0000 0.0000 SF-ALL-Macro MSR_SF_ENG_1 ALL 0.0184 0.0241 0.0180 LDC-MAX-ALL-Micro MSR_SF_ENG_1 0 0.0234 0.0310 0.0267 LDC-MAX-ALL-Micro MSR_SF_ENG_1 1 0.0000 0.0000 0.0000 LDC-MAX-ALL-Micro MSR_SF_ENG_1 ALL 0.0110 0.0207 0.0144 LDC-MAX-ALL-Macro MSR_SF_ENG_1 0 0.0345 0.0420 0.0325 LDC-MAX-ALL-Macro MSR_SF_ENG_1 1 0.0000 0.0000 0.0000 LDC-MAX-ALL-Macro MSR_SF_ENG_1 ALL 0.0209 0.0255 0.0197 LDC-MEAN-ALL-Macro MSR_SF_ENG_1 0 0.0309 0.0366 0.0284 LDC-MEAN-ALL-Macro MSR_SF_ENG_1 1 0.0000 0.0000 0.0000 LDC-MEAN-ALL-Macro MSR_SF_ENG_1 ALL 0.0188 0.0222 0.0173 *ALL-Macro Prec, Recall and F1 refer to mean-precision, mean-recall and mean-F1. NIL-DETECTION P/R/F1: 0.0074 0.0163 0.0102 ************************************************************* Run ID: MSR_SF_ENG_2 Did the run access the live Web during the evaluation window: No Did the run extract relations from the Cold Start source corpus: Yes Did the run generate meaningful confidence values: Yes Slot Filling Evaluation: Metric RunID Hop Prec Recall F1 SF-ALL-Micro MSR_SF_ENG_2 0 0.0494 0.0300 0.0373 SF-ALL-Micro MSR_SF_ENG_2 1 0.0000 0.0000 0.0000 SF-ALL-Micro MSR_SF_ENG_2 ALL 0.0331 0.0199 0.0248 SF-ALL-Macro MSR_SF_ENG_2 0 0.0732 0.0443 0.0500 SF-ALL-Macro MSR_SF_ENG_2 1 0.0000 0.0000 0.0000 SF-ALL-Macro MSR_SF_ENG_2 ALL 0.0454 0.0275 0.0310 LDC-MAX-ALL-Micro MSR_SF_ENG_2 0 0.0565 0.0327 0.0414 LDC-MAX-ALL-Micro MSR_SF_ENG_2 1 0.0000 0.0000 0.0000 LDC-MAX-ALL-Micro MSR_SF_ENG_2 ALL 0.0373 0.0217 0.0275 LDC-MAX-ALL-Macro MSR_SF_ENG_2 0 0.0862 0.0497 0.0564 LDC-MAX-ALL-Macro MSR_SF_ENG_2 1 0.0000 0.0000 0.0000 LDC-MAX-ALL-Macro MSR_SF_ENG_2 ALL 0.0524 0.0302 0.0342 LDC-MEAN-ALL-Macro MSR_SF_ENG_2 0 0.0841 0.0475 0.0542 LDC-MEAN-ALL-Macro MSR_SF_ENG_2 1 0.0000 0.0000 0.0000 LDC-MEAN-ALL-Macro MSR_SF_ENG_2 ALL 0.0510 0.0289 0.0329 *ALL-Macro Prec, Recall and F1 refer to mean-precision, mean-recall and mean-F1. NIL-DETECTION P/R/F1: 0.0000 0.0000 0.0000 ************************************************************* Run ID: MSR_SF_ENG_3 Did the run access the live Web during the evaluation window: No Did the run extract relations from the Cold Start source corpus: Yes Did the run generate meaningful confidence values: No Slot Filling Evaluation: Metric RunID Hop Prec Recall F1 SF-ALL-Micro MSR_SF_ENG_3 0 0.0215 0.0200 0.0207 SF-ALL-Micro MSR_SF_ENG_3 1 0.0000 0.0000 0.0000 SF-ALL-Micro MSR_SF_ENG_3 ALL 0.0120 0.0132 0.0126 SF-ALL-Macro MSR_SF_ENG_3 0 0.0310 0.0208 0.0207 SF-ALL-Macro MSR_SF_ENG_3 1 0.0000 0.0000 0.0000 SF-ALL-Macro MSR_SF_ENG_3 ALL 0.0192 0.0129 0.0128 LDC-MAX-ALL-Micro MSR_SF_ENG_3 0 0.0261 0.0229 0.0244 LDC-MAX-ALL-Micro MSR_SF_ENG_3 1 0.0000 0.0000 0.0000 LDC-MAX-ALL-Micro MSR_SF_ENG_3 ALL 0.0142 0.0152 0.0147 LDC-MAX-ALL-Macro MSR_SF_ENG_3 0 0.0402 0.0274 0.0269 LDC-MAX-ALL-Macro MSR_SF_ENG_3 1 0.0000 0.0000 0.0000 LDC-MAX-ALL-Macro MSR_SF_ENG_3 ALL 0.0244 0.0166 0.0163 LDC-MEAN-ALL-Macro MSR_SF_ENG_3 0 0.0348 0.0206 0.0215 LDC-MEAN-ALL-Macro MSR_SF_ENG_3 1 0.0000 0.0000 0.0000 LDC-MEAN-ALL-Macro MSR_SF_ENG_3 ALL 0.0212 0.0125 0.0131 *ALL-Macro Prec, Recall and F1 refer to mean-precision, mean-recall and mean-F1. NIL-DETECTION P/R/F1: 0.0254 0.0569 0.0351 ************************************************************* Run ID: MSR_SF_ENG_4 Did the run access the live Web during the evaluation window: No Did the run extract relations from the Cold Start source corpus: Yes Did the run generate meaningful confidence values: No Slot Filling Evaluation: Metric RunID Hop Prec Recall F1 SF-ALL-Micro MSR_SF_ENG_4 0 0.0532 0.0787 0.0634 SF-ALL-Micro MSR_SF_ENG_4 1 0.0062 0.0221 0.0096 SF-ALL-Micro MSR_SF_ENG_4 ALL 0.0272 0.0596 0.0374 SF-ALL-Macro MSR_SF_ENG_4 0 0.0973 0.0920 0.0812 SF-ALL-Macro MSR_SF_ENG_4 1 0.0144 0.0344 0.0187 SF-ALL-Macro MSR_SF_ENG_4 ALL 0.0658 0.0701 0.0575 LDC-MAX-ALL-Micro MSR_SF_ENG_4 0 0.0574 0.0801 0.0669 LDC-MAX-ALL-Micro MSR_SF_ENG_4 1 0.0072 0.0260 0.0113 LDC-MAX-ALL-Micro MSR_SF_ENG_4 ALL 0.0291 0.0620 0.0396 LDC-MAX-ALL-Macro MSR_SF_ENG_4 0 0.0995 0.0974 0.0832 LDC-MAX-ALL-Macro MSR_SF_ENG_4 1 0.0171 0.0394 0.0217 LDC-MAX-ALL-Macro MSR_SF_ENG_4 ALL 0.0671 0.0747 0.0591 LDC-MEAN-ALL-Macro MSR_SF_ENG_4 0 0.0945 0.0910 0.0778 LDC-MEAN-ALL-Macro MSR_SF_ENG_4 1 0.0171 0.0394 0.0217 LDC-MEAN-ALL-Macro MSR_SF_ENG_4 ALL 0.0641 0.0707 0.0558 *ALL-Macro Prec, Recall and F1 refer to mean-precision, mean-recall and mean-F1. NIL-DETECTION P/R/F1: 0.0074 0.0163 0.0102 ************************************************************* Run ID: MSR_SF_ENG_5 Did the run access the live Web during the evaluation window: No Did the run extract relations from the Cold Start source corpus: Yes Did the run generate meaningful confidence values: No Slot Filling Evaluation: Metric RunID Hop Prec Recall F1 SF-ALL-Micro MSR_SF_ENG_5 0 0.0167 0.0287 0.0211 SF-ALL-Micro MSR_SF_ENG_5 1 0.0000 0.0000 0.0000 SF-ALL-Micro MSR_SF_ENG_5 ALL 0.0068 0.0190 0.0100 SF-ALL-Macro MSR_SF_ENG_5 0 0.0216 0.0310 0.0222 SF-ALL-Macro MSR_SF_ENG_5 1 0.0000 0.0000 0.0000 SF-ALL-Macro MSR_SF_ENG_5 ALL 0.0134 0.0192 0.0138 LDC-MAX-ALL-Micro MSR_SF_ENG_5 0 0.0192 0.0310 0.0237 LDC-MAX-ALL-Micro MSR_SF_ENG_5 1 0.0000 0.0000 0.0000 LDC-MAX-ALL-Micro MSR_SF_ENG_5 ALL 0.0075 0.0207 0.0111 LDC-MAX-ALL-Macro MSR_SF_ENG_5 0 0.0259 0.0375 0.0265 LDC-MAX-ALL-Macro MSR_SF_ENG_5 1 0.0000 0.0000 0.0000 LDC-MAX-ALL-Macro MSR_SF_ENG_5 ALL 0.0157 0.0228 0.0161 LDC-MEAN-ALL-Macro MSR_SF_ENG_5 0 0.0248 0.0350 0.0252 LDC-MEAN-ALL-Macro MSR_SF_ENG_5 1 0.0000 0.0000 0.0000 LDC-MEAN-ALL-Macro MSR_SF_ENG_5 ALL 0.0151 0.0213 0.0153 *ALL-Macro Prec, Recall and F1 refer to mean-precision, mean-recall and mean-F1. NIL-DETECTION P/R/F1: 0.0074 0.0163 0.0102