==================================================== TAC KBP 2016 ENGLISH SLOT FILLING EVALUATION RESULTS ==================================================== Team ID: UMass_IESL Organization: College of Information and Computer Sciences, University of Massachusetts Amherst ************************************************************* Run ID: UMass_IESL_SF_ENG_1 Did the run access the live Web during the evaluation window: No Did the run extract relations from the Cold Start source corpus: Yes Did the run generate meaningful confidence values: Yes Slot Filling Evaluation: Metric RunID Hop Prec Recall F1 SF-ALL-Micro UMass_IESL_SF_ENG_1 0 0.2253 0.2047 0.2145 SF-ALL-Micro UMass_IESL_SF_ENG_1 1 0.4375 0.0172 0.0330 SF-ALL-Micro UMass_IESL_SF_ENG_1 ALL 0.2298 0.1414 0.1751 SF-ALL-Macro UMass_IESL_SF_ENG_1 0 0.1925 0.2212 0.1890 SF-ALL-Macro UMass_IESL_SF_ENG_1 1 0.0216 0.0249 0.0229 SF-ALL-Macro UMass_IESL_SF_ENG_1 ALL 0.1276 0.1466 0.1259 LDC-MAX-ALL-Micro UMass_IESL_SF_ENG_1 0 0.2288 0.2026 0.2149 LDC-MAX-ALL-Micro UMass_IESL_SF_ENG_1 1 0.4444 0.0130 0.0252 LDC-MAX-ALL-Micro UMass_IESL_SF_ENG_1 ALL 0.2323 0.1391 0.1740 LDC-MAX-ALL-Macro UMass_IESL_SF_ENG_1 0 0.1942 0.2217 0.1903 LDC-MAX-ALL-Macro UMass_IESL_SF_ENG_1 1 0.0178 0.0200 0.0187 LDC-MAX-ALL-Macro UMass_IESL_SF_ENG_1 ALL 0.1249 0.1425 0.1229 LDC-MEAN-ALL-Macro UMass_IESL_SF_ENG_1 0 0.1857 0.2123 0.1824 LDC-MEAN-ALL-Macro UMass_IESL_SF_ENG_1 1 0.0144 0.0167 0.0153 LDC-MEAN-ALL-Macro UMass_IESL_SF_ENG_1 ALL 0.1184 0.1355 0.1168 *ALL-Macro Prec, Recall and F1 refer to mean-precision, mean-recall and mean-F1. NIL-DETECTION P/R/F1: 0.2548 0.7480 0.3801 ************************************************************* Run ID: UMass_IESL_SF_ENG_2 Did the run access the live Web during the evaluation window: No Did the run extract relations from the Cold Start source corpus: Yes Did the run generate meaningful confidence values: Yes Slot Filling Evaluation: Metric RunID Hop Prec Recall F1 SF-ALL-Micro UMass_IESL_SF_ENG_2 0 0.2460 0.1910 0.2150 SF-ALL-Micro UMass_IESL_SF_ENG_2 1 0.1176 0.0245 0.0406 SF-ALL-Micro UMass_IESL_SF_ENG_2 ALL 0.2306 0.1348 0.1701 SF-ALL-Macro UMass_IESL_SF_ENG_2 0 0.1800 0.1989 0.1737 SF-ALL-Macro UMass_IESL_SF_ENG_2 1 0.0365 0.0398 0.0378 SF-ALL-Macro UMass_IESL_SF_ENG_2 ALL 0.1254 0.1384 0.1221 LDC-MAX-ALL-Micro UMass_IESL_SF_ENG_2 0 0.2442 0.1895 0.2134 LDC-MAX-ALL-Micro UMass_IESL_SF_ENG_2 1 0.1304 0.0195 0.0339 LDC-MAX-ALL-Micro UMass_IESL_SF_ENG_2 ALL 0.2342 0.1326 0.1693 LDC-MAX-ALL-Macro UMass_IESL_SF_ENG_2 0 0.1790 0.1959 0.1711 LDC-MAX-ALL-Macro UMass_IESL_SF_ENG_2 1 0.0311 0.0333 0.0320 LDC-MAX-ALL-Macro UMass_IESL_SF_ENG_2 ALL 0.1209 0.1321 0.1165 LDC-MEAN-ALL-Macro UMass_IESL_SF_ENG_2 0 0.1693 0.1837 0.1618 LDC-MEAN-ALL-Macro UMass_IESL_SF_ENG_2 1 0.0278 0.0300 0.0287 LDC-MEAN-ALL-Macro UMass_IESL_SF_ENG_2 ALL 0.1137 0.1233 0.1095 *ALL-Macro Prec, Recall and F1 refer to mean-precision, mean-recall and mean-F1. NIL-DETECTION P/R/F1: 0.2730 0.8211 0.4098 ************************************************************* Run ID: UMass_IESL_SF_ENG_3 Did the run access the live Web during the evaluation window: No Did the run extract relations from the Cold Start source corpus: Yes Did the run generate meaningful confidence values: Yes Slot Filling Evaluation: Metric RunID Hop Prec Recall F1 SF-ALL-Micro UMass_IESL_SF_ENG_3 0 0.2840 0.1461 0.1929 SF-ALL-Micro UMass_IESL_SF_ENG_3 1 0.0824 0.0735 0.0777 SF-ALL-Micro UMass_IESL_SF_ENG_3 ALL 0.1894 0.1216 0.1481 SF-ALL-Macro UMass_IESL_SF_ENG_3 0 0.1948 0.1461 0.1502 SF-ALL-Macro UMass_IESL_SF_ENG_3 1 0.0650 0.0757 0.0653 SF-ALL-Macro UMass_IESL_SF_ENG_3 ALL 0.1455 0.1194 0.1179 LDC-MAX-ALL-Micro UMass_IESL_SF_ENG_3 0 0.3023 0.1536 0.2037 LDC-MAX-ALL-Micro UMass_IESL_SF_ENG_3 1 0.0811 0.0877 0.0842 LDC-MAX-ALL-Micro UMass_IESL_SF_ENG_3 ALL 0.1879 0.1315 0.1547 LDC-MAX-ALL-Macro UMass_IESL_SF_ENG_3 0 0.2061 0.1527 0.1592 LDC-MAX-ALL-Macro UMass_IESL_SF_ENG_3 1 0.0822 0.0931 0.0815 LDC-MAX-ALL-Macro UMass_IESL_SF_ENG_3 ALL 0.1574 0.1293 0.1287 LDC-MEAN-ALL-Macro UMass_IESL_SF_ENG_3 0 0.1892 0.1416 0.1456 LDC-MEAN-ALL-Macro UMass_IESL_SF_ENG_3 1 0.0789 0.0898 0.0782 LDC-MEAN-ALL-Macro UMass_IESL_SF_ENG_3 ALL 0.1459 0.1213 0.1192 *ALL-Macro Prec, Recall and F1 refer to mean-precision, mean-recall and mean-F1. NIL-DETECTION P/R/F1: 0.2940 0.9106 0.4445 ************************************************************* Run ID: UMass_IESL_SF_ENG_4 Did the run access the live Web during the evaluation window: No Did the run extract relations from the Cold Start source corpus: Yes Did the run generate meaningful confidence values: Yes Slot Filling Evaluation: Metric RunID Hop Prec Recall F1 SF-ALL-Micro UMass_IESL_SF_ENG_4 0 0.2424 0.1698 0.1997 SF-ALL-Micro UMass_IESL_SF_ENG_4 1 0.4667 0.0172 0.0331 SF-ALL-Micro UMass_IESL_SF_ENG_4 ALL 0.2483 0.1183 0.1602 SF-ALL-Macro UMass_IESL_SF_ENG_4 0 0.1670 0.1900 0.1673 SF-ALL-Macro UMass_IESL_SF_ENG_4 1 0.0216 0.0249 0.0229 SF-ALL-Macro UMass_IESL_SF_ENG_4 ALL 0.1118 0.1273 0.1124 LDC-MAX-ALL-Micro UMass_IESL_SF_ENG_4 0 0.2441 0.1683 0.1992 LDC-MAX-ALL-Micro UMass_IESL_SF_ENG_4 1 0.5000 0.0130 0.0253 LDC-MAX-ALL-Micro UMass_IESL_SF_ENG_4 ALL 0.2488 0.1163 0.1585 LDC-MAX-ALL-Macro UMass_IESL_SF_ENG_4 0 0.1698 0.1885 0.1675 LDC-MAX-ALL-Macro UMass_IESL_SF_ENG_4 1 0.0178 0.0200 0.0187 LDC-MAX-ALL-Macro UMass_IESL_SF_ENG_4 ALL 0.1101 0.1223 0.1090 LDC-MEAN-ALL-Macro UMass_IESL_SF_ENG_4 0 0.1609 0.1784 0.1589 LDC-MEAN-ALL-Macro UMass_IESL_SF_ENG_4 1 0.0144 0.0167 0.0153 LDC-MEAN-ALL-Macro UMass_IESL_SF_ENG_4 ALL 0.1034 0.1149 0.1025 *ALL-Macro Prec, Recall and F1 refer to mean-precision, mean-recall and mean-F1. NIL-DETECTION P/R/F1: 0.2650 0.7886 0.3967 ************************************************************* Run ID: UMass_IESL_SF_ENG_5 Did the run access the live Web during the evaluation window: No Did the run extract relations from the Cold Start source corpus: Yes Did the run generate meaningful confidence values: Yes Slot Filling Evaluation: Metric RunID Hop Prec Recall F1 SF-ALL-Micro UMass_IESL_SF_ENG_5 0 0.2777 0.1598 0.2029 SF-ALL-Micro UMass_IESL_SF_ENG_5 1 0.5833 0.0172 0.0333 SF-ALL-Micro UMass_IESL_SF_ENG_5 ALL 0.2854 0.1117 0.1605 SF-ALL-Macro UMass_IESL_SF_ENG_5 0 0.1969 0.1948 0.1763 SF-ALL-Macro UMass_IESL_SF_ENG_5 1 0.0216 0.0249 0.0229 SF-ALL-Macro UMass_IESL_SF_ENG_5 ALL 0.1303 0.1302 0.1180 LDC-MAX-ALL-Micro UMass_IESL_SF_ENG_5 0 0.2893 0.1503 0.1978 LDC-MAX-ALL-Micro UMass_IESL_SF_ENG_5 1 0.5714 0.0130 0.0254 LDC-MAX-ALL-Micro UMass_IESL_SF_ENG_5 ALL 0.2954 0.1043 0.1542 LDC-MAX-ALL-Macro UMass_IESL_SF_ENG_5 0 0.1952 0.1913 0.1734 LDC-MAX-ALL-Macro UMass_IESL_SF_ENG_5 1 0.0178 0.0200 0.0187 LDC-MAX-ALL-Macro UMass_IESL_SF_ENG_5 ALL 0.1256 0.1241 0.1127 LDC-MEAN-ALL-Macro UMass_IESL_SF_ENG_5 0 0.1872 0.1838 0.1664 LDC-MEAN-ALL-Macro UMass_IESL_SF_ENG_5 1 0.0144 0.0167 0.0153 LDC-MEAN-ALL-Macro UMass_IESL_SF_ENG_5 ALL 0.1193 0.1182 0.1071 *ALL-Macro Prec, Recall and F1 refer to mean-precision, mean-recall and mean-F1. NIL-DETECTION P/R/F1: 0.2788 0.8455 0.4193