======================================================= TAC KBP 2016 ENGLISH KB CONSTRUCTION EVALUATION RESULTS ======================================================= Team ID: UMass_IESL Organization: College of Information and Computer Sciences, University of Massachusetts Amherst ************************************************************* Run ID: UMass_IESL_KB_ENG_1 Did the run access the live Web during the evaluation window: No Did the run extract relations from the Cold Start source corpus: Yes Did the run generate meaningful confidence values: Yes Entity Discovery Evaluation: ONLY English documents: Prec Recall F1 Metric 0.874 0.409 0.557 strong_mention_match 0.818 0.382 0.521 strong_typed_mention_match 0.000 0.000 0.000 entity_match 0.845 0.233 0.366 b_cubed 0.787 0.368 0.501 mention_ceaf 0.752 0.352 0.479 typed_mention_ceaf Slot Filling Evaluation: Metric RunID Hop Prec Recall F1 SF-ALL-Micro UMass_IESL_KB_ENG_1 0 0.2684 0.1411 0.1849 SF-ALL-Micro UMass_IESL_KB_ENG_1 1 0.0531 0.0858 0.0656 SF-ALL-Micro UMass_IESL_KB_ENG_1 ALL 0.1370 0.1224 0.1293 SF-ALL-Macro UMass_IESL_KB_ENG_1 0 0.1610 0.1608 0.1435 SF-ALL-Macro UMass_IESL_KB_ENG_1 1 0.0848 0.1003 0.0840 SF-ALL-Macro UMass_IESL_KB_ENG_1 ALL 0.1320 0.1378 0.1209 LDC-MAX-ALL-Micro UMass_IESL_KB_ENG_1 0 0.2709 0.1536 0.1960 LDC-MAX-ALL-Micro UMass_IESL_KB_ENG_1 1 0.0392 0.0714 0.0506 LDC-MAX-ALL-Micro UMass_IESL_KB_ENG_1 ALL 0.1278 0.1261 0.1269 LDC-MAX-ALL-Macro UMass_IESL_KB_ENG_1 0 0.1769 0.1700 0.1528 LDC-MAX-ALL-Macro UMass_IESL_KB_ENG_1 1 0.0752 0.0917 0.0751 LDC-MAX-ALL-Macro UMass_IESL_KB_ENG_1 ALL 0.1369 0.1392 0.1223 LDC-MEAN-ALL-Macro UMass_IESL_KB_ENG_1 0 0.1645 0.1566 0.1418 LDC-MEAN-ALL-Macro UMass_IESL_KB_ENG_1 1 0.0752 0.0917 0.0751 LDC-MEAN-ALL-Macro UMass_IESL_KB_ENG_1 ALL 0.1294 0.1311 0.1156 *ALL-Macro Prec, Recall and F1 refer to mean-precision, mean-recall and mean-F1. NIL-DETECTION P/R/F1: 0.2865 0.8780 0.4320 ************************************************************* Run ID: UMass_IESL_KB_ENG_2 Did the run access the live Web during the evaluation window: No Did the run extract relations from the Cold Start source corpus: Yes Did the run generate meaningful confidence values: Yes Entity Discovery Evaluation: ONLY English documents: Prec Recall F1 Metric 0.874 0.409 0.557 strong_mention_match 0.818 0.382 0.521 strong_typed_mention_match 0.000 0.000 0.000 entity_match 0.845 0.233 0.366 b_cubed 0.787 0.368 0.501 mention_ceaf 0.752 0.352 0.479 typed_mention_ceaf Slot Filling Evaluation: Metric RunID Hop Prec Recall F1 SF-ALL-Micro UMass_IESL_KB_ENG_2 0 0.3082 0.1124 0.1647 SF-ALL-Micro UMass_IESL_KB_ENG_2 1 0.0710 0.0637 0.0672 SF-ALL-Micro UMass_IESL_KB_ENG_2 ALL 0.1763 0.0959 0.1243 SF-ALL-Macro UMass_IESL_KB_ENG_2 0 0.1572 0.1327 0.1300 SF-ALL-Macro UMass_IESL_KB_ENG_2 1 0.0692 0.0763 0.0645 SF-ALL-Macro UMass_IESL_KB_ENG_2 ALL 0.1238 0.1113 0.1051 LDC-MAX-ALL-Micro UMass_IESL_KB_ENG_2 0 0.3008 0.1209 0.1725 LDC-MAX-ALL-Micro UMass_IESL_KB_ENG_2 1 0.0539 0.0519 0.0529 LDC-MAX-ALL-Micro UMass_IESL_KB_ENG_2 ALL 0.1657 0.0978 0.1230 LDC-MAX-ALL-Macro UMass_IESL_KB_ENG_2 0 0.1673 0.1342 0.1318 LDC-MAX-ALL-Macro UMass_IESL_KB_ENG_2 1 0.0571 0.0656 0.0540 LDC-MAX-ALL-Macro UMass_IESL_KB_ENG_2 ALL 0.1240 0.1072 0.1012 LDC-MEAN-ALL-Macro UMass_IESL_KB_ENG_2 0 0.1519 0.1218 0.1207 LDC-MEAN-ALL-Macro UMass_IESL_KB_ENG_2 1 0.0571 0.0656 0.0540 LDC-MEAN-ALL-Macro UMass_IESL_KB_ENG_2 ALL 0.1147 0.0997 0.0945 *ALL-Macro Prec, Recall and F1 refer to mean-precision, mean-recall and mean-F1. NIL-DETECTION P/R/F1: 0.2921 0.9024 0.4413 ************************************************************* Run ID: UMass_IESL_KB_ENG_3 Did the run access the live Web during the evaluation window: No Did the run extract relations from the Cold Start source corpus: Yes Did the run generate meaningful confidence values: Yes Entity Discovery Evaluation: ONLY English documents: Prec Recall F1 Metric 0.798 0.412 0.543 strong_mention_match 0.723 0.373 0.492 strong_typed_mention_match 0.000 0.000 0.000 entity_match 0.772 0.226 0.350 b_cubed 0.715 0.369 0.487 mention_ceaf 0.677 0.349 0.461 typed_mention_ceaf Slot Filling Evaluation: Metric RunID Hop Prec Recall F1 SF-ALL-Micro UMass_IESL_KB_ENG_3 0 0.3977 0.1286 0.1943 SF-ALL-Micro UMass_IESL_KB_ENG_3 1 0.2556 0.0833 0.1257 SF-ALL-Micro UMass_IESL_KB_ENG_3 ALL 0.3495 0.1133 0.1711 SF-ALL-Macro UMass_IESL_KB_ENG_3 0 0.1491 0.1217 0.1252 SF-ALL-Macro UMass_IESL_KB_ENG_3 1 0.0975 0.1061 0.0957 SF-ALL-Macro UMass_IESL_KB_ENG_3 ALL 0.1295 0.1158 0.1140 LDC-MAX-ALL-Micro UMass_IESL_KB_ENG_3 0 0.3900 0.1275 0.1921 LDC-MAX-ALL-Micro UMass_IESL_KB_ENG_3 1 0.2526 0.0779 0.1191 LDC-MAX-ALL-Micro UMass_IESL_KB_ENG_3 ALL 0.3458 0.1109 0.1679 LDC-MAX-ALL-Macro UMass_IESL_KB_ENG_3 0 0.1472 0.1174 0.1213 LDC-MAX-ALL-Macro UMass_IESL_KB_ENG_3 1 0.0989 0.1106 0.0977 LDC-MAX-ALL-Macro UMass_IESL_KB_ENG_3 ALL 0.1282 0.1147 0.1120 LDC-MEAN-ALL-Macro UMass_IESL_KB_ENG_3 0 0.1400 0.1105 0.1144 LDC-MEAN-ALL-Macro UMass_IESL_KB_ENG_3 1 0.0989 0.1106 0.0977 LDC-MEAN-ALL-Macro UMass_IESL_KB_ENG_3 ALL 0.1239 0.1105 0.1078 *ALL-Macro Prec, Recall and F1 refer to mean-precision, mean-recall and mean-F1. NIL-DETECTION P/R/F1: 0.3031 0.9512 0.4597 ************************************************************* Run ID: UMass_IESL_KB_ENG_4 Did the run access the live Web during the evaluation window: No Did the run extract relations from the Cold Start source corpus: Yes Did the run generate meaningful confidence values: Yes Entity Discovery Evaluation: ONLY English documents: Prec Recall F1 Metric 0.874 0.409 0.557 strong_mention_match 0.818 0.382 0.521 strong_typed_mention_match 0.000 0.000 0.000 entity_match 0.845 0.233 0.366 b_cubed 0.787 0.368 0.501 mention_ceaf 0.752 0.352 0.479 typed_mention_ceaf Slot Filling Evaluation: Metric RunID Hop Prec Recall F1 SF-ALL-Micro UMass_IESL_KB_ENG_4 0 0.2903 0.1124 0.1620 SF-ALL-Micro UMass_IESL_KB_ENG_4 1 0.1633 0.0784 0.1060 SF-ALL-Micro UMass_IESL_KB_ENG_4 ALL 0.2411 0.1009 0.1423 SF-ALL-Macro UMass_IESL_KB_ENG_4 0 0.1486 0.1425 0.1312 SF-ALL-Macro UMass_IESL_KB_ENG_4 1 0.0797 0.0929 0.0781 SF-ALL-Macro UMass_IESL_KB_ENG_4 ALL 0.1224 0.1237 0.1110 LDC-MAX-ALL-Micro UMass_IESL_KB_ENG_4 0 0.3004 0.1193 0.1708 LDC-MAX-ALL-Micro UMass_IESL_KB_ENG_4 1 0.1170 0.0649 0.0835 LDC-MAX-ALL-Micro UMass_IESL_KB_ENG_4 ALL 0.2246 0.1011 0.1394 LDC-MAX-ALL-Macro UMass_IESL_KB_ENG_4 0 0.1634 0.1487 0.1385 LDC-MAX-ALL-Macro UMass_IESL_KB_ENG_4 1 0.0684 0.0833 0.0679 LDC-MAX-ALL-Macro UMass_IESL_KB_ENG_4 ALL 0.1261 0.1230 0.1108 LDC-MEAN-ALL-Macro UMass_IESL_KB_ENG_4 0 0.1522 0.1382 0.1289 LDC-MEAN-ALL-Macro UMass_IESL_KB_ENG_4 1 0.0684 0.0833 0.0679 LDC-MEAN-ALL-Macro UMass_IESL_KB_ENG_4 ALL 0.1193 0.1166 0.1049 *ALL-Macro Prec, Recall and F1 refer to mean-precision, mean-recall and mean-F1. NIL-DETECTION P/R/F1: 0.2995 0.9350 0.4537 ************************************************************* Run ID: UMass_IESL_KB_ENG_5 Did the run access the live Web during the evaluation window: No Did the run extract relations from the Cold Start source corpus: Yes Did the run generate meaningful confidence values: Yes Entity Discovery Evaluation: ONLY English documents: Prec Recall F1 Metric 0.874 0.409 0.557 strong_mention_match 0.818 0.382 0.521 strong_typed_mention_match 0.000 0.000 0.000 entity_match 0.845 0.233 0.366 b_cubed 0.787 0.368 0.501 mention_ceaf 0.752 0.352 0.479 typed_mention_ceaf Slot Filling Evaluation: Metric RunID Hop Prec Recall F1 SF-ALL-Micro UMass_IESL_KB_ENG_5 0 0.2047 0.1511 0.1739 SF-ALL-Micro UMass_IESL_KB_ENG_5 1 0.0259 0.1103 0.0419 SF-ALL-Micro UMass_IESL_KB_ENG_5 ALL 0.0712 0.1373 0.0938 SF-ALL-Macro UMass_IESL_KB_ENG_5 0 0.1534 0.1666 0.1438 SF-ALL-Macro UMass_IESL_KB_ENG_5 1 0.1028 0.1335 0.1053 SF-ALL-Macro UMass_IESL_KB_ENG_5 ALL 0.1342 0.1540 0.1292 LDC-MAX-ALL-Micro UMass_IESL_KB_ENG_5 0 0.2110 0.1634 0.1842 LDC-MAX-ALL-Micro UMass_IESL_KB_ENG_5 1 0.0191 0.0942 0.0317 LDC-MAX-ALL-Micro UMass_IESL_KB_ENG_5 ALL 0.0647 0.1402 0.0886 LDC-MAX-ALL-Macro UMass_IESL_KB_ENG_5 0 0.1633 0.1754 0.1485 LDC-MAX-ALL-Macro UMass_IESL_KB_ENG_5 1 0.0868 0.1161 0.0891 LDC-MAX-ALL-Macro UMass_IESL_KB_ENG_5 ALL 0.1333 0.1521 0.1252 LDC-MEAN-ALL-Macro UMass_IESL_KB_ENG_5 0 0.1510 0.1644 0.1388 LDC-MEAN-ALL-Macro UMass_IESL_KB_ENG_5 1 0.0868 0.1144 0.0890 LDC-MEAN-ALL-Macro UMass_IESL_KB_ENG_5 ALL 0.1258 0.1448 0.1192 *ALL-Macro Prec, Recall and F1 refer to mean-precision, mean-recall and mean-F1. NIL-DETECTION P/R/F1: 0.2749 0.8293 0.4129