======================================================= TAC KBP 2016 ENGLISH KB CONSTRUCTION EVALUATION RESULTS ======================================================= Team ID: BBN Organization: BBN ************************************************************* Run ID: BBN_KB_ENG_1 Did the run access the live Web during the evaluation window: No Did the run extract relations from the Cold Start source corpus: Yes Did the run generate meaningful confidence values: Yes Entity Discovery Evaluation: ONLY English documents: Prec Recall F1 Metric 0.879 0.683 0.769 strong_mention_match 0.836 0.649 0.731 strong_typed_mention_match 0.000 0.000 0.000 entity_match 0.781 0.510 0.617 b_cubed 0.786 0.610 0.687 mention_ceaf 0.765 0.594 0.669 typed_mention_ceaf Slot Filling Evaluation: Metric RunID Hop Prec Recall F1 SF-ALL-Micro BBN_KB_ENG_1 0 0.4662 0.2409 0.3177 SF-ALL-Micro BBN_KB_ENG_1 1 0.2112 0.1299 0.1608 SF-ALL-Micro BBN_KB_ENG_1 ALL 0.3699 0.2035 0.2625 SF-ALL-Macro BBN_KB_ENG_1 0 0.2589 0.2305 0.2277 SF-ALL-Macro BBN_KB_ENG_1 1 0.1237 0.1351 0.1217 SF-ALL-Macro BBN_KB_ENG_1 ALL 0.2075 0.1943 0.1874 LDC-MAX-ALL-Micro BBN_KB_ENG_1 0 0.4878 0.2614 0.3404 LDC-MAX-ALL-Micro BBN_KB_ENG_1 1 0.1951 0.1299 0.1559 LDC-MAX-ALL-Micro BBN_KB_ENG_1 ALL 0.3752 0.2174 0.2753 LDC-MAX-ALL-Macro BBN_KB_ENG_1 0 0.2897 0.2521 0.2492 LDC-MAX-ALL-Macro BBN_KB_ENG_1 1 0.1257 0.1376 0.1237 LDC-MAX-ALL-Macro BBN_KB_ENG_1 ALL 0.2253 0.2071 0.2000 LDC-MEAN-ALL-Macro BBN_KB_ENG_1 0 0.2640 0.2326 0.2279 LDC-MEAN-ALL-Macro BBN_KB_ENG_1 1 0.1190 0.1295 0.1172 LDC-MEAN-ALL-Macro BBN_KB_ENG_1 ALL 0.2070 0.1921 0.1844 *ALL-Macro Prec, Recall and F1 refer to mean-precision, mean-recall and mean-F1. NIL-DETECTION P/R/F1: 0.3085 0.9756 0.4688 ************************************************************* Run ID: BBN_KB_ENG_2 Did the run access the live Web during the evaluation window: No Did the run extract relations from the Cold Start source corpus: Yes Did the run generate meaningful confidence values: Yes Entity Discovery Evaluation: ONLY English documents: Prec Recall F1 Metric 0.887 0.664 0.759 strong_mention_match 0.844 0.632 0.723 strong_typed_mention_match 0.000 0.000 0.000 entity_match 0.789 0.490 0.605 b_cubed 0.795 0.596 0.681 mention_ceaf 0.774 0.579 0.663 typed_mention_ceaf Slot Filling Evaluation: Metric RunID Hop Prec Recall F1 SF-ALL-Micro BBN_KB_ENG_2 0 0.4632 0.2197 0.2981 SF-ALL-Micro BBN_KB_ENG_2 1 0.2110 0.1225 0.1550 SF-ALL-Micro BBN_KB_ENG_2 ALL 0.3663 0.1869 0.2475 SF-ALL-Macro BBN_KB_ENG_2 0 0.2406 0.2120 0.2091 SF-ALL-Macro BBN_KB_ENG_2 1 0.1135 0.1247 0.1114 SF-ALL-Macro BBN_KB_ENG_2 ALL 0.1923 0.1788 0.1720 LDC-MAX-ALL-Micro BBN_KB_ENG_2 0 0.4916 0.2402 0.3227 LDC-MAX-ALL-Micro BBN_KB_ENG_2 1 0.1927 0.1201 0.1480 LDC-MAX-ALL-Micro BBN_KB_ENG_2 ALL 0.3747 0.2000 0.2608 LDC-MAX-ALL-Macro BBN_KB_ENG_2 0 0.2767 0.2398 0.2366 LDC-MAX-ALL-Macro BBN_KB_ENG_2 1 0.1121 0.1238 0.1100 LDC-MAX-ALL-Macro BBN_KB_ENG_2 ALL 0.2121 0.1943 0.1869 LDC-MEAN-ALL-Macro BBN_KB_ENG_2 0 0.2523 0.2212 0.2163 LDC-MEAN-ALL-Macro BBN_KB_ENG_2 1 0.1053 0.1157 0.1034 LDC-MEAN-ALL-Macro BBN_KB_ENG_2 ALL 0.1946 0.1798 0.1719 *ALL-Macro Prec, Recall and F1 refer to mean-precision, mean-recall and mean-F1. NIL-DETECTION P/R/F1: 0.3120 0.9919 0.4747 ************************************************************* Run ID: BBN_KB_ENG_3 Did the run access the live Web during the evaluation window: No Did the run extract relations from the Cold Start source corpus: Yes Did the run generate meaningful confidence values: Yes Entity Discovery Evaluation: ONLY English documents: Prec Recall F1 Metric 0.877 0.686 0.770 strong_mention_match 0.834 0.652 0.732 strong_typed_mention_match 0.000 0.000 0.000 entity_match 0.779 0.514 0.619 b_cubed 0.784 0.613 0.688 mention_ceaf 0.763 0.597 0.670 typed_mention_ceaf Slot Filling Evaluation: Metric RunID Hop Prec Recall F1 SF-ALL-Micro BBN_KB_ENG_3 0 0.4539 0.2397 0.3137 SF-ALL-Micro BBN_KB_ENG_3 1 0.1918 0.1373 0.1600 SF-ALL-Micro BBN_KB_ENG_3 ALL 0.3469 0.2051 0.2578 SF-ALL-Macro BBN_KB_ENG_3 0 0.2639 0.2351 0.2328 SF-ALL-Macro BBN_KB_ENG_3 1 0.1230 0.1372 0.1212 SF-ALL-Macro BBN_KB_ENG_3 ALL 0.2104 0.1979 0.1904 LDC-MAX-ALL-Micro BBN_KB_ENG_3 0 0.4702 0.2582 0.3333 LDC-MAX-ALL-Micro BBN_KB_ENG_3 1 0.1755 0.1396 0.1555 LDC-MAX-ALL-Micro BBN_KB_ENG_3 ALL 0.3460 0.2185 0.2678 LDC-MAX-ALL-Macro BBN_KB_ENG_3 0 0.2890 0.2535 0.2504 LDC-MAX-ALL-Macro BBN_KB_ENG_3 1 0.1248 0.1405 0.1231 LDC-MAX-ALL-Macro BBN_KB_ENG_3 ALL 0.2245 0.2091 0.2004 LDC-MEAN-ALL-Macro BBN_KB_ENG_3 0 0.2676 0.2357 0.2314 LDC-MEAN-ALL-Macro BBN_KB_ENG_3 1 0.1185 0.1310 0.1169 LDC-MEAN-ALL-Macro BBN_KB_ENG_3 ALL 0.2091 0.1946 0.1864 *ALL-Macro Prec, Recall and F1 refer to mean-precision, mean-recall and mean-F1. NIL-DETECTION P/R/F1: 0.3085 0.9756 0.4688 ************************************************************* Run ID: BBN_KB_ENG_4 Did the run access the live Web during the evaluation window: No Did the run extract relations from the Cold Start source corpus: Yes Did the run generate meaningful confidence values: Yes Entity Discovery Evaluation: ONLY English documents: Prec Recall F1 Metric 0.877 0.686 0.770 strong_mention_match 0.835 0.653 0.733 strong_typed_mention_match 0.000 0.000 0.000 entity_match 0.779 0.515 0.620 b_cubed 0.785 0.614 0.689 mention_ceaf 0.764 0.598 0.671 typed_mention_ceaf Slot Filling Evaluation: Metric RunID Hop Prec Recall F1 SF-ALL-Micro BBN_KB_ENG_4 0 0.4599 0.2434 0.3184 SF-ALL-Micro BBN_KB_ENG_4 1 0.2103 0.1299 0.1606 SF-ALL-Micro BBN_KB_ENG_4 ALL 0.3669 0.2051 0.2631 SF-ALL-Macro BBN_KB_ENG_4 0 0.2600 0.2322 0.2295 SF-ALL-Macro BBN_KB_ENG_4 1 0.1236 0.1351 0.1217 SF-ALL-Macro BBN_KB_ENG_4 ALL 0.2082 0.1953 0.1885 LDC-MAX-ALL-Micro BBN_KB_ENG_4 0 0.4806 0.2631 0.3400 LDC-MAX-ALL-Micro BBN_KB_ENG_4 1 0.1942 0.1299 0.1556 LDC-MAX-ALL-Micro BBN_KB_ENG_4 ALL 0.3715 0.2185 0.2752 LDC-MAX-ALL-Macro BBN_KB_ENG_4 0 0.2869 0.2523 0.2489 LDC-MAX-ALL-Macro BBN_KB_ENG_4 1 0.1257 0.1376 0.1237 LDC-MAX-ALL-Macro BBN_KB_ENG_4 ALL 0.2236 0.2073 0.1997 LDC-MEAN-ALL-Macro BBN_KB_ENG_4 0 0.2650 0.2339 0.2293 LDC-MEAN-ALL-Macro BBN_KB_ENG_4 1 0.1190 0.1295 0.1171 LDC-MEAN-ALL-Macro BBN_KB_ENG_4 ALL 0.2076 0.1929 0.1852 *ALL-Macro Prec, Recall and F1 refer to mean-precision, mean-recall and mean-F1. NIL-DETECTION P/R/F1: 0.3085 0.9756 0.4688 ************************************************************* Run ID: BBN_KB_ENG_5 Did the run access the live Web during the evaluation window: No Did the run extract relations from the Cold Start source corpus: Yes Did the run generate meaningful confidence values: Yes Entity Discovery Evaluation: ONLY English documents: Prec Recall F1 Metric 0.888 0.673 0.765 strong_mention_match 0.845 0.640 0.729 strong_typed_mention_match 0.000 0.000 0.000 entity_match 0.789 0.500 0.612 b_cubed 0.795 0.603 0.686 mention_ceaf 0.774 0.586 0.667 typed_mention_ceaf Slot Filling Evaluation: Metric RunID Hop Prec Recall F1 SF-ALL-Micro BBN_KB_ENG_5 0 0.4165 0.2397 0.3043 SF-ALL-Micro BBN_KB_ENG_5 1 0.1178 0.1397 0.1278 SF-ALL-Micro BBN_KB_ENG_5 ALL 0.2635 0.2060 0.2312 SF-ALL-Macro BBN_KB_ENG_5 0 0.2524 0.2283 0.2226 SF-ALL-Macro BBN_KB_ENG_5 1 0.1171 0.1380 0.1168 SF-ALL-Macro BBN_KB_ENG_5 ALL 0.2010 0.1940 0.1824 LDC-MAX-ALL-Micro BBN_KB_ENG_5 0 0.4444 0.2614 0.3292 LDC-MAX-ALL-Micro BBN_KB_ENG_5 1 0.1170 0.1429 0.1287 LDC-MAX-ALL-Micro BBN_KB_ENG_5 ALL 0.2772 0.2217 0.2464 LDC-MAX-ALL-Macro BBN_KB_ENG_5 0 0.2855 0.2533 0.2471 LDC-MAX-ALL-Macro BBN_KB_ENG_5 1 0.1191 0.1416 0.1189 LDC-MAX-ALL-Macro BBN_KB_ENG_5 ALL 0.2202 0.2094 0.1968 LDC-MEAN-ALL-Macro BBN_KB_ENG_5 0 0.2610 0.2340 0.2261 LDC-MEAN-ALL-Macro BBN_KB_ENG_5 1 0.1129 0.1315 0.1127 LDC-MEAN-ALL-Macro BBN_KB_ENG_5 ALL 0.2029 0.1938 0.1816 *ALL-Macro Prec, Recall and F1 refer to mean-precision, mean-recall and mean-F1. NIL-DETECTION P/R/F1: 0.3103 0.9837 0.4718