======================================================= TAC KBP 2016 ENGLISH KB CONSTRUCTION EVALUATION RESULTS ======================================================= Team ID: hltcoe Organization: Human Language Technology Center of Excellence ************************************************************* Run ID: hltcoe_KB_ENG_1 Did the run access the live Web during the evaluation window: No Did the run extract relations from the Cold Start source corpus: Yes Did the run generate meaningful confidence values: No Entity Discovery Evaluation: ONLY English documents: Prec Recall F1 Metric 0.756 0.642 0.694 strong_mention_match 0.716 0.609 0.658 strong_typed_mention_match 0.000 0.000 0.000 entity_match 0.636 0.465 0.537 b_cubed 0.677 0.575 0.622 mention_ceaf 0.656 0.557 0.603 typed_mention_ceaf Slot Filling Evaluation: Metric RunID Hop Prec Recall F1 SF-ALL-Micro hltcoe_KB_ENG_1 0 0.3458 0.2422 0.2849 SF-ALL-Micro hltcoe_KB_ENG_1 1 0.0877 0.1152 0.0996 SF-ALL-Micro hltcoe_KB_ENG_1 ALL 0.2197 0.1993 0.2090 SF-ALL-Macro hltcoe_KB_ENG_1 0 0.2167 0.2186 0.2087 SF-ALL-Macro hltcoe_KB_ENG_1 1 0.1204 0.1435 0.1229 SF-ALL-Macro hltcoe_KB_ENG_1 ALL 0.1801 0.1900 0.1761 LDC-MAX-ALL-Micro hltcoe_KB_ENG_1 0 0.3449 0.2435 0.2854 LDC-MAX-ALL-Micro hltcoe_KB_ENG_1 1 0.0853 0.1039 0.0937 LDC-MAX-ALL-Micro hltcoe_KB_ENG_1 ALL 0.2243 0.1967 0.2096 LDC-MAX-ALL-Macro hltcoe_KB_ENG_1 0 0.2202 0.2188 0.2085 LDC-MAX-ALL-Macro hltcoe_KB_ENG_1 1 0.1061 0.1262 0.1074 LDC-MAX-ALL-Macro hltcoe_KB_ENG_1 ALL 0.1754 0.1824 0.1688 LDC-MEAN-ALL-Macro hltcoe_KB_ENG_1 0 0.2128 0.2127 0.2021 LDC-MEAN-ALL-Macro hltcoe_KB_ENG_1 1 0.1046 0.1231 0.1056 LDC-MEAN-ALL-Macro hltcoe_KB_ENG_1 ALL 0.1703 0.1775 0.1642 *ALL-Macro Prec, Recall and F1 refer to mean-precision, mean-recall and mean-F1. NIL-DETECTION P/R/F1: 0.3031 0.9512 0.4597 ************************************************************* Run ID: hltcoe_KB_ENG_2 Did the run access the live Web during the evaluation window: No Did the run extract relations from the Cold Start source corpus: Yes Did the run generate meaningful confidence values: No Entity Discovery Evaluation: ONLY English documents: Prec Recall F1 Metric 0.756 0.642 0.694 strong_mention_match 0.716 0.609 0.658 strong_typed_mention_match 0.000 0.000 0.000 entity_match 0.635 0.465 0.537 b_cubed 0.677 0.575 0.622 mention_ceaf 0.655 0.557 0.602 typed_mention_ceaf Slot Filling Evaluation: Metric RunID Hop Prec Recall F1 SF-ALL-Micro hltcoe_KB_ENG_2 0 0.3715 0.2310 0.2848 SF-ALL-Micro hltcoe_KB_ENG_2 1 0.1033 0.1005 0.1019 SF-ALL-Micro hltcoe_KB_ENG_2 ALL 0.2525 0.1869 0.2148 SF-ALL-Macro hltcoe_KB_ENG_2 0 0.2161 0.2052 0.2008 SF-ALL-Macro hltcoe_KB_ENG_2 1 0.1022 0.1178 0.1031 SF-ALL-Macro hltcoe_KB_ENG_2 ALL 0.1729 0.1720 0.1636 LDC-MAX-ALL-Micro hltcoe_KB_ENG_2 0 0.3625 0.2304 0.2817 LDC-MAX-ALL-Micro hltcoe_KB_ENG_2 1 0.0792 0.0877 0.0832 LDC-MAX-ALL-Micro hltcoe_KB_ENG_2 ALL 0.2301 0.1826 0.2036 LDC-MAX-ALL-Macro hltcoe_KB_ENG_2 0 0.2192 0.2042 0.1993 LDC-MAX-ALL-Macro hltcoe_KB_ENG_2 1 0.0835 0.0984 0.0837 LDC-MAX-ALL-Macro hltcoe_KB_ENG_2 ALL 0.1659 0.1626 0.1539 LDC-MEAN-ALL-Macro hltcoe_KB_ENG_2 0 0.2105 0.1988 0.1929 LDC-MEAN-ALL-Macro hltcoe_KB_ENG_2 1 0.0824 0.0959 0.0823 LDC-MEAN-ALL-Macro hltcoe_KB_ENG_2 ALL 0.1602 0.1584 0.1495 *ALL-Macro Prec, Recall and F1 refer to mean-precision, mean-recall and mean-F1. NIL-DETECTION P/R/F1: 0.3120 0.9919 0.4747 ************************************************************* Run ID: hltcoe_KB_ENG_3 Did the run access the live Web during the evaluation window: No Did the run extract relations from the Cold Start source corpus: Yes Did the run generate meaningful confidence values: No Entity Discovery Evaluation: ONLY English documents: Prec Recall F1 Metric 0.756 0.642 0.694 strong_mention_match 0.716 0.609 0.658 strong_typed_mention_match 0.000 0.000 0.000 entity_match 0.635 0.463 0.535 b_cubed 0.676 0.574 0.621 mention_ceaf 0.655 0.556 0.601 typed_mention_ceaf Slot Filling Evaluation: Metric RunID Hop Prec Recall F1 SF-ALL-Micro hltcoe_KB_ENG_3 0 0.3742 0.2322 0.2866 SF-ALL-Micro hltcoe_KB_ENG_3 1 0.1017 0.1005 0.1011 SF-ALL-Micro hltcoe_KB_ENG_3 ALL 0.2522 0.1878 0.2153 SF-ALL-Macro hltcoe_KB_ENG_3 0 0.2101 0.2092 0.1991 SF-ALL-Macro hltcoe_KB_ENG_3 1 0.1022 0.1178 0.1030 SF-ALL-Macro hltcoe_KB_ENG_3 ALL 0.1691 0.1745 0.1626 LDC-MAX-ALL-Micro hltcoe_KB_ENG_3 0 0.3675 0.2288 0.2820 LDC-MAX-ALL-Micro hltcoe_KB_ENG_3 1 0.1027 0.0877 0.0946 LDC-MAX-ALL-Micro hltcoe_KB_ENG_3 ALL 0.2593 0.1815 0.2136 LDC-MAX-ALL-Macro hltcoe_KB_ENG_3 0 0.2135 0.2063 0.1970 LDC-MAX-ALL-Macro hltcoe_KB_ENG_3 1 0.0835 0.0984 0.0836 LDC-MAX-ALL-Macro hltcoe_KB_ENG_3 ALL 0.1624 0.1639 0.1525 LDC-MEAN-ALL-Macro hltcoe_KB_ENG_3 0 0.2052 0.2020 0.1913 LDC-MEAN-ALL-Macro hltcoe_KB_ENG_3 1 0.0824 0.0959 0.0823 LDC-MEAN-ALL-Macro hltcoe_KB_ENG_3 ALL 0.1570 0.1603 0.1485 *ALL-Macro Prec, Recall and F1 refer to mean-precision, mean-recall and mean-F1. NIL-DETECTION P/R/F1: 0.3120 0.9919 0.4747 ************************************************************* Run ID: hltcoe_KB_ENG_4 Did the run access the live Web during the evaluation window: No Did the run extract relations from the Cold Start source corpus: Yes Did the run generate meaningful confidence values: No Entity Discovery Evaluation: ONLY English documents: Prec Recall F1 Metric 0.812 0.638 0.715 strong_mention_match 0.772 0.606 0.679 strong_typed_mention_match 0.000 0.000 0.000 entity_match 0.644 0.429 0.515 b_cubed 0.692 0.543 0.609 mention_ceaf 0.669 0.525 0.589 typed_mention_ceaf Slot Filling Evaluation: Metric RunID Hop Prec Recall F1 SF-ALL-Micro hltcoe_KB_ENG_4 0 0.3212 0.2322 0.2696 SF-ALL-Micro hltcoe_KB_ENG_4 1 0.0417 0.0980 0.0585 SF-ALL-Micro hltcoe_KB_ENG_4 ALL 0.1469 0.1869 0.1645 SF-ALL-Macro hltcoe_KB_ENG_4 0 0.2375 0.2280 0.2232 SF-ALL-Macro hltcoe_KB_ENG_4 1 0.1106 0.1339 0.1150 SF-ALL-Macro hltcoe_KB_ENG_4 ALL 0.1893 0.1923 0.1821 LDC-MAX-ALL-Micro hltcoe_KB_ENG_4 0 0.3118 0.2288 0.2639 LDC-MAX-ALL-Micro hltcoe_KB_ENG_4 1 0.0424 0.0877 0.0571 LDC-MAX-ALL-Micro hltcoe_KB_ENG_4 ALL 0.1538 0.1815 0.1665 LDC-MAX-ALL-Macro hltcoe_KB_ENG_4 0 0.2312 0.2215 0.2151 LDC-MAX-ALL-Macro hltcoe_KB_ENG_4 1 0.0992 0.1183 0.1017 LDC-MAX-ALL-Macro hltcoe_KB_ENG_4 ALL 0.1794 0.1810 0.1706 LDC-MEAN-ALL-Macro hltcoe_KB_ENG_4 0 0.2223 0.2122 0.2064 LDC-MEAN-ALL-Macro hltcoe_KB_ENG_4 1 0.0980 0.1167 0.1003 LDC-MEAN-ALL-Macro hltcoe_KB_ENG_4 ALL 0.1735 0.1747 0.1647 *ALL-Macro Prec, Recall and F1 refer to mean-precision, mean-recall and mean-F1. NIL-DETECTION P/R/F1: 0.3013 0.9431 0.4567 ************************************************************* Run ID: hltcoe_KB_ENG_5 Did the run access the live Web during the evaluation window: No Did the run extract relations from the Cold Start source corpus: Yes Did the run generate meaningful confidence values: No Entity Discovery Evaluation: ONLY English documents: Prec Recall F1 Metric 0.756 0.642 0.694 strong_mention_match 0.716 0.609 0.658 strong_typed_mention_match 0.000 0.000 0.000 entity_match 0.636 0.465 0.537 b_cubed 0.677 0.575 0.622 mention_ceaf 0.656 0.557 0.603 typed_mention_ceaf Slot Filling Evaluation: Metric RunID Hop Prec Recall F1 SF-ALL-Micro hltcoe_KB_ENG_5 0 0.3476 0.2434 0.2863 SF-ALL-Micro hltcoe_KB_ENG_5 1 0.0877 0.1152 0.0996 SF-ALL-Micro hltcoe_KB_ENG_5 ALL 0.2206 0.2002 0.2099 SF-ALL-Macro hltcoe_KB_ENG_5 0 0.2198 0.2216 0.2118 SF-ALL-Macro hltcoe_KB_ENG_5 1 0.1204 0.1435 0.1229 SF-ALL-Macro hltcoe_KB_ENG_5 ALL 0.1820 0.1919 0.1780 LDC-MAX-ALL-Micro hltcoe_KB_ENG_5 0 0.3488 0.2451 0.2879 LDC-MAX-ALL-Micro hltcoe_KB_ENG_5 1 0.0853 0.1039 0.0937 LDC-MAX-ALL-Micro hltcoe_KB_ENG_5 ALL 0.2261 0.1978 0.2110 LDC-MAX-ALL-Macro hltcoe_KB_ENG_5 0 0.2245 0.2231 0.2128 LDC-MAX-ALL-Macro hltcoe_KB_ENG_5 1 0.1061 0.1262 0.1074 LDC-MAX-ALL-Macro hltcoe_KB_ENG_5 ALL 0.1781 0.1851 0.1714 LDC-MEAN-ALL-Macro hltcoe_KB_ENG_5 0 0.2150 0.2149 0.2043 LDC-MEAN-ALL-Macro hltcoe_KB_ENG_5 1 0.1046 0.1231 0.1056 LDC-MEAN-ALL-Macro hltcoe_KB_ENG_5 ALL 0.1716 0.1788 0.1655 *ALL-Macro Prec, Recall and F1 refer to mean-precision, mean-recall and mean-F1. NIL-DETECTION P/R/F1: 0.3031 0.9512 0.4597