===================================================================== TAC KBP 2017 ENGLISH KB CONSTRUCTION: COMPOSITE KB EVALUATION RESULTS ===================================================================== Team ID: hltcoe Organization: Human Language Technology Center of Excellence ************************************************************* Run ID: hltcoe_KB_ENG_1 Did the run access the live Web during the evaluation window: No Did the run extract relations from the Cold Start source corpus: Yes Did the run generate meaningful confidence values: Yes Did the run attempt the full Entity Discovery and Linking task: Yes Did the run attempt the full Event Nugget Detection and Coreference task: No Did the run include SF relations: Yes Did the run attempt the Event Argument Extraction and Linking task: No Did the run include Sentiment relations: No --------------------- Composite KB Evaluation (All slot types): Scores based on confidence values and Average Precision (AP); k=3 justifications allowed: Metric RunID Hop AP SF-ALL-Macro hltcoe_KB_ENG_1 0 0.0873 SF-ALL-Macro hltcoe_KB_ENG_1 1 0.0097 SF-ALL-Macro hltcoe_KB_ENG_1 ALL 0.0698 LDC-MEAN-ALL-Macro hltcoe_KB_ENG_1 0 0.1040 LDC-MEAN-ALL-Macro hltcoe_KB_ENG_1 1 0.0107 LDC-MEAN-ALL-Macro hltcoe_KB_ENG_1 ALL 0.0847 (PRIMARY METRIC) *ALL-Macro AP refer to mean of corresponding AP values. Scores based on confidence values and Average Precision (AP); only k=1 justification allowed: Metric RunID Hop AP SF-ALL-Macro hltcoe_KB_ENG_1 0 0.0941 SF-ALL-Macro hltcoe_KB_ENG_1 1 0.0097 SF-ALL-Macro hltcoe_KB_ENG_1 ALL 0.0773 LDC-MEAN-ALL-Macro hltcoe_KB_ENG_1 0 0.1137 LDC-MEAN-ALL-Macro hltcoe_KB_ENG_1 1 0.0131 LDC-MEAN-ALL-Macro hltcoe_KB_ENG_1 ALL 0.0939 *ALL-Macro AP refer to mean of corresponding AP values. Scores based on P/R/F1; only k=1 justification allowed: Metric RunID Hop Prec Recall F1 SF-ALL-Micro hltcoe_KB_ENG_1 0 0.2175 0.1031 0.1399 SF-ALL-Micro hltcoe_KB_ENG_1 1 0.0317 0.0712 0.0438 SF-ALL-Micro hltcoe_KB_ENG_1 ALL 0.1092 0.0959 0.1021 SF-ALL-Macro hltcoe_KB_ENG_1 0 0.0943 0.1040 0.0912 SF-ALL-Macro hltcoe_KB_ENG_1 1 0.0709 0.0716 0.0677 SF-ALL-Macro hltcoe_KB_ENG_1 ALL 0.0862 0.0928 0.0831 LDC-MAX-ALL-Micro hltcoe_KB_ENG_1 0 0.2681 0.1318 0.1767 LDC-MAX-ALL-Micro hltcoe_KB_ENG_1 1 0.0326 0.0899 0.0478 LDC-MAX-ALL-Micro hltcoe_KB_ENG_1 ALL 0.1250 0.1228 0.1239 LDC-MAX-ALL-Macro hltcoe_KB_ENG_1 0 0.1244 0.1399 0.1209 LDC-MAX-ALL-Macro hltcoe_KB_ENG_1 1 0.0925 0.0918 0.0877 LDC-MAX-ALL-Macro hltcoe_KB_ENG_1 ALL 0.1135 0.1235 0.1096 LDC-MEAN-ALL-Macro hltcoe_KB_ENG_1 0 0.1089 0.1258 0.1071 LDC-MEAN-ALL-Macro hltcoe_KB_ENG_1 1 0.0885 0.0885 0.0843 LDC-MEAN-ALL-Macro hltcoe_KB_ENG_1 ALL 0.1019 0.1131 0.0993 *ALL-Macro Prec, Recall and F1 refer to mean-precision, mean-recall and mean-F1. --------------------- Composite KB Evaluation (SF-SLOTS only): Scores based on confidence values and Average Precision (AP); k=3 justifications allowed: Metric RunID Hop AP SF-ALL-Macro hltcoe_KB_ENG_1 0 0.2015 SF-ALL-Macro hltcoe_KB_ENG_1 1 0.0271 SF-ALL-Macro hltcoe_KB_ENG_1 ALL 0.1610 LDC-MEAN-ALL-Macro hltcoe_KB_ENG_1 0 0.2242 LDC-MEAN-ALL-Macro hltcoe_KB_ENG_1 1 0.0282 LDC-MEAN-ALL-Macro hltcoe_KB_ENG_1 ALL 0.1827 (PRIMARY METRIC) *ALL-Macro AP refer to mean of corresponding AP values. Scores based on confidence values and Average Precision (AP); only k=1 justification allowed: Metric RunID Hop AP SF-ALL-Macro hltcoe_KB_ENG_1 0 0.2171 SF-ALL-Macro hltcoe_KB_ENG_1 1 0.0272 SF-ALL-Macro hltcoe_KB_ENG_1 ALL 0.1785 LDC-MEAN-ALL-Macro hltcoe_KB_ENG_1 0 0.2452 LDC-MEAN-ALL-Macro hltcoe_KB_ENG_1 1 0.0345 LDC-MEAN-ALL-Macro hltcoe_KB_ENG_1 ALL 0.2024 *ALL-Macro AP refer to mean of corresponding AP values. Scores based on P/R/F1; only k=1 justification allowed: Metric RunID Hop Prec Recall F1 SF-ALL-Micro hltcoe_KB_ENG_1 0 0.2175 0.2081 0.2127 SF-ALL-Micro hltcoe_KB_ENG_1 1 0.0317 0.1339 0.0512 SF-ALL-Micro hltcoe_KB_ENG_1 ALL 0.1092 0.1903 0.1388 SF-ALL-Macro hltcoe_KB_ENG_1 0 0.2176 0.2400 0.2105 SF-ALL-Macro hltcoe_KB_ENG_1 1 0.1307 0.1320 0.1247 SF-ALL-Macro hltcoe_KB_ENG_1 ALL 0.1831 0.1972 0.1765 LDC-MAX-ALL-Micro hltcoe_KB_ENG_1 0 0.2737 0.2538 0.2634 LDC-MAX-ALL-Micro hltcoe_KB_ENG_1 1 0.0326 0.1582 0.0541 LDC-MAX-ALL-Micro hltcoe_KB_ENG_1 ALL 0.1261 0.2317 0.1633 LDC-MAX-ALL-Macro hltcoe_KB_ENG_1 0 0.2681 0.3016 0.2606 LDC-MAX-ALL-Macro hltcoe_KB_ENG_1 1 0.1631 0.1619 0.1548 LDC-MAX-ALL-Macro hltcoe_KB_ENG_1 ALL 0.2273 0.2474 0.2195 LDC-MEAN-ALL-Macro hltcoe_KB_ENG_1 0 0.2347 0.2712 0.2308 LDC-MEAN-ALL-Macro hltcoe_KB_ENG_1 1 0.1562 0.1562 0.1488 LDC-MEAN-ALL-Macro hltcoe_KB_ENG_1 ALL 0.2042 0.2266 0.1990 *ALL-Macro Prec, Recall and F1 refer to mean-precision, mean-recall and mean-F1. --------------------- Composite KB Evaluation (EVENT-SLOTS only): Scores based on confidence values and Average Precision (AP); k=3 justifications allowed: SUMMARY: This section provides summary of AP scores Metric RunID Hop AP SF-ALL-Macro hltcoe_KB_ENG_1 0 0.0000 SF-ALL-Macro hltcoe_KB_ENG_1 ALL 0.0000 LDC-MEAN-ALL-Macro hltcoe_KB_ENG_1 0 0.0000 LDC-MEAN-ALL-Macro hltcoe_KB_ENG_1 ALL 0.0000 (PRIMARY METRIC) *ALL-Macro AP refer to mean of corresponding AP values. Scores based on confidence values and Average Precision (AP); only k=1 justification allowed: SUMMARY: This section provides summary of AP scores Metric RunID Hop AP SF-ALL-Macro hltcoe_KB_ENG_1 0 0.0000 SF-ALL-Macro hltcoe_KB_ENG_1 ALL 0.0000 LDC-MEAN-ALL-Macro hltcoe_KB_ENG_1 0 0.0000 LDC-MEAN-ALL-Macro hltcoe_KB_ENG_1 ALL 0.0000 *ALL-Macro AP refer to mean of corresponding AP values. Scores based on P/R/F1; only k=1 justification allowed: Metric RunID Hop Prec Recall F1 SF-ALL-Micro hltcoe_KB_ENG_1 0 0.0000 0.0000 0.0000 SF-ALL-Micro hltcoe_KB_ENG_1 ALL 0.0000 0.0000 0.0000 SF-ALL-Macro hltcoe_KB_ENG_1 0 0.0000 0.0000 0.0000 SF-ALL-Macro hltcoe_KB_ENG_1 ALL 0.0000 0.0000 0.0000 LDC-MAX-ALL-Micro hltcoe_KB_ENG_1 0 0.0000 0.0000 0.0000 LDC-MAX-ALL-Micro hltcoe_KB_ENG_1 ALL 0.0000 0.0000 0.0000 LDC-MAX-ALL-Macro hltcoe_KB_ENG_1 0 0.0000 0.0000 0.0000 LDC-MAX-ALL-Macro hltcoe_KB_ENG_1 ALL 0.0000 0.0000 0.0000 LDC-MEAN-ALL-Macro hltcoe_KB_ENG_1 0 0.0000 0.0000 0.0000 LDC-MEAN-ALL-Macro hltcoe_KB_ENG_1 ALL 0.0000 0.0000 0.0000 *ALL-Macro Prec, Recall and F1 refer to mean-precision, mean-recall and mean-F1. --------------------- Composite KB Evaluation (SENTIMENT-SLOTS only): Scores based on confidence values and Average Precision (AP); k=3 justifications allowed: Metric RunID Hop AP SF-ALL-Macro hltcoe_KB_ENG_1 0 0.0000 SF-ALL-Macro hltcoe_KB_ENG_1 1 0.0000 SF-ALL-Macro hltcoe_KB_ENG_1 ALL 0.0000 LDC-MEAN-ALL-Macro hltcoe_KB_ENG_1 0 0.0000 LDC-MEAN-ALL-Macro hltcoe_KB_ENG_1 1 0.0000 LDC-MEAN-ALL-Macro hltcoe_KB_ENG_1 ALL 0.0000 (PRIMARY METRIC) *ALL-Macro AP refer to mean of corresponding AP values. Scores based on confidence values and Average Precision (AP); only k=1 justification allowed: Metric RunID Hop AP SF-ALL-Macro hltcoe_KB_ENG_1 0 0.0000 SF-ALL-Macro hltcoe_KB_ENG_1 1 0.0000 SF-ALL-Macro hltcoe_KB_ENG_1 ALL 0.0000 LDC-MEAN-ALL-Macro hltcoe_KB_ENG_1 0 0.0000 LDC-MEAN-ALL-Macro hltcoe_KB_ENG_1 1 0.0000 LDC-MEAN-ALL-Macro hltcoe_KB_ENG_1 ALL 0.0000 *ALL-Macro AP refer to mean of corresponding AP values. Scores based on P/R/F1; only k=1 justification allowed: Metric RunID Hop Prec Recall F1 SF-ALL-Micro hltcoe_KB_ENG_1 0 0.0000 0.0000 0.0000 SF-ALL-Micro hltcoe_KB_ENG_1 1 0.0000 0.0000 0.0000 SF-ALL-Micro hltcoe_KB_ENG_1 ALL 0.0000 0.0000 0.0000 SF-ALL-Macro hltcoe_KB_ENG_1 0 0.0000 0.0000 0.0000 SF-ALL-Macro hltcoe_KB_ENG_1 1 0.0000 0.0000 0.0000 SF-ALL-Macro hltcoe_KB_ENG_1 ALL 0.0000 0.0000 0.0000 LDC-MAX-ALL-Micro hltcoe_KB_ENG_1 0 0.0000 0.0000 0.0000 LDC-MAX-ALL-Micro hltcoe_KB_ENG_1 1 0.0000 0.0000 0.0000 LDC-MAX-ALL-Micro hltcoe_KB_ENG_1 ALL 0.0000 0.0000 0.0000 LDC-MAX-ALL-Macro hltcoe_KB_ENG_1 0 0.0000 0.0000 0.0000 LDC-MAX-ALL-Macro hltcoe_KB_ENG_1 1 0.0000 0.0000 0.0000 LDC-MAX-ALL-Macro hltcoe_KB_ENG_1 ALL 0.0000 0.0000 0.0000 LDC-MEAN-ALL-Macro hltcoe_KB_ENG_1 0 0.0000 0.0000 0.0000 LDC-MEAN-ALL-Macro hltcoe_KB_ENG_1 1 0.0000 0.0000 0.0000 LDC-MEAN-ALL-Macro hltcoe_KB_ENG_1 ALL 0.0000 0.0000 0.0000 *ALL-Macro Prec, Recall and F1 refer to mean-precision, mean-recall and mean-F1. ************************************************************* Run ID: hltcoe_KB_ENG_2 Did the run access the live Web during the evaluation window: No Did the run extract relations from the Cold Start source corpus: Yes Did the run generate meaningful confidence values: Yes Did the run attempt the full Entity Discovery and Linking task: Yes Did the run attempt the full Event Nugget Detection and Coreference task: No Did the run include SF relations: Yes Did the run attempt the Event Argument Extraction and Linking task: No Did the run include Sentiment relations: No --------------------- Composite KB Evaluation (All slot types): Scores based on confidence values and Average Precision (AP); k=3 justifications allowed: Metric RunID Hop AP SF-ALL-Macro hltcoe_KB_ENG_2 0 0.0876 SF-ALL-Macro hltcoe_KB_ENG_2 1 0.0082 SF-ALL-Macro hltcoe_KB_ENG_2 ALL 0.0699 LDC-MEAN-ALL-Macro hltcoe_KB_ENG_2 0 0.1038 LDC-MEAN-ALL-Macro hltcoe_KB_ENG_2 1 0.0099 LDC-MEAN-ALL-Macro hltcoe_KB_ENG_2 ALL 0.0845 (PRIMARY METRIC) *ALL-Macro AP refer to mean of corresponding AP values. Scores based on confidence values and Average Precision (AP); only k=1 justification allowed: Metric RunID Hop AP SF-ALL-Macro hltcoe_KB_ENG_2 0 0.0963 SF-ALL-Macro hltcoe_KB_ENG_2 1 0.0093 SF-ALL-Macro hltcoe_KB_ENG_2 ALL 0.0796 LDC-MEAN-ALL-Macro hltcoe_KB_ENG_2 0 0.1153 LDC-MEAN-ALL-Macro hltcoe_KB_ENG_2 1 0.0131 LDC-MEAN-ALL-Macro hltcoe_KB_ENG_2 ALL 0.0955 *ALL-Macro AP refer to mean of corresponding AP values. Scores based on P/R/F1; only k=1 justification allowed: Metric RunID Hop Prec Recall F1 SF-ALL-Micro hltcoe_KB_ENG_2 0 0.3305 0.1006 0.1542 SF-ALL-Micro hltcoe_KB_ENG_2 1 0.0296 0.0683 0.0413 SF-ALL-Micro hltcoe_KB_ENG_2 ALL 0.1226 0.0932 0.1059 SF-ALL-Macro hltcoe_KB_ENG_2 0 0.0957 0.1042 0.0933 SF-ALL-Macro hltcoe_KB_ENG_2 1 0.0670 0.0685 0.0644 SF-ALL-Macro hltcoe_KB_ENG_2 ALL 0.0858 0.0919 0.0833 LDC-MAX-ALL-Micro hltcoe_KB_ENG_2 0 0.3447 0.1298 0.1886 LDC-MAX-ALL-Micro hltcoe_KB_ENG_2 1 0.1029 0.0899 0.0960 LDC-MAX-ALL-Micro hltcoe_KB_ENG_2 ALL 0.2504 0.1212 0.1634 LDC-MAX-ALL-Macro hltcoe_KB_ENG_2 0 0.1279 0.1426 0.1246 LDC-MAX-ALL-Macro hltcoe_KB_ENG_2 1 0.0927 0.0918 0.0879 LDC-MAX-ALL-Macro hltcoe_KB_ENG_2 ALL 0.1159 0.1252 0.1121 LDC-MEAN-ALL-Macro hltcoe_KB_ENG_2 0 0.1095 0.1259 0.1083 LDC-MEAN-ALL-Macro hltcoe_KB_ENG_2 1 0.0852 0.0859 0.0815 LDC-MEAN-ALL-Macro hltcoe_KB_ENG_2 ALL 0.1012 0.1122 0.0991 *ALL-Macro Prec, Recall and F1 refer to mean-precision, mean-recall and mean-F1. --------------------- Composite KB Evaluation (SF-SLOTS only): Scores based on confidence values and Average Precision (AP); k=3 justifications allowed: Metric RunID Hop AP SF-ALL-Macro hltcoe_KB_ENG_2 0 0.2022 SF-ALL-Macro hltcoe_KB_ENG_2 1 0.0231 SF-ALL-Macro hltcoe_KB_ENG_2 ALL 0.1614 LDC-MEAN-ALL-Macro hltcoe_KB_ENG_2 0 0.2238 LDC-MEAN-ALL-Macro hltcoe_KB_ENG_2 1 0.0262 LDC-MEAN-ALL-Macro hltcoe_KB_ENG_2 ALL 0.1822 (PRIMARY METRIC) *ALL-Macro AP refer to mean of corresponding AP values. Scores based on confidence values and Average Precision (AP); only k=1 justification allowed: Metric RunID Hop AP SF-ALL-Macro hltcoe_KB_ENG_2 0 0.2224 SF-ALL-Macro hltcoe_KB_ENG_2 1 0.0261 SF-ALL-Macro hltcoe_KB_ENG_2 ALL 0.1838 LDC-MEAN-ALL-Macro hltcoe_KB_ENG_2 0 0.2485 LDC-MEAN-ALL-Macro hltcoe_KB_ENG_2 1 0.0346 LDC-MEAN-ALL-Macro hltcoe_KB_ENG_2 ALL 0.2058 *ALL-Macro AP refer to mean of corresponding AP values. Scores based on P/R/F1; only k=1 justification allowed: Metric RunID Hop Prec Recall F1 SF-ALL-Micro hltcoe_KB_ENG_2 0 0.3305 0.2029 0.2515 SF-ALL-Micro hltcoe_KB_ENG_2 1 0.0296 0.1284 0.0481 SF-ALL-Micro hltcoe_KB_ENG_2 ALL 0.1226 0.1850 0.1475 SF-ALL-Macro hltcoe_KB_ENG_2 0 0.2208 0.2406 0.2153 SF-ALL-Macro hltcoe_KB_ENG_2 1 0.1234 0.1263 0.1186 SF-ALL-Macro hltcoe_KB_ENG_2 ALL 0.1822 0.1953 0.1770 LDC-MAX-ALL-Micro hltcoe_KB_ENG_2 0 0.3385 0.2500 0.2876 LDC-MAX-ALL-Micro hltcoe_KB_ENG_2 1 0.0317 0.1582 0.0529 LDC-MAX-ALL-Micro hltcoe_KB_ENG_2 ALL 0.1328 0.2287 0.1680 LDC-MAX-ALL-Macro hltcoe_KB_ENG_2 0 0.2758 0.3073 0.2686 LDC-MAX-ALL-Macro hltcoe_KB_ENG_2 1 0.1636 0.1619 0.1551 LDC-MAX-ALL-Macro hltcoe_KB_ENG_2 ALL 0.2322 0.2509 0.2245 LDC-MEAN-ALL-Macro hltcoe_KB_ENG_2 0 0.2361 0.2714 0.2335 LDC-MEAN-ALL-Macro hltcoe_KB_ENG_2 1 0.1502 0.1515 0.1437 LDC-MEAN-ALL-Macro hltcoe_KB_ENG_2 ALL 0.2028 0.2248 0.1987 *ALL-Macro Prec, Recall and F1 refer to mean-precision, mean-recall and mean-F1. --------------------- Composite KB Evaluation (EVENT-SLOTS only): Scores based on confidence values and Average Precision (AP); k=3 justifications allowed: SUMMARY: This section provides summary of AP scores Metric RunID Hop AP SF-ALL-Macro hltcoe_KB_ENG_2 0 0.0000 SF-ALL-Macro hltcoe_KB_ENG_2 ALL 0.0000 LDC-MEAN-ALL-Macro hltcoe_KB_ENG_2 0 0.0000 LDC-MEAN-ALL-Macro hltcoe_KB_ENG_2 ALL 0.0000 (PRIMARY METRIC) *ALL-Macro AP refer to mean of corresponding AP values. Scores based on confidence values and Average Precision (AP); only k=1 justification allowed: SUMMARY: This section provides summary of AP scores Metric RunID Hop AP SF-ALL-Macro hltcoe_KB_ENG_2 0 0.0000 SF-ALL-Macro hltcoe_KB_ENG_2 ALL 0.0000 LDC-MEAN-ALL-Macro hltcoe_KB_ENG_2 0 0.0000 LDC-MEAN-ALL-Macro hltcoe_KB_ENG_2 ALL 0.0000 *ALL-Macro AP refer to mean of corresponding AP values. Scores based on P/R/F1; only k=1 justification allowed: Metric RunID Hop Prec Recall F1 SF-ALL-Micro hltcoe_KB_ENG_2 0 0.0000 0.0000 0.0000 SF-ALL-Micro hltcoe_KB_ENG_2 ALL 0.0000 0.0000 0.0000 SF-ALL-Macro hltcoe_KB_ENG_2 0 0.0000 0.0000 0.0000 SF-ALL-Macro hltcoe_KB_ENG_2 ALL 0.0000 0.0000 0.0000 LDC-MAX-ALL-Micro hltcoe_KB_ENG_2 0 0.0000 0.0000 0.0000 LDC-MAX-ALL-Micro hltcoe_KB_ENG_2 ALL 0.0000 0.0000 0.0000 LDC-MAX-ALL-Macro hltcoe_KB_ENG_2 0 0.0000 0.0000 0.0000 LDC-MAX-ALL-Macro hltcoe_KB_ENG_2 ALL 0.0000 0.0000 0.0000 LDC-MEAN-ALL-Macro hltcoe_KB_ENG_2 0 0.0000 0.0000 0.0000 LDC-MEAN-ALL-Macro hltcoe_KB_ENG_2 ALL 0.0000 0.0000 0.0000 *ALL-Macro Prec, Recall and F1 refer to mean-precision, mean-recall and mean-F1. --------------------- Composite KB Evaluation (SENTIMENT-SLOTS only): Scores based on confidence values and Average Precision (AP); k=3 justifications allowed: Metric RunID Hop AP SF-ALL-Macro hltcoe_KB_ENG_2 0 0.0000 SF-ALL-Macro hltcoe_KB_ENG_2 1 0.0000 SF-ALL-Macro hltcoe_KB_ENG_2 ALL 0.0000 LDC-MEAN-ALL-Macro hltcoe_KB_ENG_2 0 0.0000 LDC-MEAN-ALL-Macro hltcoe_KB_ENG_2 1 0.0000 LDC-MEAN-ALL-Macro hltcoe_KB_ENG_2 ALL 0.0000 (PRIMARY METRIC) *ALL-Macro AP refer to mean of corresponding AP values. Scores based on confidence values and Average Precision (AP); only k=1 justification allowed: Metric RunID Hop AP SF-ALL-Macro hltcoe_KB_ENG_2 0 0.0000 SF-ALL-Macro hltcoe_KB_ENG_2 1 0.0000 SF-ALL-Macro hltcoe_KB_ENG_2 ALL 0.0000 LDC-MEAN-ALL-Macro hltcoe_KB_ENG_2 0 0.0000 LDC-MEAN-ALL-Macro hltcoe_KB_ENG_2 1 0.0000 LDC-MEAN-ALL-Macro hltcoe_KB_ENG_2 ALL 0.0000 *ALL-Macro AP refer to mean of corresponding AP values. Scores based on P/R/F1; only k=1 justification allowed: Metric RunID Hop Prec Recall F1 SF-ALL-Micro hltcoe_KB_ENG_2 0 0.0000 0.0000 0.0000 SF-ALL-Micro hltcoe_KB_ENG_2 1 0.0000 0.0000 0.0000 SF-ALL-Micro hltcoe_KB_ENG_2 ALL 0.0000 0.0000 0.0000 SF-ALL-Macro hltcoe_KB_ENG_2 0 0.0000 0.0000 0.0000 SF-ALL-Macro hltcoe_KB_ENG_2 1 0.0000 0.0000 0.0000 SF-ALL-Macro hltcoe_KB_ENG_2 ALL 0.0000 0.0000 0.0000 LDC-MAX-ALL-Micro hltcoe_KB_ENG_2 0 0.0000 0.0000 0.0000 LDC-MAX-ALL-Micro hltcoe_KB_ENG_2 1 0.0000 0.0000 0.0000 LDC-MAX-ALL-Micro hltcoe_KB_ENG_2 ALL 0.0000 0.0000 0.0000 LDC-MAX-ALL-Macro hltcoe_KB_ENG_2 0 0.0000 0.0000 0.0000 LDC-MAX-ALL-Macro hltcoe_KB_ENG_2 1 0.0000 0.0000 0.0000 LDC-MAX-ALL-Macro hltcoe_KB_ENG_2 ALL 0.0000 0.0000 0.0000 LDC-MEAN-ALL-Macro hltcoe_KB_ENG_2 0 0.0000 0.0000 0.0000 LDC-MEAN-ALL-Macro hltcoe_KB_ENG_2 1 0.0000 0.0000 0.0000 LDC-MEAN-ALL-Macro hltcoe_KB_ENG_2 ALL 0.0000 0.0000 0.0000 *ALL-Macro Prec, Recall and F1 refer to mean-precision, mean-recall and mean-F1. ************************************************************* Run ID: hltcoe_KB_ENG_3 Did the run access the live Web during the evaluation window: No Did the run extract relations from the Cold Start source corpus: Yes Did the run generate meaningful confidence values: Yes Did the run attempt the full Entity Discovery and Linking task: Yes Did the run attempt the full Event Nugget Detection and Coreference task: No Did the run include SF relations: Yes Did the run attempt the Event Argument Extraction and Linking task: No Did the run include Sentiment relations: No --------------------- Composite KB Evaluation (All slot types): Scores based on confidence values and Average Precision (AP); k=3 justifications allowed: Metric RunID Hop AP SF-ALL-Macro hltcoe_KB_ENG_3 0 0.0661 SF-ALL-Macro hltcoe_KB_ENG_3 1 0.0035 SF-ALL-Macro hltcoe_KB_ENG_3 ALL 0.0567 LDC-MEAN-ALL-Macro hltcoe_KB_ENG_3 0 0.0796 LDC-MEAN-ALL-Macro hltcoe_KB_ENG_3 1 0.0051 LDC-MEAN-ALL-Macro hltcoe_KB_ENG_3 ALL 0.0688 (PRIMARY METRIC) *ALL-Macro AP refer to mean of corresponding AP values. Scores based on confidence values and Average Precision (AP); only k=1 justification allowed: Metric RunID Hop AP SF-ALL-Macro hltcoe_KB_ENG_3 0 0.0756 SF-ALL-Macro hltcoe_KB_ENG_3 1 0.0044 SF-ALL-Macro hltcoe_KB_ENG_3 ALL 0.0641 LDC-MEAN-ALL-Macro hltcoe_KB_ENG_3 0 0.0909 LDC-MEAN-ALL-Macro hltcoe_KB_ENG_3 1 0.0077 LDC-MEAN-ALL-Macro hltcoe_KB_ENG_3 ALL 0.0773 *ALL-Macro AP refer to mean of corresponding AP values. Scores based on P/R/F1; only k=1 justification allowed: Metric RunID Hop Prec Recall F1 SF-ALL-Micro hltcoe_KB_ENG_3 0 0.3344 0.0860 0.1368 SF-ALL-Micro hltcoe_KB_ENG_3 1 0.2143 0.0480 0.0784 SF-ALL-Micro hltcoe_KB_ENG_3 ALL 0.3099 0.0774 0.1238 SF-ALL-Macro hltcoe_KB_ENG_3 0 0.0802 0.0840 0.0766 SF-ALL-Macro hltcoe_KB_ENG_3 1 0.0538 0.0469 0.0484 SF-ALL-Macro hltcoe_KB_ENG_3 ALL 0.0711 0.0712 0.0669 LDC-MAX-ALL-Micro hltcoe_KB_ENG_3 0 0.3802 0.1179 0.1800 LDC-MAX-ALL-Micro hltcoe_KB_ENG_3 1 0.2093 0.0647 0.0989 LDC-MAX-ALL-Micro hltcoe_KB_ENG_3 ALL 0.3434 0.1064 0.1625 LDC-MAX-ALL-Macro hltcoe_KB_ENG_3 0 0.1104 0.1163 0.1039 LDC-MAX-ALL-Macro hltcoe_KB_ENG_3 1 0.0775 0.0673 0.0696 LDC-MAX-ALL-Macro hltcoe_KB_ENG_3 ALL 0.0991 0.0996 0.0922 LDC-MEAN-ALL-Macro hltcoe_KB_ENG_3 0 0.0928 0.1024 0.0897 LDC-MEAN-ALL-Macro hltcoe_KB_ENG_3 1 0.0704 0.0619 0.0637 LDC-MEAN-ALL-Macro hltcoe_KB_ENG_3 ALL 0.0852 0.0885 0.0808 *ALL-Macro Prec, Recall and F1 refer to mean-precision, mean-recall and mean-F1. --------------------- Composite KB Evaluation (SF-SLOTS only): Scores based on confidence values and Average Precision (AP); k=3 justifications allowed: Metric RunID Hop AP SF-ALL-Macro hltcoe_KB_ENG_3 0 0.1525 SF-ALL-Macro hltcoe_KB_ENG_3 1 0.0098 SF-ALL-Macro hltcoe_KB_ENG_3 ALL 0.1310 LDC-MEAN-ALL-Macro hltcoe_KB_ENG_3 0 0.1716 LDC-MEAN-ALL-Macro hltcoe_KB_ENG_3 1 0.0133 LDC-MEAN-ALL-Macro hltcoe_KB_ENG_3 ALL 0.1483 (PRIMARY METRIC) *ALL-Macro AP refer to mean of corresponding AP values. Scores based on confidence values and Average Precision (AP); only k=1 justification allowed: Metric RunID Hop AP SF-ALL-Macro hltcoe_KB_ENG_3 0 0.1744 SF-ALL-Macro hltcoe_KB_ENG_3 1 0.0124 SF-ALL-Macro hltcoe_KB_ENG_3 ALL 0.1481 LDC-MEAN-ALL-Macro hltcoe_KB_ENG_3 0 0.1959 LDC-MEAN-ALL-Macro hltcoe_KB_ENG_3 1 0.0203 LDC-MEAN-ALL-Macro hltcoe_KB_ENG_3 ALL 0.1667 *ALL-Macro AP refer to mean of corresponding AP values. Scores based on P/R/F1; only k=1 justification allowed: Metric RunID Hop Prec Recall F1 SF-ALL-Micro hltcoe_KB_ENG_3 0 0.3344 0.1736 0.2285 SF-ALL-Micro hltcoe_KB_ENG_3 1 0.2143 0.0902 0.1269 SF-ALL-Micro hltcoe_KB_ENG_3 ALL 0.3099 0.1535 0.2054 SF-ALL-Macro hltcoe_KB_ENG_3 0 0.1850 0.1939 0.1767 SF-ALL-Macro hltcoe_KB_ENG_3 1 0.0992 0.0865 0.0892 SF-ALL-Macro hltcoe_KB_ENG_3 ALL 0.1510 0.1513 0.1420 LDC-MAX-ALL-Micro hltcoe_KB_ENG_3 0 0.3778 0.2271 0.2837 LDC-MAX-ALL-Micro hltcoe_KB_ENG_3 1 0.2093 0.1139 0.1475 LDC-MAX-ALL-Micro hltcoe_KB_ENG_3 ALL 0.3416 0.2009 0.2530 LDC-MAX-ALL-Macro hltcoe_KB_ENG_3 0 0.2379 0.2508 0.2240 LDC-MAX-ALL-Macro hltcoe_KB_ENG_3 1 0.1368 0.1187 0.1229 LDC-MAX-ALL-Macro hltcoe_KB_ENG_3 ALL 0.1986 0.1995 0.1847 LDC-MEAN-ALL-Macro hltcoe_KB_ENG_3 0 0.2001 0.2207 0.1933 LDC-MEAN-ALL-Macro hltcoe_KB_ENG_3 1 0.1242 0.1093 0.1124 LDC-MEAN-ALL-Macro hltcoe_KB_ENG_3 ALL 0.1707 0.1774 0.1619 *ALL-Macro Prec, Recall and F1 refer to mean-precision, mean-recall and mean-F1. --------------------- Composite KB Evaluation (EVENT-SLOTS only): Scores based on confidence values and Average Precision (AP); k=3 justifications allowed: SUMMARY: This section provides summary of AP scores Metric RunID Hop AP SF-ALL-Macro hltcoe_KB_ENG_3 0 0.0000 SF-ALL-Macro hltcoe_KB_ENG_3 ALL 0.0000 LDC-MEAN-ALL-Macro hltcoe_KB_ENG_3 0 0.0000 LDC-MEAN-ALL-Macro hltcoe_KB_ENG_3 ALL 0.0000 (PRIMARY METRIC) *ALL-Macro AP refer to mean of corresponding AP values. Scores based on confidence values and Average Precision (AP); only k=1 justification allowed: SUMMARY: This section provides summary of AP scores Metric RunID Hop AP SF-ALL-Macro hltcoe_KB_ENG_3 0 0.0000 SF-ALL-Macro hltcoe_KB_ENG_3 ALL 0.0000 LDC-MEAN-ALL-Macro hltcoe_KB_ENG_3 0 0.0000 LDC-MEAN-ALL-Macro hltcoe_KB_ENG_3 ALL 0.0000 *ALL-Macro AP refer to mean of corresponding AP values. Scores based on P/R/F1; only k=1 justification allowed: Metric RunID Hop Prec Recall F1 SF-ALL-Micro hltcoe_KB_ENG_3 0 0.0000 0.0000 0.0000 SF-ALL-Micro hltcoe_KB_ENG_3 ALL 0.0000 0.0000 0.0000 SF-ALL-Macro hltcoe_KB_ENG_3 0 0.0000 0.0000 0.0000 SF-ALL-Macro hltcoe_KB_ENG_3 ALL 0.0000 0.0000 0.0000 LDC-MAX-ALL-Micro hltcoe_KB_ENG_3 0 0.0000 0.0000 0.0000 LDC-MAX-ALL-Micro hltcoe_KB_ENG_3 ALL 0.0000 0.0000 0.0000 LDC-MAX-ALL-Macro hltcoe_KB_ENG_3 0 0.0000 0.0000 0.0000 LDC-MAX-ALL-Macro hltcoe_KB_ENG_3 ALL 0.0000 0.0000 0.0000 LDC-MEAN-ALL-Macro hltcoe_KB_ENG_3 0 0.0000 0.0000 0.0000 LDC-MEAN-ALL-Macro hltcoe_KB_ENG_3 ALL 0.0000 0.0000 0.0000 *ALL-Macro Prec, Recall and F1 refer to mean-precision, mean-recall and mean-F1. --------------------- Composite KB Evaluation (SENTIMENT-SLOTS only): Scores based on confidence values and Average Precision (AP); k=3 justifications allowed: Metric RunID Hop AP SF-ALL-Macro hltcoe_KB_ENG_3 0 0.0000 SF-ALL-Macro hltcoe_KB_ENG_3 1 0.0000 SF-ALL-Macro hltcoe_KB_ENG_3 ALL 0.0000 LDC-MEAN-ALL-Macro hltcoe_KB_ENG_3 0 0.0000 LDC-MEAN-ALL-Macro hltcoe_KB_ENG_3 1 0.0000 LDC-MEAN-ALL-Macro hltcoe_KB_ENG_3 ALL 0.0000 (PRIMARY METRIC) *ALL-Macro AP refer to mean of corresponding AP values. Scores based on confidence values and Average Precision (AP); only k=1 justification allowed: Metric RunID Hop AP SF-ALL-Macro hltcoe_KB_ENG_3 0 0.0000 SF-ALL-Macro hltcoe_KB_ENG_3 1 0.0000 SF-ALL-Macro hltcoe_KB_ENG_3 ALL 0.0000 LDC-MEAN-ALL-Macro hltcoe_KB_ENG_3 0 0.0000 LDC-MEAN-ALL-Macro hltcoe_KB_ENG_3 1 0.0000 LDC-MEAN-ALL-Macro hltcoe_KB_ENG_3 ALL 0.0000 *ALL-Macro AP refer to mean of corresponding AP values. Scores based on P/R/F1; only k=1 justification allowed: Metric RunID Hop Prec Recall F1 SF-ALL-Micro hltcoe_KB_ENG_3 0 0.0000 0.0000 0.0000 SF-ALL-Micro hltcoe_KB_ENG_3 1 0.0000 0.0000 0.0000 SF-ALL-Micro hltcoe_KB_ENG_3 ALL 0.0000 0.0000 0.0000 SF-ALL-Macro hltcoe_KB_ENG_3 0 0.0000 0.0000 0.0000 SF-ALL-Macro hltcoe_KB_ENG_3 1 0.0000 0.0000 0.0000 SF-ALL-Macro hltcoe_KB_ENG_3 ALL 0.0000 0.0000 0.0000 LDC-MAX-ALL-Micro hltcoe_KB_ENG_3 0 0.0000 0.0000 0.0000 LDC-MAX-ALL-Micro hltcoe_KB_ENG_3 1 0.0000 0.0000 0.0000 LDC-MAX-ALL-Micro hltcoe_KB_ENG_3 ALL 0.0000 0.0000 0.0000 LDC-MAX-ALL-Macro hltcoe_KB_ENG_3 0 0.0000 0.0000 0.0000 LDC-MAX-ALL-Macro hltcoe_KB_ENG_3 1 0.0000 0.0000 0.0000 LDC-MAX-ALL-Macro hltcoe_KB_ENG_3 ALL 0.0000 0.0000 0.0000 LDC-MEAN-ALL-Macro hltcoe_KB_ENG_3 0 0.0000 0.0000 0.0000 LDC-MEAN-ALL-Macro hltcoe_KB_ENG_3 1 0.0000 0.0000 0.0000 LDC-MEAN-ALL-Macro hltcoe_KB_ENG_3 ALL 0.0000 0.0000 0.0000 *ALL-Macro Prec, Recall and F1 refer to mean-precision, mean-recall and mean-F1. ************************************************************* Run ID: hltcoe_KB_ENG_4 Did the run access the live Web during the evaluation window: No Did the run extract relations from the Cold Start source corpus: Yes Did the run generate meaningful confidence values: Yes Did the run attempt the full Entity Discovery and Linking task: Yes Did the run attempt the full Event Nugget Detection and Coreference task: No Did the run include SF relations: Yes Did the run attempt the Event Argument Extraction and Linking task: No Did the run include Sentiment relations: No --------------------- Composite KB Evaluation (All slot types): Scores based on confidence values and Average Precision (AP); k=3 justifications allowed: Metric RunID Hop AP SF-ALL-Macro hltcoe_KB_ENG_4 0 0.0911 SF-ALL-Macro hltcoe_KB_ENG_4 1 0.0097 SF-ALL-Macro hltcoe_KB_ENG_4 ALL 0.0735 LDC-MEAN-ALL-Macro hltcoe_KB_ENG_4 0 0.1096 LDC-MEAN-ALL-Macro hltcoe_KB_ENG_4 1 0.0107 LDC-MEAN-ALL-Macro hltcoe_KB_ENG_4 ALL 0.0903 (PRIMARY METRIC) *ALL-Macro AP refer to mean of corresponding AP values. Scores based on confidence values and Average Precision (AP); only k=1 justification allowed: Metric RunID Hop AP SF-ALL-Macro hltcoe_KB_ENG_4 0 0.0963 SF-ALL-Macro hltcoe_KB_ENG_4 1 0.0097 SF-ALL-Macro hltcoe_KB_ENG_4 ALL 0.0796 LDC-MEAN-ALL-Macro hltcoe_KB_ENG_4 0 0.1169 LDC-MEAN-ALL-Macro hltcoe_KB_ENG_4 1 0.0131 LDC-MEAN-ALL-Macro hltcoe_KB_ENG_4 ALL 0.0970 *ALL-Macro AP refer to mean of corresponding AP values. Scores based on P/R/F1; only k=1 justification allowed: Metric RunID Hop Prec Recall F1 SF-ALL-Micro hltcoe_KB_ENG_4 0 0.3073 0.1044 0.1559 SF-ALL-Micro hltcoe_KB_ENG_4 1 0.0307 0.0712 0.0429 SF-ALL-Micro hltcoe_KB_ENG_4 ALL 0.1227 0.0969 0.1083 SF-ALL-Macro hltcoe_KB_ENG_4 0 0.0970 0.1072 0.0946 SF-ALL-Macro hltcoe_KB_ENG_4 1 0.0709 0.0716 0.0677 SF-ALL-Macro hltcoe_KB_ENG_4 ALL 0.0880 0.0950 0.0854 LDC-MAX-ALL-Micro hltcoe_KB_ENG_4 0 0.3007 0.1338 0.1852 LDC-MAX-ALL-Micro hltcoe_KB_ENG_4 1 0.0980 0.0899 0.0938 LDC-MAX-ALL-Micro hltcoe_KB_ENG_4 ALL 0.2273 0.1243 0.1607 LDC-MAX-ALL-Macro hltcoe_KB_ENG_4 0 0.1279 0.1455 0.1255 LDC-MAX-ALL-Macro hltcoe_KB_ENG_4 1 0.0925 0.0918 0.0877 LDC-MAX-ALL-Macro hltcoe_KB_ENG_4 ALL 0.1158 0.1271 0.1126 LDC-MEAN-ALL-Macro hltcoe_KB_ENG_4 0 0.1125 0.1314 0.1117 LDC-MEAN-ALL-Macro hltcoe_KB_ENG_4 1 0.0885 0.0885 0.0843 LDC-MEAN-ALL-Macro hltcoe_KB_ENG_4 ALL 0.1043 0.1167 0.1023 *ALL-Macro Prec, Recall and F1 refer to mean-precision, mean-recall and mean-F1. --------------------- Composite KB Evaluation (SF-SLOTS only): Scores based on confidence values and Average Precision (AP); k=3 justifications allowed: Metric RunID Hop AP SF-ALL-Macro hltcoe_KB_ENG_4 0 0.2102 SF-ALL-Macro hltcoe_KB_ENG_4 1 0.0271 SF-ALL-Macro hltcoe_KB_ENG_4 ALL 0.1697 LDC-MEAN-ALL-Macro hltcoe_KB_ENG_4 0 0.2362 LDC-MEAN-ALL-Macro hltcoe_KB_ENG_4 1 0.0282 LDC-MEAN-ALL-Macro hltcoe_KB_ENG_4 ALL 0.1946 (PRIMARY METRIC) *ALL-Macro AP refer to mean of corresponding AP values. Scores based on confidence values and Average Precision (AP); only k=1 justification allowed: Metric RunID Hop AP SF-ALL-Macro hltcoe_KB_ENG_4 0 0.2223 SF-ALL-Macro hltcoe_KB_ENG_4 1 0.0272 SF-ALL-Macro hltcoe_KB_ENG_4 ALL 0.1837 LDC-MEAN-ALL-Macro hltcoe_KB_ENG_4 0 0.2520 LDC-MEAN-ALL-Macro hltcoe_KB_ENG_4 1 0.0345 LDC-MEAN-ALL-Macro hltcoe_KB_ENG_4 ALL 0.2091 *ALL-Macro AP refer to mean of corresponding AP values. Scores based on P/R/F1; only k=1 justification allowed: Metric RunID Hop Prec Recall F1 SF-ALL-Micro hltcoe_KB_ENG_4 0 0.3073 0.2107 0.2500 SF-ALL-Micro hltcoe_KB_ENG_4 1 0.0307 0.1339 0.0500 SF-ALL-Micro hltcoe_KB_ENG_4 ALL 0.1227 0.1923 0.1498 SF-ALL-Macro hltcoe_KB_ENG_4 0 0.2240 0.2475 0.2184 SF-ALL-Macro hltcoe_KB_ENG_4 1 0.1307 0.1320 0.1247 SF-ALL-Macro hltcoe_KB_ENG_4 ALL 0.1870 0.2017 0.1813 LDC-MAX-ALL-Micro hltcoe_KB_ENG_4 0 0.3435 0.2576 0.2944 LDC-MAX-ALL-Micro hltcoe_KB_ENG_4 1 0.1012 0.1582 0.1235 LDC-MAX-ALL-Micro hltcoe_KB_ENG_4 ALL 0.2500 0.2346 0.2421 LDC-MAX-ALL-Macro hltcoe_KB_ENG_4 0 0.2757 0.3136 0.2705 LDC-MAX-ALL-Macro hltcoe_KB_ENG_4 1 0.1631 0.1619 0.1548 LDC-MAX-ALL-Macro hltcoe_KB_ENG_4 ALL 0.2320 0.2547 0.2256 LDC-MEAN-ALL-Macro hltcoe_KB_ENG_4 0 0.2426 0.2832 0.2408 LDC-MEAN-ALL-Macro hltcoe_KB_ENG_4 1 0.1562 0.1562 0.1488 LDC-MEAN-ALL-Macro hltcoe_KB_ENG_4 ALL 0.2091 0.2339 0.2050 *ALL-Macro Prec, Recall and F1 refer to mean-precision, mean-recall and mean-F1. --------------------- Composite KB Evaluation (EVENT-SLOTS only): Scores based on confidence values and Average Precision (AP); k=3 justifications allowed: SUMMARY: This section provides summary of AP scores Metric RunID Hop AP SF-ALL-Macro hltcoe_KB_ENG_4 0 0.0000 SF-ALL-Macro hltcoe_KB_ENG_4 ALL 0.0000 LDC-MEAN-ALL-Macro hltcoe_KB_ENG_4 0 0.0000 LDC-MEAN-ALL-Macro hltcoe_KB_ENG_4 ALL 0.0000 (PRIMARY METRIC) *ALL-Macro AP refer to mean of corresponding AP values. Scores based on confidence values and Average Precision (AP); only k=1 justification allowed: SUMMARY: This section provides summary of AP scores Metric RunID Hop AP SF-ALL-Macro hltcoe_KB_ENG_4 0 0.0000 SF-ALL-Macro hltcoe_KB_ENG_4 ALL 0.0000 LDC-MEAN-ALL-Macro hltcoe_KB_ENG_4 0 0.0000 LDC-MEAN-ALL-Macro hltcoe_KB_ENG_4 ALL 0.0000 *ALL-Macro AP refer to mean of corresponding AP values. Scores based on P/R/F1; only k=1 justification allowed: Metric RunID Hop Prec Recall F1 SF-ALL-Micro hltcoe_KB_ENG_4 0 0.0000 0.0000 0.0000 SF-ALL-Micro hltcoe_KB_ENG_4 ALL 0.0000 0.0000 0.0000 SF-ALL-Macro hltcoe_KB_ENG_4 0 0.0000 0.0000 0.0000 SF-ALL-Macro hltcoe_KB_ENG_4 ALL 0.0000 0.0000 0.0000 LDC-MAX-ALL-Micro hltcoe_KB_ENG_4 0 0.0000 0.0000 0.0000 LDC-MAX-ALL-Micro hltcoe_KB_ENG_4 ALL 0.0000 0.0000 0.0000 LDC-MAX-ALL-Macro hltcoe_KB_ENG_4 0 0.0000 0.0000 0.0000 LDC-MAX-ALL-Macro hltcoe_KB_ENG_4 ALL 0.0000 0.0000 0.0000 LDC-MEAN-ALL-Macro hltcoe_KB_ENG_4 0 0.0000 0.0000 0.0000 LDC-MEAN-ALL-Macro hltcoe_KB_ENG_4 ALL 0.0000 0.0000 0.0000 *ALL-Macro Prec, Recall and F1 refer to mean-precision, mean-recall and mean-F1. --------------------- Composite KB Evaluation (SENTIMENT-SLOTS only): Scores based on confidence values and Average Precision (AP); k=3 justifications allowed: Metric RunID Hop AP SF-ALL-Macro hltcoe_KB_ENG_4 0 0.0000 SF-ALL-Macro hltcoe_KB_ENG_4 1 0.0000 SF-ALL-Macro hltcoe_KB_ENG_4 ALL 0.0000 LDC-MEAN-ALL-Macro hltcoe_KB_ENG_4 0 0.0000 LDC-MEAN-ALL-Macro hltcoe_KB_ENG_4 1 0.0000 LDC-MEAN-ALL-Macro hltcoe_KB_ENG_4 ALL 0.0000 (PRIMARY METRIC) *ALL-Macro AP refer to mean of corresponding AP values. Scores based on confidence values and Average Precision (AP); only k=1 justification allowed: Metric RunID Hop AP SF-ALL-Macro hltcoe_KB_ENG_4 0 0.0000 SF-ALL-Macro hltcoe_KB_ENG_4 1 0.0000 SF-ALL-Macro hltcoe_KB_ENG_4 ALL 0.0000 LDC-MEAN-ALL-Macro hltcoe_KB_ENG_4 0 0.0000 LDC-MEAN-ALL-Macro hltcoe_KB_ENG_4 1 0.0000 LDC-MEAN-ALL-Macro hltcoe_KB_ENG_4 ALL 0.0000 *ALL-Macro AP refer to mean of corresponding AP values. Scores based on P/R/F1; only k=1 justification allowed: Metric RunID Hop Prec Recall F1 SF-ALL-Micro hltcoe_KB_ENG_4 0 0.0000 0.0000 0.0000 SF-ALL-Micro hltcoe_KB_ENG_4 1 0.0000 0.0000 0.0000 SF-ALL-Micro hltcoe_KB_ENG_4 ALL 0.0000 0.0000 0.0000 SF-ALL-Macro hltcoe_KB_ENG_4 0 0.0000 0.0000 0.0000 SF-ALL-Macro hltcoe_KB_ENG_4 1 0.0000 0.0000 0.0000 SF-ALL-Macro hltcoe_KB_ENG_4 ALL 0.0000 0.0000 0.0000 LDC-MAX-ALL-Micro hltcoe_KB_ENG_4 0 0.0000 0.0000 0.0000 LDC-MAX-ALL-Micro hltcoe_KB_ENG_4 1 0.0000 0.0000 0.0000 LDC-MAX-ALL-Micro hltcoe_KB_ENG_4 ALL 0.0000 0.0000 0.0000 LDC-MAX-ALL-Macro hltcoe_KB_ENG_4 0 0.0000 0.0000 0.0000 LDC-MAX-ALL-Macro hltcoe_KB_ENG_4 1 0.0000 0.0000 0.0000 LDC-MAX-ALL-Macro hltcoe_KB_ENG_4 ALL 0.0000 0.0000 0.0000 LDC-MEAN-ALL-Macro hltcoe_KB_ENG_4 0 0.0000 0.0000 0.0000 LDC-MEAN-ALL-Macro hltcoe_KB_ENG_4 1 0.0000 0.0000 0.0000 LDC-MEAN-ALL-Macro hltcoe_KB_ENG_4 ALL 0.0000 0.0000 0.0000 *ALL-Macro Prec, Recall and F1 refer to mean-precision, mean-recall and mean-F1.