===================================================================== TAC KBP 2017 ENGLISH KB CONSTRUCTION: COMPOSITE KB EVALUATION RESULTS ===================================================================== Team ID: A2KD_Adept Organization: Raytheon BBN Technologies ************************************************************* Run ID: A2KD_Adept_KB_ENG_1 Did the run access the live Web during the evaluation window: No Did the run extract relations from the Cold Start source corpus: Yes Did the run generate meaningful confidence values: Yes Did the run attempt the full Entity Discovery and Linking task: Yes Did the run attempt the full Event Nugget Detection and Coreference task: No Did the run include SF relations: Yes Did the run attempt the Event Argument Extraction and Linking task: Yes Did the run include Sentiment relations: No --------------------- Composite KB Evaluation (All slot types): Scores based on confidence values and Average Precision (AP); k=3 justifications allowed: Metric RunID Hop AP SF-ALL-Macro A2KD_Adept_KB_ENG_1 0 0.0920 SF-ALL-Macro A2KD_Adept_KB_ENG_1 1 0.0057 SF-ALL-Macro A2KD_Adept_KB_ENG_1 ALL 0.0837 LDC-MEAN-ALL-Macro A2KD_Adept_KB_ENG_1 0 0.1033 LDC-MEAN-ALL-Macro A2KD_Adept_KB_ENG_1 1 0.0050 LDC-MEAN-ALL-Macro A2KD_Adept_KB_ENG_1 ALL 0.0940 (PRIMARY METRIC) *ALL-Macro AP refer to mean of corresponding AP values. Scores based on confidence values and Average Precision (AP); only k=1 justification allowed: Metric RunID Hop AP SF-ALL-Macro A2KD_Adept_KB_ENG_1 0 0.1273 SF-ALL-Macro A2KD_Adept_KB_ENG_1 1 0.0179 SF-ALL-Macro A2KD_Adept_KB_ENG_1 ALL 0.1196 LDC-MEAN-ALL-Macro A2KD_Adept_KB_ENG_1 0 0.1407 LDC-MEAN-ALL-Macro A2KD_Adept_KB_ENG_1 1 0.0184 LDC-MEAN-ALL-Macro A2KD_Adept_KB_ENG_1 ALL 0.1314 *ALL-Macro AP refer to mean of corresponding AP values. Scores based on P/R/F1; only k=1 justification allowed: Metric RunID Hop Prec Recall F1 SF-ALL-Micro A2KD_Adept_KB_ENG_1 0 0.2452 0.1699 0.2007 SF-ALL-Micro A2KD_Adept_KB_ENG_1 1 0.0180 0.0538 0.0270 SF-ALL-Micro A2KD_Adept_KB_ENG_1 ALL 0.1181 0.1435 0.1296 SF-ALL-Macro A2KD_Adept_KB_ENG_1 0 0.1430 0.1509 0.1293 SF-ALL-Macro A2KD_Adept_KB_ENG_1 1 0.0384 0.0391 0.0371 SF-ALL-Macro A2KD_Adept_KB_ENG_1 ALL 0.1070 0.1125 0.0976 LDC-MAX-ALL-Micro A2KD_Adept_KB_ENG_1 0 0.2747 0.2398 0.2561 LDC-MAX-ALL-Micro A2KD_Adept_KB_ENG_1 1 0.0271 0.0683 0.0388 LDC-MAX-ALL-Micro A2KD_Adept_KB_ENG_1 ALL 0.1649 0.2028 0.1819 LDC-MAX-ALL-Macro A2KD_Adept_KB_ENG_1 0 0.1899 0.2132 0.1783 LDC-MAX-ALL-Macro A2KD_Adept_KB_ENG_1 1 0.0420 0.0444 0.0413 LDC-MAX-ALL-Macro A2KD_Adept_KB_ENG_1 ALL 0.1394 0.1555 0.1314 LDC-MEAN-ALL-Macro A2KD_Adept_KB_ENG_1 0 0.1545 0.1703 0.1433 LDC-MEAN-ALL-Macro A2KD_Adept_KB_ENG_1 1 0.0378 0.0386 0.0366 LDC-MEAN-ALL-Macro A2KD_Adept_KB_ENG_1 ALL 0.1146 0.1253 0.1068 *ALL-Macro Prec, Recall and F1 refer to mean-precision, mean-recall and mean-F1. --------------------- Composite KB Evaluation (SF-SLOTS only): Scores based on confidence values and Average Precision (AP); k=3 justifications allowed: Metric RunID Hop AP SF-ALL-Macro A2KD_Adept_KB_ENG_1 0 0.0854 SF-ALL-Macro A2KD_Adept_KB_ENG_1 1 0.0132 SF-ALL-Macro A2KD_Adept_KB_ENG_1 ALL 0.0663 LDC-MEAN-ALL-Macro A2KD_Adept_KB_ENG_1 0 0.0908 LDC-MEAN-ALL-Macro A2KD_Adept_KB_ENG_1 1 0.0108 LDC-MEAN-ALL-Macro A2KD_Adept_KB_ENG_1 ALL 0.0707 (PRIMARY METRIC) *ALL-Macro AP refer to mean of corresponding AP values. Scores based on confidence values and Average Precision (AP); only k=1 justification allowed: Metric RunID Hop AP SF-ALL-Macro A2KD_Adept_KB_ENG_1 0 0.1242 SF-ALL-Macro A2KD_Adept_KB_ENG_1 1 0.0289 SF-ALL-Macro A2KD_Adept_KB_ENG_1 ALL 0.1063 LDC-MEAN-ALL-Macro A2KD_Adept_KB_ENG_1 0 0.1307 LDC-MEAN-ALL-Macro A2KD_Adept_KB_ENG_1 1 0.0236 LDC-MEAN-ALL-Macro A2KD_Adept_KB_ENG_1 ALL 0.1108 *ALL-Macro AP refer to mean of corresponding AP values. Scores based on P/R/F1; only k=1 justification allowed: Metric RunID Hop Prec Recall F1 SF-ALL-Micro A2KD_Adept_KB_ENG_1 0 0.1804 0.0984 0.1274 SF-ALL-Micro A2KD_Adept_KB_ENG_1 1 0.0211 0.0765 0.0331 SF-ALL-Micro A2KD_Adept_KB_ENG_1 ALL 0.0725 0.0932 0.0815 SF-ALL-Macro A2KD_Adept_KB_ENG_1 0 0.1373 0.1351 0.1221 SF-ALL-Macro A2KD_Adept_KB_ENG_1 1 0.0620 0.0617 0.0593 SF-ALL-Macro A2KD_Adept_KB_ENG_1 ALL 0.1074 0.1060 0.0972 LDC-MAX-ALL-Micro A2KD_Adept_KB_ENG_1 0 0.1837 0.1202 0.1453 LDC-MAX-ALL-Micro A2KD_Adept_KB_ENG_1 1 0.0252 0.0759 0.0378 LDC-MAX-ALL-Micro A2KD_Adept_KB_ENG_1 ALL 0.0915 0.1100 0.0999 LDC-MAX-ALL-Macro A2KD_Adept_KB_ENG_1 0 0.1736 0.1715 0.1555 LDC-MAX-ALL-Macro A2KD_Adept_KB_ENG_1 1 0.0621 0.0621 0.0597 LDC-MAX-ALL-Macro A2KD_Adept_KB_ENG_1 ALL 0.1303 0.1290 0.1183 LDC-MEAN-ALL-Macro A2KD_Adept_KB_ENG_1 0 0.1428 0.1417 0.1271 LDC-MEAN-ALL-Macro A2KD_Adept_KB_ENG_1 1 0.0563 0.0563 0.0540 LDC-MEAN-ALL-Macro A2KD_Adept_KB_ENG_1 ALL 0.1092 0.1086 0.0987 *ALL-Macro Prec, Recall and F1 refer to mean-precision, mean-recall and mean-F1. --------------------- Composite KB Evaluation (EVENT-SLOTS only): Scores based on confidence values and Average Precision (AP); k=3 justifications allowed: SUMMARY: This section provides summary of AP scores Metric RunID Hop AP SF-ALL-Macro A2KD_Adept_KB_ENG_1 0 0.1459 SF-ALL-Macro A2KD_Adept_KB_ENG_1 ALL 0.1459 LDC-MEAN-ALL-Macro A2KD_Adept_KB_ENG_1 0 0.1707 LDC-MEAN-ALL-Macro A2KD_Adept_KB_ENG_1 ALL 0.1707 (PRIMARY METRIC) *ALL-Macro AP refer to mean of corresponding AP values. Scores based on confidence values and Average Precision (AP); only k=1 justification allowed: SUMMARY: This section provides summary of AP scores Metric RunID Hop AP SF-ALL-Macro A2KD_Adept_KB_ENG_1 0 0.1951 SF-ALL-Macro A2KD_Adept_KB_ENG_1 ALL 0.1951 LDC-MEAN-ALL-Macro A2KD_Adept_KB_ENG_1 0 0.2234 LDC-MEAN-ALL-Macro A2KD_Adept_KB_ENG_1 ALL 0.2234 *ALL-Macro AP refer to mean of corresponding AP values. Scores based on P/R/F1; only k=1 justification allowed: Metric RunID Hop Prec Recall F1 SF-ALL-Micro A2KD_Adept_KB_ENG_1 0 0.2867 0.3287 0.3063 SF-ALL-Micro A2KD_Adept_KB_ENG_1 ALL 0.2867 0.3287 0.3063 SF-ALL-Macro A2KD_Adept_KB_ENG_1 0 0.2217 0.2452 0.2028 SF-ALL-Macro A2KD_Adept_KB_ENG_1 ALL 0.2217 0.2452 0.2028 LDC-MAX-ALL-Micro A2KD_Adept_KB_ENG_1 0 0.3039 0.4799 0.3721 LDC-MAX-ALL-Micro A2KD_Adept_KB_ENG_1 ALL 0.3039 0.4799 0.3721 LDC-MAX-ALL-Macro A2KD_Adept_KB_ENG_1 0 0.3053 0.3730 0.2962 LDC-MAX-ALL-Macro A2KD_Adept_KB_ENG_1 ALL 0.3053 0.3730 0.2962 LDC-MEAN-ALL-Macro A2KD_Adept_KB_ENG_1 0 0.2463 0.2919 0.2353 LDC-MEAN-ALL-Macro A2KD_Adept_KB_ENG_1 ALL 0.2463 0.2919 0.2353 *ALL-Macro Prec, Recall and F1 refer to mean-precision, mean-recall and mean-F1. --------------------- Composite KB Evaluation (SENTIMENT-SLOTS only): Scores based on confidence values and Average Precision (AP); k=3 justifications allowed: Metric RunID Hop AP SF-ALL-Macro A2KD_Adept_KB_ENG_1 0 0.0000 SF-ALL-Macro A2KD_Adept_KB_ENG_1 1 0.0000 SF-ALL-Macro A2KD_Adept_KB_ENG_1 ALL 0.0000 LDC-MEAN-ALL-Macro A2KD_Adept_KB_ENG_1 0 0.0000 LDC-MEAN-ALL-Macro A2KD_Adept_KB_ENG_1 1 0.0000 LDC-MEAN-ALL-Macro A2KD_Adept_KB_ENG_1 ALL 0.0000 (PRIMARY METRIC) *ALL-Macro AP refer to mean of corresponding AP values. Scores based on confidence values and Average Precision (AP); only k=1 justification allowed: Metric RunID Hop AP SF-ALL-Macro A2KD_Adept_KB_ENG_1 0 0.0000 SF-ALL-Macro A2KD_Adept_KB_ENG_1 1 0.0000 SF-ALL-Macro A2KD_Adept_KB_ENG_1 ALL 0.0000 LDC-MEAN-ALL-Macro A2KD_Adept_KB_ENG_1 0 0.0000 LDC-MEAN-ALL-Macro A2KD_Adept_KB_ENG_1 1 0.0000 LDC-MEAN-ALL-Macro A2KD_Adept_KB_ENG_1 ALL 0.0000 *ALL-Macro AP refer to mean of corresponding AP values. Scores based on P/R/F1; only k=1 justification allowed: Metric RunID Hop Prec Recall F1 SF-ALL-Micro A2KD_Adept_KB_ENG_1 0 0.0000 0.0000 0.0000 SF-ALL-Micro A2KD_Adept_KB_ENG_1 1 0.0000 0.0000 0.0000 SF-ALL-Micro A2KD_Adept_KB_ENG_1 ALL 0.0000 0.0000 0.0000 SF-ALL-Macro A2KD_Adept_KB_ENG_1 0 0.0000 0.0000 0.0000 SF-ALL-Macro A2KD_Adept_KB_ENG_1 1 0.0000 0.0000 0.0000 SF-ALL-Macro A2KD_Adept_KB_ENG_1 ALL 0.0000 0.0000 0.0000 LDC-MAX-ALL-Micro A2KD_Adept_KB_ENG_1 0 0.0000 0.0000 0.0000 LDC-MAX-ALL-Micro A2KD_Adept_KB_ENG_1 1 0.0000 0.0000 0.0000 LDC-MAX-ALL-Micro A2KD_Adept_KB_ENG_1 ALL 0.0000 0.0000 0.0000 LDC-MAX-ALL-Macro A2KD_Adept_KB_ENG_1 0 0.0000 0.0000 0.0000 LDC-MAX-ALL-Macro A2KD_Adept_KB_ENG_1 1 0.0000 0.0000 0.0000 LDC-MAX-ALL-Macro A2KD_Adept_KB_ENG_1 ALL 0.0000 0.0000 0.0000 LDC-MEAN-ALL-Macro A2KD_Adept_KB_ENG_1 0 0.0000 0.0000 0.0000 LDC-MEAN-ALL-Macro A2KD_Adept_KB_ENG_1 1 0.0000 0.0000 0.0000 LDC-MEAN-ALL-Macro A2KD_Adept_KB_ENG_1 ALL 0.0000 0.0000 0.0000 *ALL-Macro Prec, Recall and F1 refer to mean-precision, mean-recall and mean-F1. ************************************************************* Run ID: A2KD_Adept_KB_ENG_2 Did the run access the live Web during the evaluation window: No Did the run extract relations from the Cold Start source corpus: Yes Did the run generate meaningful confidence values: Yes Did the run attempt the full Entity Discovery and Linking task: Yes Did the run attempt the full Event Nugget Detection and Coreference task: No Did the run include SF relations: Yes Did the run attempt the Event Argument Extraction and Linking task: Yes Did the run include Sentiment relations: No --------------------- Composite KB Evaluation (All slot types): Scores based on confidence values and Average Precision (AP); k=3 justifications allowed: Metric RunID Hop AP SF-ALL-Macro A2KD_Adept_KB_ENG_2 0 0.0920 SF-ALL-Macro A2KD_Adept_KB_ENG_2 1 0.0057 SF-ALL-Macro A2KD_Adept_KB_ENG_2 ALL 0.0837 LDC-MEAN-ALL-Macro A2KD_Adept_KB_ENG_2 0 0.1033 LDC-MEAN-ALL-Macro A2KD_Adept_KB_ENG_2 1 0.0050 LDC-MEAN-ALL-Macro A2KD_Adept_KB_ENG_2 ALL 0.0940 (PRIMARY METRIC) *ALL-Macro AP refer to mean of corresponding AP values. Scores based on confidence values and Average Precision (AP); only k=1 justification allowed: Metric RunID Hop AP SF-ALL-Macro A2KD_Adept_KB_ENG_2 0 0.1273 SF-ALL-Macro A2KD_Adept_KB_ENG_2 1 0.0179 SF-ALL-Macro A2KD_Adept_KB_ENG_2 ALL 0.1196 LDC-MEAN-ALL-Macro A2KD_Adept_KB_ENG_2 0 0.1407 LDC-MEAN-ALL-Macro A2KD_Adept_KB_ENG_2 1 0.0184 LDC-MEAN-ALL-Macro A2KD_Adept_KB_ENG_2 ALL 0.1314 *ALL-Macro AP refer to mean of corresponding AP values. Scores based on P/R/F1; only k=1 justification allowed: Metric RunID Hop Prec Recall F1 SF-ALL-Micro A2KD_Adept_KB_ENG_2 0 0.2446 0.1699 0.2005 SF-ALL-Micro A2KD_Adept_KB_ENG_2 1 0.0177 0.0538 0.0266 SF-ALL-Micro A2KD_Adept_KB_ENG_2 ALL 0.1168 0.1435 0.1288 SF-ALL-Macro A2KD_Adept_KB_ENG_2 0 0.1430 0.1509 0.1293 SF-ALL-Macro A2KD_Adept_KB_ENG_2 1 0.0377 0.0391 0.0366 SF-ALL-Macro A2KD_Adept_KB_ENG_2 ALL 0.1068 0.1125 0.0974 LDC-MAX-ALL-Micro A2KD_Adept_KB_ENG_2 0 0.2616 0.2398 0.2503 LDC-MAX-ALL-Micro A2KD_Adept_KB_ENG_2 1 0.0222 0.0683 0.0336 LDC-MAX-ALL-Micro A2KD_Adept_KB_ENG_2 ALL 0.1467 0.2028 0.1703 LDC-MAX-ALL-Macro A2KD_Adept_KB_ENG_2 0 0.1899 0.2132 0.1783 LDC-MAX-ALL-Macro A2KD_Adept_KB_ENG_2 1 0.0411 0.0444 0.0406 LDC-MAX-ALL-Macro A2KD_Adept_KB_ENG_2 ALL 0.1391 0.1555 0.1312 LDC-MEAN-ALL-Macro A2KD_Adept_KB_ENG_2 0 0.1545 0.1703 0.1433 LDC-MEAN-ALL-Macro A2KD_Adept_KB_ENG_2 1 0.0369 0.0386 0.0359 LDC-MEAN-ALL-Macro A2KD_Adept_KB_ENG_2 ALL 0.1143 0.1253 0.1066 *ALL-Macro Prec, Recall and F1 refer to mean-precision, mean-recall and mean-F1. --------------------- Composite KB Evaluation (SF-SLOTS only): Scores based on confidence values and Average Precision (AP); k=3 justifications allowed: Metric RunID Hop AP SF-ALL-Macro A2KD_Adept_KB_ENG_2 0 0.0854 SF-ALL-Macro A2KD_Adept_KB_ENG_2 1 0.0132 SF-ALL-Macro A2KD_Adept_KB_ENG_2 ALL 0.0663 LDC-MEAN-ALL-Macro A2KD_Adept_KB_ENG_2 0 0.0908 LDC-MEAN-ALL-Macro A2KD_Adept_KB_ENG_2 1 0.0108 LDC-MEAN-ALL-Macro A2KD_Adept_KB_ENG_2 ALL 0.0707 (PRIMARY METRIC) *ALL-Macro AP refer to mean of corresponding AP values. Scores based on confidence values and Average Precision (AP); only k=1 justification allowed: Metric RunID Hop AP SF-ALL-Macro A2KD_Adept_KB_ENG_2 0 0.1242 SF-ALL-Macro A2KD_Adept_KB_ENG_2 1 0.0289 SF-ALL-Macro A2KD_Adept_KB_ENG_2 ALL 0.1063 LDC-MEAN-ALL-Macro A2KD_Adept_KB_ENG_2 0 0.1307 LDC-MEAN-ALL-Macro A2KD_Adept_KB_ENG_2 1 0.0236 LDC-MEAN-ALL-Macro A2KD_Adept_KB_ENG_2 ALL 0.1108 *ALL-Macro AP refer to mean of corresponding AP values. Scores based on P/R/F1; only k=1 justification allowed: Metric RunID Hop Prec Recall F1 SF-ALL-Micro A2KD_Adept_KB_ENG_2 0 0.1792 0.0984 0.1271 SF-ALL-Micro A2KD_Adept_KB_ENG_2 1 0.0205 0.0765 0.0324 SF-ALL-Micro A2KD_Adept_KB_ENG_2 ALL 0.0710 0.0932 0.0806 SF-ALL-Macro A2KD_Adept_KB_ENG_2 0 0.1373 0.1351 0.1221 SF-ALL-Macro A2KD_Adept_KB_ENG_2 1 0.0608 0.0617 0.0582 SF-ALL-Macro A2KD_Adept_KB_ENG_2 ALL 0.1069 0.1060 0.0968 LDC-MAX-ALL-Micro A2KD_Adept_KB_ENG_2 0 0.2032 0.1202 0.1511 LDC-MAX-ALL-Micro A2KD_Adept_KB_ENG_2 1 0.0232 0.0759 0.0356 LDC-MAX-ALL-Micro A2KD_Adept_KB_ENG_2 ALL 0.0907 0.1100 0.0994 LDC-MAX-ALL-Macro A2KD_Adept_KB_ENG_2 0 0.1736 0.1715 0.1555 LDC-MAX-ALL-Macro A2KD_Adept_KB_ENG_2 1 0.0605 0.0621 0.0585 LDC-MAX-ALL-Macro A2KD_Adept_KB_ENG_2 ALL 0.1297 0.1290 0.1178 LDC-MEAN-ALL-Macro A2KD_Adept_KB_ENG_2 0 0.1428 0.1417 0.1271 LDC-MEAN-ALL-Macro A2KD_Adept_KB_ENG_2 1 0.0548 0.0563 0.0527 LDC-MEAN-ALL-Macro A2KD_Adept_KB_ENG_2 ALL 0.1086 0.1086 0.0982 *ALL-Macro Prec, Recall and F1 refer to mean-precision, mean-recall and mean-F1. --------------------- Composite KB Evaluation (EVENT-SLOTS only): Scores based on confidence values and Average Precision (AP); k=3 justifications allowed: SUMMARY: This section provides summary of AP scores Metric RunID Hop AP SF-ALL-Macro A2KD_Adept_KB_ENG_2 0 0.1459 SF-ALL-Macro A2KD_Adept_KB_ENG_2 ALL 0.1459 LDC-MEAN-ALL-Macro A2KD_Adept_KB_ENG_2 0 0.1707 LDC-MEAN-ALL-Macro A2KD_Adept_KB_ENG_2 ALL 0.1707 (PRIMARY METRIC) *ALL-Macro AP refer to mean of corresponding AP values. Scores based on confidence values and Average Precision (AP); only k=1 justification allowed: SUMMARY: This section provides summary of AP scores Metric RunID Hop AP SF-ALL-Macro A2KD_Adept_KB_ENG_2 0 0.1951 SF-ALL-Macro A2KD_Adept_KB_ENG_2 ALL 0.1951 LDC-MEAN-ALL-Macro A2KD_Adept_KB_ENG_2 0 0.2234 LDC-MEAN-ALL-Macro A2KD_Adept_KB_ENG_2 ALL 0.2234 *ALL-Macro AP refer to mean of corresponding AP values. Scores based on P/R/F1; only k=1 justification allowed: Metric RunID Hop Prec Recall F1 SF-ALL-Micro A2KD_Adept_KB_ENG_2 0 0.2867 0.3287 0.3063 SF-ALL-Micro A2KD_Adept_KB_ENG_2 ALL 0.2867 0.3287 0.3063 SF-ALL-Macro A2KD_Adept_KB_ENG_2 0 0.2217 0.2452 0.2028 SF-ALL-Macro A2KD_Adept_KB_ENG_2 ALL 0.2217 0.2452 0.2028 LDC-MAX-ALL-Micro A2KD_Adept_KB_ENG_2 0 0.3070 0.4799 0.3745 LDC-MAX-ALL-Micro A2KD_Adept_KB_ENG_2 ALL 0.3070 0.4799 0.3745 LDC-MAX-ALL-Macro A2KD_Adept_KB_ENG_2 0 0.3053 0.3730 0.2962 LDC-MAX-ALL-Macro A2KD_Adept_KB_ENG_2 ALL 0.3053 0.3730 0.2962 LDC-MEAN-ALL-Macro A2KD_Adept_KB_ENG_2 0 0.2463 0.2919 0.2353 LDC-MEAN-ALL-Macro A2KD_Adept_KB_ENG_2 ALL 0.2463 0.2919 0.2353 *ALL-Macro Prec, Recall and F1 refer to mean-precision, mean-recall and mean-F1. --------------------- Composite KB Evaluation (SENTIMENT-SLOTS only): Scores based on confidence values and Average Precision (AP); k=3 justifications allowed: Metric RunID Hop AP SF-ALL-Macro A2KD_Adept_KB_ENG_2 0 0.0000 SF-ALL-Macro A2KD_Adept_KB_ENG_2 1 0.0000 SF-ALL-Macro A2KD_Adept_KB_ENG_2 ALL 0.0000 LDC-MEAN-ALL-Macro A2KD_Adept_KB_ENG_2 0 0.0000 LDC-MEAN-ALL-Macro A2KD_Adept_KB_ENG_2 1 0.0000 LDC-MEAN-ALL-Macro A2KD_Adept_KB_ENG_2 ALL 0.0000 (PRIMARY METRIC) *ALL-Macro AP refer to mean of corresponding AP values. Scores based on confidence values and Average Precision (AP); only k=1 justification allowed: Metric RunID Hop AP SF-ALL-Macro A2KD_Adept_KB_ENG_2 0 0.0000 SF-ALL-Macro A2KD_Adept_KB_ENG_2 1 0.0000 SF-ALL-Macro A2KD_Adept_KB_ENG_2 ALL 0.0000 LDC-MEAN-ALL-Macro A2KD_Adept_KB_ENG_2 0 0.0000 LDC-MEAN-ALL-Macro A2KD_Adept_KB_ENG_2 1 0.0000 LDC-MEAN-ALL-Macro A2KD_Adept_KB_ENG_2 ALL 0.0000 *ALL-Macro AP refer to mean of corresponding AP values. Scores based on P/R/F1; only k=1 justification allowed: Metric RunID Hop Prec Recall F1 SF-ALL-Micro A2KD_Adept_KB_ENG_2 0 0.0000 0.0000 0.0000 SF-ALL-Micro A2KD_Adept_KB_ENG_2 1 0.0000 0.0000 0.0000 SF-ALL-Micro A2KD_Adept_KB_ENG_2 ALL 0.0000 0.0000 0.0000 SF-ALL-Macro A2KD_Adept_KB_ENG_2 0 0.0000 0.0000 0.0000 SF-ALL-Macro A2KD_Adept_KB_ENG_2 1 0.0000 0.0000 0.0000 SF-ALL-Macro A2KD_Adept_KB_ENG_2 ALL 0.0000 0.0000 0.0000 LDC-MAX-ALL-Micro A2KD_Adept_KB_ENG_2 0 0.0000 0.0000 0.0000 LDC-MAX-ALL-Micro A2KD_Adept_KB_ENG_2 1 0.0000 0.0000 0.0000 LDC-MAX-ALL-Micro A2KD_Adept_KB_ENG_2 ALL 0.0000 0.0000 0.0000 LDC-MAX-ALL-Macro A2KD_Adept_KB_ENG_2 0 0.0000 0.0000 0.0000 LDC-MAX-ALL-Macro A2KD_Adept_KB_ENG_2 1 0.0000 0.0000 0.0000 LDC-MAX-ALL-Macro A2KD_Adept_KB_ENG_2 ALL 0.0000 0.0000 0.0000 LDC-MEAN-ALL-Macro A2KD_Adept_KB_ENG_2 0 0.0000 0.0000 0.0000 LDC-MEAN-ALL-Macro A2KD_Adept_KB_ENG_2 1 0.0000 0.0000 0.0000 LDC-MEAN-ALL-Macro A2KD_Adept_KB_ENG_2 ALL 0.0000 0.0000 0.0000 *ALL-Macro Prec, Recall and F1 refer to mean-precision, mean-recall and mean-F1. ************************************************************* Run ID: A2KD_Adept_KB_ENG_3 Did the run access the live Web during the evaluation window: No Did the run extract relations from the Cold Start source corpus: Yes Did the run generate meaningful confidence values: Yes Did the run attempt the full Entity Discovery and Linking task: Yes Did the run attempt the full Event Nugget Detection and Coreference task: No Did the run include SF relations: Yes Did the run attempt the Event Argument Extraction and Linking task: Yes Did the run include Sentiment relations: No --------------------- Composite KB Evaluation (All slot types): Scores based on confidence values and Average Precision (AP); k=3 justifications allowed: Metric RunID Hop AP SF-ALL-Macro A2KD_Adept_KB_ENG_3 0 0.0954 SF-ALL-Macro A2KD_Adept_KB_ENG_3 1 0.0111 SF-ALL-Macro A2KD_Adept_KB_ENG_3 ALL 0.0888 LDC-MEAN-ALL-Macro A2KD_Adept_KB_ENG_3 0 0.1038 LDC-MEAN-ALL-Macro A2KD_Adept_KB_ENG_3 1 0.0125 LDC-MEAN-ALL-Macro A2KD_Adept_KB_ENG_3 ALL 0.0963 (PRIMARY METRIC) *ALL-Macro AP refer to mean of corresponding AP values. Scores based on confidence values and Average Precision (AP); only k=1 justification allowed: Metric RunID Hop AP SF-ALL-Macro A2KD_Adept_KB_ENG_3 0 0.1296 SF-ALL-Macro A2KD_Adept_KB_ENG_3 1 0.0230 SF-ALL-Macro A2KD_Adept_KB_ENG_3 ALL 0.1235 LDC-MEAN-ALL-Macro A2KD_Adept_KB_ENG_3 0 0.1422 LDC-MEAN-ALL-Macro A2KD_Adept_KB_ENG_3 1 0.0258 LDC-MEAN-ALL-Macro A2KD_Adept_KB_ENG_3 ALL 0.1349 *ALL-Macro AP refer to mean of corresponding AP values. Scores based on P/R/F1; only k=1 justification allowed: Metric RunID Hop Prec Recall F1 SF-ALL-Micro A2KD_Adept_KB_ENG_3 0 0.2462 0.1823 0.2095 SF-ALL-Micro A2KD_Adept_KB_ENG_3 1 0.0112 0.0494 0.0183 SF-ALL-Micro A2KD_Adept_KB_ENG_3 ALL 0.0967 0.1521 0.1182 SF-ALL-Macro A2KD_Adept_KB_ENG_3 0 0.1413 0.1545 0.1273 SF-ALL-Macro A2KD_Adept_KB_ENG_3 1 0.0392 0.0418 0.0384 SF-ALL-Macro A2KD_Adept_KB_ENG_3 ALL 0.1062 0.1157 0.0967 LDC-MAX-ALL-Micro A2KD_Adept_KB_ENG_3 0 0.2706 0.2379 0.2532 LDC-MAX-ALL-Micro A2KD_Adept_KB_ENG_3 1 0.0189 0.0540 0.0280 LDC-MAX-ALL-Micro A2KD_Adept_KB_ENG_3 ALL 0.1516 0.1981 0.1718 LDC-MAX-ALL-Macro A2KD_Adept_KB_ENG_3 0 0.1800 0.2004 0.1664 LDC-MAX-ALL-Macro A2KD_Adept_KB_ENG_3 1 0.0423 0.0459 0.0419 LDC-MAX-ALL-Macro A2KD_Adept_KB_ENG_3 ALL 0.1329 0.1476 0.1239 LDC-MEAN-ALL-Macro A2KD_Adept_KB_ENG_3 0 0.1534 0.1731 0.1419 LDC-MEAN-ALL-Macro A2KD_Adept_KB_ENG_3 1 0.0391 0.0426 0.0386 LDC-MEAN-ALL-Macro A2KD_Adept_KB_ENG_3 ALL 0.1143 0.1285 0.1066 *ALL-Macro Prec, Recall and F1 refer to mean-precision, mean-recall and mean-F1. --------------------- Composite KB Evaluation (SF-SLOTS only): Scores based on confidence values and Average Precision (AP); k=3 justifications allowed: Metric RunID Hop AP SF-ALL-Macro A2KD_Adept_KB_ENG_3 0 0.0907 SF-ALL-Macro A2KD_Adept_KB_ENG_3 1 0.0309 SF-ALL-Macro A2KD_Adept_KB_ENG_3 ALL 0.0753 LDC-MEAN-ALL-Macro A2KD_Adept_KB_ENG_3 0 0.0969 LDC-MEAN-ALL-Macro A2KD_Adept_KB_ENG_3 1 0.0330 LDC-MEAN-ALL-Macro A2KD_Adept_KB_ENG_3 ALL 0.0807 (PRIMARY METRIC) *ALL-Macro AP refer to mean of corresponding AP values. Scores based on confidence values and Average Precision (AP); only k=1 justification allowed: Metric RunID Hop AP SF-ALL-Macro A2KD_Adept_KB_ENG_3 0 0.1372 SF-ALL-Macro A2KD_Adept_KB_ENG_3 1 0.0460 SF-ALL-Macro A2KD_Adept_KB_ENG_3 ALL 0.1231 LDC-MEAN-ALL-Macro A2KD_Adept_KB_ENG_3 0 0.1457 LDC-MEAN-ALL-Macro A2KD_Adept_KB_ENG_3 1 0.0453 LDC-MEAN-ALL-Macro A2KD_Adept_KB_ENG_3 ALL 0.1299 *ALL-Macro AP refer to mean of corresponding AP values. Scores based on P/R/F1; only k=1 justification allowed: Metric RunID Hop Prec Recall F1 SF-ALL-Micro A2KD_Adept_KB_ENG_3 0 0.1720 0.1019 0.1280 SF-ALL-Micro A2KD_Adept_KB_ENG_3 1 0.0125 0.0820 0.0217 SF-ALL-Micro A2KD_Adept_KB_ENG_3 ALL 0.0480 0.0971 0.0642 SF-ALL-Macro A2KD_Adept_KB_ENG_3 0 0.1385 0.1485 0.1294 SF-ALL-Macro A2KD_Adept_KB_ENG_3 1 0.0646 0.0693 0.0631 SF-ALL-Macro A2KD_Adept_KB_ENG_3 ALL 0.1092 0.1171 0.1031 LDC-MAX-ALL-Micro A2KD_Adept_KB_ENG_3 0 0.2000 0.1202 0.1502 LDC-MAX-ALL-Micro A2KD_Adept_KB_ENG_3 1 0.0087 0.0823 0.0158 LDC-MAX-ALL-Micro A2KD_Adept_KB_ENG_3 ALL 0.0421 0.1114 0.0611 LDC-MAX-ALL-Macro A2KD_Adept_KB_ENG_3 0 0.1657 0.1730 0.1525 LDC-MAX-ALL-Macro A2KD_Adept_KB_ENG_3 1 0.0653 0.0715 0.0644 LDC-MAX-ALL-Macro A2KD_Adept_KB_ENG_3 ALL 0.1267 0.1336 0.1183 LDC-MEAN-ALL-Macro A2KD_Adept_KB_ENG_3 0 0.1465 0.1572 0.1373 LDC-MEAN-ALL-Macro A2KD_Adept_KB_ENG_3 1 0.0595 0.0658 0.0587 LDC-MEAN-ALL-Macro A2KD_Adept_KB_ENG_3 ALL 0.1127 0.1217 0.1068 *ALL-Macro Prec, Recall and F1 refer to mean-precision, mean-recall and mean-F1. --------------------- Composite KB Evaluation (EVENT-SLOTS only): Scores based on confidence values and Average Precision (AP); k=3 justifications allowed: SUMMARY: This section provides summary of AP scores Metric RunID Hop AP SF-ALL-Macro A2KD_Adept_KB_ENG_3 0 0.1490 SF-ALL-Macro A2KD_Adept_KB_ENG_3 ALL 0.1490 LDC-MEAN-ALL-Macro A2KD_Adept_KB_ENG_3 0 0.1644 LDC-MEAN-ALL-Macro A2KD_Adept_KB_ENG_3 ALL 0.1644 (PRIMARY METRIC) *ALL-Macro AP refer to mean of corresponding AP values. Scores based on confidence values and Average Precision (AP); only k=1 justification allowed: SUMMARY: This section provides summary of AP scores Metric RunID Hop AP SF-ALL-Macro A2KD_Adept_KB_ENG_3 0 0.1863 SF-ALL-Macro A2KD_Adept_KB_ENG_3 ALL 0.1863 LDC-MEAN-ALL-Macro A2KD_Adept_KB_ENG_3 0 0.2082 LDC-MEAN-ALL-Macro A2KD_Adept_KB_ENG_3 ALL 0.2082 *ALL-Macro AP refer to mean of corresponding AP values. Scores based on P/R/F1; only k=1 justification allowed: Metric RunID Hop Prec Recall F1 SF-ALL-Micro A2KD_Adept_KB_ENG_3 0 0.2950 0.3577 0.3234 SF-ALL-Micro A2KD_Adept_KB_ENG_3 ALL 0.2950 0.3577 0.3234 SF-ALL-Macro A2KD_Adept_KB_ENG_3 0 0.2158 0.2394 0.1892 SF-ALL-Macro A2KD_Adept_KB_ENG_3 ALL 0.2158 0.2394 0.1892 LDC-MAX-ALL-Micro A2KD_Adept_KB_ENG_3 0 0.3149 0.4745 0.3786 LDC-MAX-ALL-Micro A2KD_Adept_KB_ENG_3 ALL 0.3149 0.4745 0.3786 LDC-MAX-ALL-Macro A2KD_Adept_KB_ENG_3 0 0.2878 0.3352 0.2670 LDC-MAX-ALL-Macro A2KD_Adept_KB_ENG_3 ALL 0.2878 0.3352 0.2670 LDC-MEAN-ALL-Macro A2KD_Adept_KB_ENG_3 0 0.2385 0.2794 0.2184 LDC-MEAN-ALL-Macro A2KD_Adept_KB_ENG_3 ALL 0.2385 0.2794 0.2184 *ALL-Macro Prec, Recall and F1 refer to mean-precision, mean-recall and mean-F1. --------------------- Composite KB Evaluation (SENTIMENT-SLOTS only): Scores based on confidence values and Average Precision (AP); k=3 justifications allowed: Metric RunID Hop AP SF-ALL-Macro A2KD_Adept_KB_ENG_3 0 0.0000 SF-ALL-Macro A2KD_Adept_KB_ENG_3 1 0.0000 SF-ALL-Macro A2KD_Adept_KB_ENG_3 ALL 0.0000 LDC-MEAN-ALL-Macro A2KD_Adept_KB_ENG_3 0 0.0000 LDC-MEAN-ALL-Macro A2KD_Adept_KB_ENG_3 1 0.0000 LDC-MEAN-ALL-Macro A2KD_Adept_KB_ENG_3 ALL 0.0000 (PRIMARY METRIC) *ALL-Macro AP refer to mean of corresponding AP values. Scores based on confidence values and Average Precision (AP); only k=1 justification allowed: Metric RunID Hop AP SF-ALL-Macro A2KD_Adept_KB_ENG_3 0 0.0000 SF-ALL-Macro A2KD_Adept_KB_ENG_3 1 0.0000 SF-ALL-Macro A2KD_Adept_KB_ENG_3 ALL 0.0000 LDC-MEAN-ALL-Macro A2KD_Adept_KB_ENG_3 0 0.0000 LDC-MEAN-ALL-Macro A2KD_Adept_KB_ENG_3 1 0.0000 LDC-MEAN-ALL-Macro A2KD_Adept_KB_ENG_3 ALL 0.0000 *ALL-Macro AP refer to mean of corresponding AP values. Scores based on P/R/F1; only k=1 justification allowed: Metric RunID Hop Prec Recall F1 SF-ALL-Micro A2KD_Adept_KB_ENG_3 0 0.0000 0.0000 0.0000 SF-ALL-Micro A2KD_Adept_KB_ENG_3 1 0.0000 0.0000 0.0000 SF-ALL-Micro A2KD_Adept_KB_ENG_3 ALL 0.0000 0.0000 0.0000 SF-ALL-Macro A2KD_Adept_KB_ENG_3 0 0.0000 0.0000 0.0000 SF-ALL-Macro A2KD_Adept_KB_ENG_3 1 0.0000 0.0000 0.0000 SF-ALL-Macro A2KD_Adept_KB_ENG_3 ALL 0.0000 0.0000 0.0000 LDC-MAX-ALL-Micro A2KD_Adept_KB_ENG_3 0 0.0000 0.0000 0.0000 LDC-MAX-ALL-Micro A2KD_Adept_KB_ENG_3 1 0.0000 0.0000 0.0000 LDC-MAX-ALL-Micro A2KD_Adept_KB_ENG_3 ALL 0.0000 0.0000 0.0000 LDC-MAX-ALL-Macro A2KD_Adept_KB_ENG_3 0 0.0000 0.0000 0.0000 LDC-MAX-ALL-Macro A2KD_Adept_KB_ENG_3 1 0.0000 0.0000 0.0000 LDC-MAX-ALL-Macro A2KD_Adept_KB_ENG_3 ALL 0.0000 0.0000 0.0000 LDC-MEAN-ALL-Macro A2KD_Adept_KB_ENG_3 0 0.0000 0.0000 0.0000 LDC-MEAN-ALL-Macro A2KD_Adept_KB_ENG_3 1 0.0000 0.0000 0.0000 LDC-MEAN-ALL-Macro A2KD_Adept_KB_ENG_3 ALL 0.0000 0.0000 0.0000 *ALL-Macro Prec, Recall and F1 refer to mean-precision, mean-recall and mean-F1. ************************************************************* Run ID: A2KD_Adept_KB_ENG_4 Did the run access the live Web during the evaluation window: No Did the run extract relations from the Cold Start source corpus: Yes Did the run generate meaningful confidence values: Yes Did the run attempt the full Entity Discovery and Linking task: Yes Did the run attempt the full Event Nugget Detection and Coreference task: No Did the run include SF relations: Yes Did the run attempt the Event Argument Extraction and Linking task: Yes Did the run include Sentiment relations: No --------------------- Composite KB Evaluation (All slot types): Scores based on confidence values and Average Precision (AP); k=3 justifications allowed: Metric RunID Hop AP SF-ALL-Macro A2KD_Adept_KB_ENG_4 0 0.0954 SF-ALL-Macro A2KD_Adept_KB_ENG_4 1 0.0110 SF-ALL-Macro A2KD_Adept_KB_ENG_4 ALL 0.0888 LDC-MEAN-ALL-Macro A2KD_Adept_KB_ENG_4 0 0.1038 LDC-MEAN-ALL-Macro A2KD_Adept_KB_ENG_4 1 0.0125 LDC-MEAN-ALL-Macro A2KD_Adept_KB_ENG_4 ALL 0.0963 (PRIMARY METRIC) *ALL-Macro AP refer to mean of corresponding AP values. Scores based on confidence values and Average Precision (AP); only k=1 justification allowed: Metric RunID Hop AP SF-ALL-Macro A2KD_Adept_KB_ENG_4 0 0.1296 SF-ALL-Macro A2KD_Adept_KB_ENG_4 1 0.0230 SF-ALL-Macro A2KD_Adept_KB_ENG_4 ALL 0.1235 LDC-MEAN-ALL-Macro A2KD_Adept_KB_ENG_4 0 0.1421 LDC-MEAN-ALL-Macro A2KD_Adept_KB_ENG_4 1 0.0258 LDC-MEAN-ALL-Macro A2KD_Adept_KB_ENG_4 ALL 0.1348 *ALL-Macro AP refer to mean of corresponding AP values. Scores based on P/R/F1; only k=1 justification allowed: Metric RunID Hop Prec Recall F1 SF-ALL-Micro A2KD_Adept_KB_ENG_4 0 0.2454 0.1823 0.2092 SF-ALL-Micro A2KD_Adept_KB_ENG_4 1 0.0111 0.0494 0.0181 SF-ALL-Micro A2KD_Adept_KB_ENG_4 ALL 0.0959 0.1521 0.1176 SF-ALL-Macro A2KD_Adept_KB_ENG_4 0 0.1413 0.1545 0.1273 SF-ALL-Macro A2KD_Adept_KB_ENG_4 1 0.0385 0.0418 0.0378 SF-ALL-Macro A2KD_Adept_KB_ENG_4 ALL 0.1059 0.1157 0.0965 LDC-MAX-ALL-Micro A2KD_Adept_KB_ENG_4 0 0.2614 0.2379 0.2491 LDC-MAX-ALL-Micro A2KD_Adept_KB_ENG_4 1 0.0085 0.0540 0.0147 LDC-MAX-ALL-Micro A2KD_Adept_KB_ENG_4 ALL 0.0951 0.1981 0.1285 LDC-MAX-ALL-Macro A2KD_Adept_KB_ENG_4 0 0.1800 0.2004 0.1664 LDC-MAX-ALL-Macro A2KD_Adept_KB_ENG_4 1 0.0414 0.0459 0.0412 LDC-MAX-ALL-Macro A2KD_Adept_KB_ENG_4 ALL 0.1326 0.1476 0.1236 LDC-MEAN-ALL-Macro A2KD_Adept_KB_ENG_4 0 0.1534 0.1731 0.1419 LDC-MEAN-ALL-Macro A2KD_Adept_KB_ENG_4 1 0.0382 0.0426 0.0379 LDC-MEAN-ALL-Macro A2KD_Adept_KB_ENG_4 ALL 0.1140 0.1285 0.1063 *ALL-Macro Prec, Recall and F1 refer to mean-precision, mean-recall and mean-F1. --------------------- Composite KB Evaluation (SF-SLOTS only): Scores based on confidence values and Average Precision (AP); k=3 justifications allowed: Metric RunID Hop AP SF-ALL-Macro A2KD_Adept_KB_ENG_4 0 0.0907 SF-ALL-Macro A2KD_Adept_KB_ENG_4 1 0.0309 SF-ALL-Macro A2KD_Adept_KB_ENG_4 ALL 0.0753 LDC-MEAN-ALL-Macro A2KD_Adept_KB_ENG_4 0 0.0969 LDC-MEAN-ALL-Macro A2KD_Adept_KB_ENG_4 1 0.0330 LDC-MEAN-ALL-Macro A2KD_Adept_KB_ENG_4 ALL 0.0807 (PRIMARY METRIC) *ALL-Macro AP refer to mean of corresponding AP values. Scores based on confidence values and Average Precision (AP); only k=1 justification allowed: Metric RunID Hop AP SF-ALL-Macro A2KD_Adept_KB_ENG_4 0 0.1371 SF-ALL-Macro A2KD_Adept_KB_ENG_4 1 0.0460 SF-ALL-Macro A2KD_Adept_KB_ENG_4 ALL 0.1230 LDC-MEAN-ALL-Macro A2KD_Adept_KB_ENG_4 0 0.1456 LDC-MEAN-ALL-Macro A2KD_Adept_KB_ENG_4 1 0.0453 LDC-MEAN-ALL-Macro A2KD_Adept_KB_ENG_4 ALL 0.1299 *ALL-Macro AP refer to mean of corresponding AP values. Scores based on P/R/F1; only k=1 justification allowed: Metric RunID Hop Prec Recall F1 SF-ALL-Micro A2KD_Adept_KB_ENG_4 0 0.1705 0.1019 0.1276 SF-ALL-Micro A2KD_Adept_KB_ENG_4 1 0.0123 0.0820 0.0214 SF-ALL-Micro A2KD_Adept_KB_ENG_4 ALL 0.0473 0.0971 0.0636 SF-ALL-Macro A2KD_Adept_KB_ENG_4 0 0.1384 0.1485 0.1292 SF-ALL-Macro A2KD_Adept_KB_ENG_4 1 0.0633 0.0693 0.0621 SF-ALL-Macro A2KD_Adept_KB_ENG_4 ALL 0.1087 0.1171 0.1026 LDC-MAX-ALL-Micro A2KD_Adept_KB_ENG_4 0 0.2072 0.1202 0.1522 LDC-MAX-ALL-Micro A2KD_Adept_KB_ENG_4 1 0.0266 0.0823 0.0402 LDC-MAX-ALL-Micro A2KD_Adept_KB_ENG_4 ALL 0.0958 0.1114 0.1031 LDC-MAX-ALL-Macro A2KD_Adept_KB_ENG_4 0 0.1657 0.1730 0.1525 LDC-MAX-ALL-Macro A2KD_Adept_KB_ENG_4 1 0.0637 0.0715 0.0632 LDC-MAX-ALL-Macro A2KD_Adept_KB_ENG_4 ALL 0.1261 0.1336 0.1178 LDC-MEAN-ALL-Macro A2KD_Adept_KB_ENG_4 0 0.1464 0.1572 0.1372 LDC-MEAN-ALL-Macro A2KD_Adept_KB_ENG_4 1 0.0579 0.0658 0.0574 LDC-MEAN-ALL-Macro A2KD_Adept_KB_ENG_4 ALL 0.1120 0.1217 0.1062 *ALL-Macro Prec, Recall and F1 refer to mean-precision, mean-recall and mean-F1. --------------------- Composite KB Evaluation (EVENT-SLOTS only): Scores based on confidence values and Average Precision (AP); k=3 justifications allowed: SUMMARY: This section provides summary of AP scores Metric RunID Hop AP SF-ALL-Macro A2KD_Adept_KB_ENG_4 0 0.1490 SF-ALL-Macro A2KD_Adept_KB_ENG_4 ALL 0.1490 LDC-MEAN-ALL-Macro A2KD_Adept_KB_ENG_4 0 0.1644 LDC-MEAN-ALL-Macro A2KD_Adept_KB_ENG_4 ALL 0.1644 (PRIMARY METRIC) *ALL-Macro AP refer to mean of corresponding AP values. Scores based on confidence values and Average Precision (AP); only k=1 justification allowed: SUMMARY: This section provides summary of AP scores Metric RunID Hop AP SF-ALL-Macro A2KD_Adept_KB_ENG_4 0 0.1863 SF-ALL-Macro A2KD_Adept_KB_ENG_4 ALL 0.1863 LDC-MEAN-ALL-Macro A2KD_Adept_KB_ENG_4 0 0.2082 LDC-MEAN-ALL-Macro A2KD_Adept_KB_ENG_4 ALL 0.2082 *ALL-Macro AP refer to mean of corresponding AP values. Scores based on P/R/F1; only k=1 justification allowed: Metric RunID Hop Prec Recall F1 SF-ALL-Micro A2KD_Adept_KB_ENG_4 0 0.2950 0.3577 0.3234 SF-ALL-Micro A2KD_Adept_KB_ENG_4 ALL 0.2950 0.3577 0.3234 SF-ALL-Macro A2KD_Adept_KB_ENG_4 0 0.2158 0.2394 0.1892 SF-ALL-Macro A2KD_Adept_KB_ENG_4 ALL 0.2158 0.2394 0.1892 LDC-MAX-ALL-Micro A2KD_Adept_KB_ENG_4 0 0.3133 0.4745 0.3774 LDC-MAX-ALL-Micro A2KD_Adept_KB_ENG_4 ALL 0.3133 0.4745 0.3774 LDC-MAX-ALL-Macro A2KD_Adept_KB_ENG_4 0 0.2878 0.3352 0.2670 LDC-MAX-ALL-Macro A2KD_Adept_KB_ENG_4 ALL 0.2878 0.3352 0.2670 LDC-MEAN-ALL-Macro A2KD_Adept_KB_ENG_4 0 0.2385 0.2794 0.2184 LDC-MEAN-ALL-Macro A2KD_Adept_KB_ENG_4 ALL 0.2385 0.2794 0.2184 *ALL-Macro Prec, Recall and F1 refer to mean-precision, mean-recall and mean-F1. --------------------- Composite KB Evaluation (SENTIMENT-SLOTS only): Scores based on confidence values and Average Precision (AP); k=3 justifications allowed: Metric RunID Hop AP SF-ALL-Macro A2KD_Adept_KB_ENG_4 0 0.0000 SF-ALL-Macro A2KD_Adept_KB_ENG_4 1 0.0000 SF-ALL-Macro A2KD_Adept_KB_ENG_4 ALL 0.0000 LDC-MEAN-ALL-Macro A2KD_Adept_KB_ENG_4 0 0.0000 LDC-MEAN-ALL-Macro A2KD_Adept_KB_ENG_4 1 0.0000 LDC-MEAN-ALL-Macro A2KD_Adept_KB_ENG_4 ALL 0.0000 (PRIMARY METRIC) *ALL-Macro AP refer to mean of corresponding AP values. Scores based on confidence values and Average Precision (AP); only k=1 justification allowed: Metric RunID Hop AP SF-ALL-Macro A2KD_Adept_KB_ENG_4 0 0.0000 SF-ALL-Macro A2KD_Adept_KB_ENG_4 1 0.0000 SF-ALL-Macro A2KD_Adept_KB_ENG_4 ALL 0.0000 LDC-MEAN-ALL-Macro A2KD_Adept_KB_ENG_4 0 0.0000 LDC-MEAN-ALL-Macro A2KD_Adept_KB_ENG_4 1 0.0000 LDC-MEAN-ALL-Macro A2KD_Adept_KB_ENG_4 ALL 0.0000 *ALL-Macro AP refer to mean of corresponding AP values. Scores based on P/R/F1; only k=1 justification allowed: Metric RunID Hop Prec Recall F1 SF-ALL-Micro A2KD_Adept_KB_ENG_4 0 0.0000 0.0000 0.0000 SF-ALL-Micro A2KD_Adept_KB_ENG_4 1 0.0000 0.0000 0.0000 SF-ALL-Micro A2KD_Adept_KB_ENG_4 ALL 0.0000 0.0000 0.0000 SF-ALL-Macro A2KD_Adept_KB_ENG_4 0 0.0000 0.0000 0.0000 SF-ALL-Macro A2KD_Adept_KB_ENG_4 1 0.0000 0.0000 0.0000 SF-ALL-Macro A2KD_Adept_KB_ENG_4 ALL 0.0000 0.0000 0.0000 LDC-MAX-ALL-Micro A2KD_Adept_KB_ENG_4 0 0.0000 0.0000 0.0000 LDC-MAX-ALL-Micro A2KD_Adept_KB_ENG_4 1 0.0000 0.0000 0.0000 LDC-MAX-ALL-Micro A2KD_Adept_KB_ENG_4 ALL 0.0000 0.0000 0.0000 LDC-MAX-ALL-Macro A2KD_Adept_KB_ENG_4 0 0.0000 0.0000 0.0000 LDC-MAX-ALL-Macro A2KD_Adept_KB_ENG_4 1 0.0000 0.0000 0.0000 LDC-MAX-ALL-Macro A2KD_Adept_KB_ENG_4 ALL 0.0000 0.0000 0.0000 LDC-MEAN-ALL-Macro A2KD_Adept_KB_ENG_4 0 0.0000 0.0000 0.0000 LDC-MEAN-ALL-Macro A2KD_Adept_KB_ENG_4 1 0.0000 0.0000 0.0000 LDC-MEAN-ALL-Macro A2KD_Adept_KB_ENG_4 ALL 0.0000 0.0000 0.0000 *ALL-Macro Prec, Recall and F1 refer to mean-precision, mean-recall and mean-F1.