=========================================================================== TAC KBP 2017 CROSS-LINGUAL KB CONSTRUCTION: COMPOSITE KB EVALUATION RESULTS =========================================================================== Team ID: hltcoe Organization: Human Language Technology Center of Excellence ************************************************************* Run ID: hltcoe_KB_XLING_1 Did the run access the live Web during the evaluation window: No Did the run extract relations from the Cold Start source corpus: Yes Did the run generate meaningful confidence values: Yes Run number of the English CSKB system that is most closely configured to the English component of this run: 1 Run number of the Spanish CSKB system that is most closely configured to the Spanish component of this run: NA Run number of the Chinese CSKB system that is most closely configured to the Chinese component of this run: 1 Did the run attempt the full Entity Discovery and Linking task: Yes Did the run attempt the full Event Nugget Detection and Coreference task: No Did the run include SF relations: Yes Did the run attempt the Event Argument Extraction and Linking task: No Did the run include Sentiment relations: No --------------------- Composite KB Evaluation (All slot types): Scores based on confidence values and Average Precision (AP); k=3 justifications allowed: Metric RunID Hop AP SF-ALL-Macro hltcoe_KB_XLING_1 0 0.0385 SF-ALL-Macro hltcoe_KB_XLING_1 1 0.0033 SF-ALL-Macro hltcoe_KB_XLING_1 ALL 0.0280 LDC-MEAN-ALL-Macro hltcoe_KB_XLING_1 0 0.0377 LDC-MEAN-ALL-Macro hltcoe_KB_XLING_1 1 0.0033 LDC-MEAN-ALL-Macro hltcoe_KB_XLING_1 ALL 0.0274 (PRIMARY METRIC) *ALL-Macro AP refer to mean of corresponding AP values. Scores based on confidence values and Average Precision (AP); only k=1 justification allowed: Metric RunID Hop AP SF-ALL-Macro hltcoe_KB_XLING_1 0 0.0437 SF-ALL-Macro hltcoe_KB_XLING_1 1 0.0023 SF-ALL-Macro hltcoe_KB_XLING_1 ALL 0.0316 LDC-MEAN-ALL-Macro hltcoe_KB_XLING_1 0 0.0428 LDC-MEAN-ALL-Macro hltcoe_KB_XLING_1 1 0.0022 LDC-MEAN-ALL-Macro hltcoe_KB_XLING_1 ALL 0.0310 *ALL-Macro AP refer to mean of corresponding AP values. Scores based on P/R/F1; only k=1 justification allowed: Metric RunID Hop Prec Recall F1 SF-ALL-Micro hltcoe_KB_XLING_1 0 0.2584 0.0469 0.0794 SF-ALL-Micro hltcoe_KB_XLING_1 1 0.0403 0.0157 0.0226 SF-ALL-Micro hltcoe_KB_XLING_1 ALL 0.1561 0.0378 0.0609 SF-ALL-Macro hltcoe_KB_XLING_1 0 0.0533 0.0496 0.0466 SF-ALL-Macro hltcoe_KB_XLING_1 1 0.0202 0.0208 0.0196 SF-ALL-Macro hltcoe_KB_XLING_1 ALL 0.0388 0.0370 0.0348 LDC-MAX-ALL-Micro hltcoe_KB_XLING_1 0 0.2714 0.1093 0.1559 LDC-MAX-ALL-Micro hltcoe_KB_XLING_1 1 0.1293 0.0390 0.0599 LDC-MAX-ALL-Micro hltcoe_KB_XLING_1 ALL 0.2379 0.0888 0.1293 LDC-MAX-ALL-Macro hltcoe_KB_XLING_1 0 0.1217 0.1113 0.1057 LDC-MAX-ALL-Macro hltcoe_KB_XLING_1 1 0.0492 0.0500 0.0474 LDC-MAX-ALL-Macro hltcoe_KB_XLING_1 ALL 0.0901 0.0846 0.0803 LDC-MEAN-ALL-Macro hltcoe_KB_XLING_1 0 0.0522 0.0486 0.0456 LDC-MEAN-ALL-Macro hltcoe_KB_XLING_1 1 0.0200 0.0206 0.0193 LDC-MEAN-ALL-Macro hltcoe_KB_XLING_1 ALL 0.0382 0.0364 0.0342 *ALL-Macro Prec, Recall and F1 refer to mean-precision, mean-recall and mean-F1. --------------------- Composite KB Evaluation (SF-SLOTS only): Scores based on confidence values and Average Precision (AP); k=3 justifications allowed: Metric RunID Hop AP SF-ALL-Macro hltcoe_KB_XLING_1 0 0.0918 SF-ALL-Macro hltcoe_KB_XLING_1 1 0.0094 SF-ALL-Macro hltcoe_KB_XLING_1 ALL 0.0668 LDC-MEAN-ALL-Macro hltcoe_KB_XLING_1 0 0.0908 LDC-MEAN-ALL-Macro hltcoe_KB_XLING_1 1 0.0094 LDC-MEAN-ALL-Macro hltcoe_KB_XLING_1 ALL 0.0660 (PRIMARY METRIC) *ALL-Macro AP refer to mean of corresponding AP values. Scores based on confidence values and Average Precision (AP); only k=1 justification allowed: Metric RunID Hop AP SF-ALL-Macro hltcoe_KB_XLING_1 0 0.1042 SF-ALL-Macro hltcoe_KB_XLING_1 1 0.0064 SF-ALL-Macro hltcoe_KB_XLING_1 ALL 0.0755 LDC-MEAN-ALL-Macro hltcoe_KB_XLING_1 0 0.1030 LDC-MEAN-ALL-Macro hltcoe_KB_XLING_1 1 0.0064 LDC-MEAN-ALL-Macro hltcoe_KB_XLING_1 ALL 0.0746 *ALL-Macro AP refer to mean of corresponding AP values. Scores based on P/R/F1; only k=1 justification allowed: Metric RunID Hop Prec Recall F1 SF-ALL-Micro hltcoe_KB_XLING_1 0 0.2584 0.1055 0.1498 SF-ALL-Micro hltcoe_KB_XLING_1 1 0.0403 0.0432 0.0417 SF-ALL-Micro hltcoe_KB_XLING_1 ALL 0.1561 0.0898 0.1140 SF-ALL-Macro hltcoe_KB_XLING_1 0 0.1272 0.1184 0.1112 SF-ALL-Macro hltcoe_KB_XLING_1 1 0.0396 0.0407 0.0383 SF-ALL-Macro hltcoe_KB_XLING_1 ALL 0.0845 0.0806 0.0757 LDC-MAX-ALL-Micro hltcoe_KB_XLING_1 0 0.2718 0.2463 0.2584 LDC-MAX-ALL-Micro hltcoe_KB_XLING_1 1 0.1303 0.1079 0.1181 LDC-MAX-ALL-Micro hltcoe_KB_XLING_1 ALL 0.2385 0.2115 0.2242 LDC-MAX-ALL-Macro hltcoe_KB_XLING_1 0 0.2931 0.2680 0.2545 LDC-MAX-ALL-Macro hltcoe_KB_XLING_1 1 0.0974 0.0990 0.0939 LDC-MAX-ALL-Macro hltcoe_KB_XLING_1 ALL 0.1984 0.1862 0.1768 LDC-MEAN-ALL-Macro hltcoe_KB_XLING_1 0 0.1257 0.1171 0.1099 LDC-MEAN-ALL-Macro hltcoe_KB_XLING_1 1 0.0396 0.0407 0.0383 LDC-MEAN-ALL-Macro hltcoe_KB_XLING_1 ALL 0.0840 0.0801 0.0753 *ALL-Macro Prec, Recall and F1 refer to mean-precision, mean-recall and mean-F1. --------------------- Composite KB Evaluation (EVENT-SLOTS only): Scores based on confidence values and Average Precision (AP); k=3 justifications allowed: SUMMARY: This section provides summary of AP scores Metric RunID Hop AP SF-ALL-Macro hltcoe_KB_XLING_1 0 0.0000 SF-ALL-Macro hltcoe_KB_XLING_1 ALL 0.0000 LDC-MEAN-ALL-Macro hltcoe_KB_XLING_1 0 0.0000 LDC-MEAN-ALL-Macro hltcoe_KB_XLING_1 ALL 0.0000 (PRIMARY METRIC) *ALL-Macro AP refer to mean of corresponding AP values. Scores based on confidence values and Average Precision (AP); only k=1 justification allowed: SUMMARY: This section provides summary of AP scores Metric RunID Hop AP SF-ALL-Macro hltcoe_KB_XLING_1 0 0.0000 SF-ALL-Macro hltcoe_KB_XLING_1 ALL 0.0000 LDC-MEAN-ALL-Macro hltcoe_KB_XLING_1 0 0.0000 LDC-MEAN-ALL-Macro hltcoe_KB_XLING_1 ALL 0.0000 *ALL-Macro AP refer to mean of corresponding AP values. Scores based on P/R/F1; only k=1 justification allowed: Metric RunID Hop Prec Recall F1 SF-ALL-Micro hltcoe_KB_XLING_1 0 0.0000 0.0000 0.0000 SF-ALL-Micro hltcoe_KB_XLING_1 ALL 0.0000 0.0000 0.0000 SF-ALL-Macro hltcoe_KB_XLING_1 0 0.0000 0.0000 0.0000 SF-ALL-Macro hltcoe_KB_XLING_1 ALL 0.0000 0.0000 0.0000 LDC-MAX-ALL-Micro hltcoe_KB_XLING_1 0 0.0000 0.0000 0.0000 LDC-MAX-ALL-Micro hltcoe_KB_XLING_1 ALL 0.0000 0.0000 0.0000 LDC-MAX-ALL-Macro hltcoe_KB_XLING_1 0 0.0000 0.0000 0.0000 LDC-MAX-ALL-Macro hltcoe_KB_XLING_1 ALL 0.0000 0.0000 0.0000 LDC-MEAN-ALL-Macro hltcoe_KB_XLING_1 0 0.0000 0.0000 0.0000 LDC-MEAN-ALL-Macro hltcoe_KB_XLING_1 ALL 0.0000 0.0000 0.0000 *ALL-Macro Prec, Recall and F1 refer to mean-precision, mean-recall and mean-F1. --------------------- Composite KB Evaluation (SENTIMENT-SLOTS only): Scores based on confidence values and Average Precision (AP); k=3 justifications allowed: Metric RunID Hop AP SF-ALL-Macro hltcoe_KB_XLING_1 0 0.0000 SF-ALL-Macro hltcoe_KB_XLING_1 1 0.0000 SF-ALL-Macro hltcoe_KB_XLING_1 ALL 0.0000 LDC-MEAN-ALL-Macro hltcoe_KB_XLING_1 0 0.0000 LDC-MEAN-ALL-Macro hltcoe_KB_XLING_1 1 0.0000 LDC-MEAN-ALL-Macro hltcoe_KB_XLING_1 ALL 0.0000 (PRIMARY METRIC) *ALL-Macro AP refer to mean of corresponding AP values. Scores based on confidence values and Average Precision (AP); only k=1 justification allowed: Metric RunID Hop AP SF-ALL-Macro hltcoe_KB_XLING_1 0 0.0000 SF-ALL-Macro hltcoe_KB_XLING_1 1 0.0000 SF-ALL-Macro hltcoe_KB_XLING_1 ALL 0.0000 LDC-MEAN-ALL-Macro hltcoe_KB_XLING_1 0 0.0000 LDC-MEAN-ALL-Macro hltcoe_KB_XLING_1 1 0.0000 LDC-MEAN-ALL-Macro hltcoe_KB_XLING_1 ALL 0.0000 *ALL-Macro AP refer to mean of corresponding AP values. Scores based on P/R/F1; only k=1 justification allowed: Metric RunID Hop Prec Recall F1 SF-ALL-Micro hltcoe_KB_XLING_1 0 0.0000 0.0000 0.0000 SF-ALL-Micro hltcoe_KB_XLING_1 1 0.0000 0.0000 0.0000 SF-ALL-Micro hltcoe_KB_XLING_1 ALL 0.0000 0.0000 0.0000 SF-ALL-Macro hltcoe_KB_XLING_1 0 0.0000 0.0000 0.0000 SF-ALL-Macro hltcoe_KB_XLING_1 1 0.0000 0.0000 0.0000 SF-ALL-Macro hltcoe_KB_XLING_1 ALL 0.0000 0.0000 0.0000 LDC-MAX-ALL-Micro hltcoe_KB_XLING_1 0 0.0000 0.0000 0.0000 LDC-MAX-ALL-Micro hltcoe_KB_XLING_1 1 0.0000 0.0000 0.0000 LDC-MAX-ALL-Micro hltcoe_KB_XLING_1 ALL 0.0000 0.0000 0.0000 LDC-MAX-ALL-Macro hltcoe_KB_XLING_1 0 0.0000 0.0000 0.0000 LDC-MAX-ALL-Macro hltcoe_KB_XLING_1 1 0.0000 0.0000 0.0000 LDC-MAX-ALL-Macro hltcoe_KB_XLING_1 ALL 0.0000 0.0000 0.0000 LDC-MEAN-ALL-Macro hltcoe_KB_XLING_1 0 0.0000 0.0000 0.0000 LDC-MEAN-ALL-Macro hltcoe_KB_XLING_1 1 0.0000 0.0000 0.0000 LDC-MEAN-ALL-Macro hltcoe_KB_XLING_1 ALL 0.0000 0.0000 0.0000 *ALL-Macro Prec, Recall and F1 refer to mean-precision, mean-recall and mean-F1. ************************************************************* Run ID: hltcoe_KB_XLING_2 Did the run access the live Web during the evaluation window: No Did the run extract relations from the Cold Start source corpus: Yes Did the run generate meaningful confidence values: Yes Run number of the English CSKB system that is most closely configured to the English component of this run: 1 Run number of the Spanish CSKB system that is most closely configured to the Spanish component of this run: NA Run number of the Chinese CSKB system that is most closely configured to the Chinese component of this run: 1 Did the run attempt the full Entity Discovery and Linking task: Yes Did the run attempt the full Event Nugget Detection and Coreference task: No Did the run include SF relations: Yes Did the run attempt the Event Argument Extraction and Linking task: No Did the run include Sentiment relations: No --------------------- Composite KB Evaluation (All slot types): Scores based on confidence values and Average Precision (AP); k=3 justifications allowed: Metric RunID Hop AP SF-ALL-Macro hltcoe_KB_XLING_2 0 0.0384 SF-ALL-Macro hltcoe_KB_XLING_2 1 0.0032 SF-ALL-Macro hltcoe_KB_XLING_2 ALL 0.0279 LDC-MEAN-ALL-Macro hltcoe_KB_XLING_2 0 0.0376 LDC-MEAN-ALL-Macro hltcoe_KB_XLING_2 1 0.0032 LDC-MEAN-ALL-Macro hltcoe_KB_XLING_2 ALL 0.0273 (PRIMARY METRIC) *ALL-Macro AP refer to mean of corresponding AP values. Scores based on confidence values and Average Precision (AP); only k=1 justification allowed: Metric RunID Hop AP SF-ALL-Macro hltcoe_KB_XLING_2 0 0.0429 SF-ALL-Macro hltcoe_KB_XLING_2 1 0.0022 SF-ALL-Macro hltcoe_KB_XLING_2 ALL 0.0308 LDC-MEAN-ALL-Macro hltcoe_KB_XLING_2 0 0.0420 LDC-MEAN-ALL-Macro hltcoe_KB_XLING_2 1 0.0021 LDC-MEAN-ALL-Macro hltcoe_KB_XLING_2 ALL 0.0302 *ALL-Macro AP refer to mean of corresponding AP values. Scores based on P/R/F1; only k=1 justification allowed: Metric RunID Hop Prec Recall F1 SF-ALL-Micro hltcoe_KB_XLING_2 0 0.2603 0.0469 0.0795 SF-ALL-Micro hltcoe_KB_XLING_2 1 0.0380 0.0152 0.0218 SF-ALL-Micro hltcoe_KB_XLING_2 ALL 0.1540 0.0377 0.0605 SF-ALL-Macro hltcoe_KB_XLING_2 0 0.0528 0.0496 0.0465 SF-ALL-Macro hltcoe_KB_XLING_2 1 0.0192 0.0200 0.0187 SF-ALL-Macro hltcoe_KB_XLING_2 ALL 0.0381 0.0367 0.0343 LDC-MAX-ALL-Micro hltcoe_KB_XLING_2 0 0.2748 0.1103 0.1574 LDC-MAX-ALL-Micro hltcoe_KB_XLING_2 1 0.1635 0.0390 0.0630 LDC-MAX-ALL-Micro hltcoe_KB_XLING_2 ALL 0.2528 0.0894 0.1321 LDC-MAX-ALL-Macro hltcoe_KB_XLING_2 0 0.1216 0.1114 0.1057 LDC-MAX-ALL-Macro hltcoe_KB_XLING_2 1 0.0492 0.0500 0.0474 LDC-MAX-ALL-Macro hltcoe_KB_XLING_2 ALL 0.0901 0.0847 0.0803 LDC-MEAN-ALL-Macro hltcoe_KB_XLING_2 0 0.0518 0.0486 0.0455 LDC-MEAN-ALL-Macro hltcoe_KB_XLING_2 1 0.0190 0.0198 0.0185 LDC-MEAN-ALL-Macro hltcoe_KB_XLING_2 ALL 0.0375 0.0361 0.0337 *ALL-Macro Prec, Recall and F1 refer to mean-precision, mean-recall and mean-F1. --------------------- Composite KB Evaluation (SF-SLOTS only): Scores based on confidence values and Average Precision (AP); k=3 justifications allowed: Metric RunID Hop AP SF-ALL-Macro hltcoe_KB_XLING_2 0 0.0917 SF-ALL-Macro hltcoe_KB_XLING_2 1 0.0091 SF-ALL-Macro hltcoe_KB_XLING_2 ALL 0.0666 LDC-MEAN-ALL-Macro hltcoe_KB_XLING_2 0 0.0906 LDC-MEAN-ALL-Macro hltcoe_KB_XLING_2 1 0.0091 LDC-MEAN-ALL-Macro hltcoe_KB_XLING_2 ALL 0.0658 (PRIMARY METRIC) *ALL-Macro AP refer to mean of corresponding AP values. Scores based on confidence values and Average Precision (AP); only k=1 justification allowed: Metric RunID Hop AP SF-ALL-Macro hltcoe_KB_XLING_2 0 0.1023 SF-ALL-Macro hltcoe_KB_XLING_2 1 0.0062 SF-ALL-Macro hltcoe_KB_XLING_2 ALL 0.0736 LDC-MEAN-ALL-Macro hltcoe_KB_XLING_2 0 0.1011 LDC-MEAN-ALL-Macro hltcoe_KB_XLING_2 1 0.0062 LDC-MEAN-ALL-Macro hltcoe_KB_XLING_2 ALL 0.0727 *ALL-Macro AP refer to mean of corresponding AP values. Scores based on P/R/F1; only k=1 justification allowed: Metric RunID Hop Prec Recall F1 SF-ALL-Micro hltcoe_KB_XLING_2 0 0.2603 0.1055 0.1501 SF-ALL-Micro hltcoe_KB_XLING_2 1 0.0380 0.0419 0.0399 SF-ALL-Micro hltcoe_KB_XLING_2 ALL 0.1540 0.0895 0.1132 SF-ALL-Macro hltcoe_KB_XLING_2 0 0.1261 0.1184 0.1109 SF-ALL-Macro hltcoe_KB_XLING_2 1 0.0375 0.0392 0.0366 SF-ALL-Macro hltcoe_KB_XLING_2 ALL 0.0830 0.0798 0.0747 LDC-MAX-ALL-Micro hltcoe_KB_XLING_2 0 0.2732 0.2484 0.2602 LDC-MAX-ALL-Micro hltcoe_KB_XLING_2 1 0.1283 0.1079 0.1172 LDC-MAX-ALL-Micro hltcoe_KB_XLING_2 ALL 0.2388 0.2131 0.2252 LDC-MAX-ALL-Macro hltcoe_KB_XLING_2 0 0.2930 0.2684 0.2546 LDC-MAX-ALL-Macro hltcoe_KB_XLING_2 1 0.0974 0.0990 0.0939 LDC-MAX-ALL-Macro hltcoe_KB_XLING_2 ALL 0.1983 0.1865 0.1768 LDC-MEAN-ALL-Macro hltcoe_KB_XLING_2 0 0.1247 0.1170 0.1096 LDC-MEAN-ALL-Macro hltcoe_KB_XLING_2 1 0.0375 0.0392 0.0366 LDC-MEAN-ALL-Macro hltcoe_KB_XLING_2 ALL 0.0825 0.0794 0.0743 *ALL-Macro Prec, Recall and F1 refer to mean-precision, mean-recall and mean-F1. --------------------- Composite KB Evaluation (EVENT-SLOTS only): Scores based on confidence values and Average Precision (AP); k=3 justifications allowed: SUMMARY: This section provides summary of AP scores Metric RunID Hop AP SF-ALL-Macro hltcoe_KB_XLING_2 0 0.0000 SF-ALL-Macro hltcoe_KB_XLING_2 ALL 0.0000 LDC-MEAN-ALL-Macro hltcoe_KB_XLING_2 0 0.0000 LDC-MEAN-ALL-Macro hltcoe_KB_XLING_2 ALL 0.0000 (PRIMARY METRIC) *ALL-Macro AP refer to mean of corresponding AP values. Scores based on confidence values and Average Precision (AP); only k=1 justification allowed: SUMMARY: This section provides summary of AP scores Metric RunID Hop AP SF-ALL-Macro hltcoe_KB_XLING_2 0 0.0000 SF-ALL-Macro hltcoe_KB_XLING_2 ALL 0.0000 LDC-MEAN-ALL-Macro hltcoe_KB_XLING_2 0 0.0000 LDC-MEAN-ALL-Macro hltcoe_KB_XLING_2 ALL 0.0000 *ALL-Macro AP refer to mean of corresponding AP values. Scores based on P/R/F1; only k=1 justification allowed: Metric RunID Hop Prec Recall F1 SF-ALL-Micro hltcoe_KB_XLING_2 0 0.0000 0.0000 0.0000 SF-ALL-Micro hltcoe_KB_XLING_2 ALL 0.0000 0.0000 0.0000 SF-ALL-Macro hltcoe_KB_XLING_2 0 0.0000 0.0000 0.0000 SF-ALL-Macro hltcoe_KB_XLING_2 ALL 0.0000 0.0000 0.0000 LDC-MAX-ALL-Micro hltcoe_KB_XLING_2 0 0.0000 0.0000 0.0000 LDC-MAX-ALL-Micro hltcoe_KB_XLING_2 ALL 0.0000 0.0000 0.0000 LDC-MAX-ALL-Macro hltcoe_KB_XLING_2 0 0.0000 0.0000 0.0000 LDC-MAX-ALL-Macro hltcoe_KB_XLING_2 ALL 0.0000 0.0000 0.0000 LDC-MEAN-ALL-Macro hltcoe_KB_XLING_2 0 0.0000 0.0000 0.0000 LDC-MEAN-ALL-Macro hltcoe_KB_XLING_2 ALL 0.0000 0.0000 0.0000 *ALL-Macro Prec, Recall and F1 refer to mean-precision, mean-recall and mean-F1. --------------------- Composite KB Evaluation (SENTIMENT-SLOTS only): Scores based on confidence values and Average Precision (AP); k=3 justifications allowed: Metric RunID Hop AP SF-ALL-Macro hltcoe_KB_XLING_2 0 0.0000 SF-ALL-Macro hltcoe_KB_XLING_2 1 0.0000 SF-ALL-Macro hltcoe_KB_XLING_2 ALL 0.0000 LDC-MEAN-ALL-Macro hltcoe_KB_XLING_2 0 0.0000 LDC-MEAN-ALL-Macro hltcoe_KB_XLING_2 1 0.0000 LDC-MEAN-ALL-Macro hltcoe_KB_XLING_2 ALL 0.0000 (PRIMARY METRIC) *ALL-Macro AP refer to mean of corresponding AP values. Scores based on confidence values and Average Precision (AP); only k=1 justification allowed: Metric RunID Hop AP SF-ALL-Macro hltcoe_KB_XLING_2 0 0.0000 SF-ALL-Macro hltcoe_KB_XLING_2 1 0.0000 SF-ALL-Macro hltcoe_KB_XLING_2 ALL 0.0000 LDC-MEAN-ALL-Macro hltcoe_KB_XLING_2 0 0.0000 LDC-MEAN-ALL-Macro hltcoe_KB_XLING_2 1 0.0000 LDC-MEAN-ALL-Macro hltcoe_KB_XLING_2 ALL 0.0000 *ALL-Macro AP refer to mean of corresponding AP values. Scores based on P/R/F1; only k=1 justification allowed: Metric RunID Hop Prec Recall F1 SF-ALL-Micro hltcoe_KB_XLING_2 0 0.0000 0.0000 0.0000 SF-ALL-Micro hltcoe_KB_XLING_2 1 0.0000 0.0000 0.0000 SF-ALL-Micro hltcoe_KB_XLING_2 ALL 0.0000 0.0000 0.0000 SF-ALL-Macro hltcoe_KB_XLING_2 0 0.0000 0.0000 0.0000 SF-ALL-Macro hltcoe_KB_XLING_2 1 0.0000 0.0000 0.0000 SF-ALL-Macro hltcoe_KB_XLING_2 ALL 0.0000 0.0000 0.0000 LDC-MAX-ALL-Micro hltcoe_KB_XLING_2 0 0.0000 0.0000 0.0000 LDC-MAX-ALL-Micro hltcoe_KB_XLING_2 1 0.0000 0.0000 0.0000 LDC-MAX-ALL-Micro hltcoe_KB_XLING_2 ALL 0.0000 0.0000 0.0000 LDC-MAX-ALL-Macro hltcoe_KB_XLING_2 0 0.0000 0.0000 0.0000 LDC-MAX-ALL-Macro hltcoe_KB_XLING_2 1 0.0000 0.0000 0.0000 LDC-MAX-ALL-Macro hltcoe_KB_XLING_2 ALL 0.0000 0.0000 0.0000 LDC-MEAN-ALL-Macro hltcoe_KB_XLING_2 0 0.0000 0.0000 0.0000 LDC-MEAN-ALL-Macro hltcoe_KB_XLING_2 1 0.0000 0.0000 0.0000 LDC-MEAN-ALL-Macro hltcoe_KB_XLING_2 ALL 0.0000 0.0000 0.0000 *ALL-Macro Prec, Recall and F1 refer to mean-precision, mean-recall and mean-F1. ************************************************************* Run ID: hltcoe_KB_XLING_3 Did the run access the live Web during the evaluation window: No Did the run extract relations from the Cold Start source corpus: Yes Did the run generate meaningful confidence values: Yes Run number of the English CSKB system that is most closely configured to the English component of this run: 2 Run number of the Spanish CSKB system that is most closely configured to the Spanish component of this run: NA Run number of the Chinese CSKB system that is most closely configured to the Chinese component of this run: 1 Did the run attempt the full Entity Discovery and Linking task: Yes Did the run attempt the full Event Nugget Detection and Coreference task: No Did the run include SF relations: Yes Did the run attempt the Event Argument Extraction and Linking task: No Did the run include Sentiment relations: No --------------------- Composite KB Evaluation (All slot types): Scores based on confidence values and Average Precision (AP); k=3 justifications allowed: Metric RunID Hop AP SF-ALL-Macro hltcoe_KB_XLING_3 0 0.0373 SF-ALL-Macro hltcoe_KB_XLING_3 1 0.0028 SF-ALL-Macro hltcoe_KB_XLING_3 ALL 0.0266 LDC-MEAN-ALL-Macro hltcoe_KB_XLING_3 0 0.0365 LDC-MEAN-ALL-Macro hltcoe_KB_XLING_3 1 0.0028 LDC-MEAN-ALL-Macro hltcoe_KB_XLING_3 ALL 0.0261 (PRIMARY METRIC) *ALL-Macro AP refer to mean of corresponding AP values. Scores based on confidence values and Average Precision (AP); only k=1 justification allowed: Metric RunID Hop AP SF-ALL-Macro hltcoe_KB_XLING_3 0 0.0431 SF-ALL-Macro hltcoe_KB_XLING_3 1 0.0023 SF-ALL-Macro hltcoe_KB_XLING_3 ALL 0.0311 LDC-MEAN-ALL-Macro hltcoe_KB_XLING_3 0 0.0422 LDC-MEAN-ALL-Macro hltcoe_KB_XLING_3 1 0.0022 LDC-MEAN-ALL-Macro hltcoe_KB_XLING_3 ALL 0.0305 *ALL-Macro AP refer to mean of corresponding AP values. Scores based on P/R/F1; only k=1 justification allowed: Metric RunID Hop Prec Recall F1 SF-ALL-Micro hltcoe_KB_XLING_3 0 0.3054 0.0465 0.0808 SF-ALL-Micro hltcoe_KB_XLING_3 1 0.0382 0.0157 0.0223 SF-ALL-Micro hltcoe_KB_XLING_3 ALL 0.1647 0.0375 0.0611 SF-ALL-Macro hltcoe_KB_XLING_3 0 0.0528 0.0486 0.0458 SF-ALL-Macro hltcoe_KB_XLING_3 1 0.0211 0.0208 0.0198 SF-ALL-Macro hltcoe_KB_XLING_3 ALL 0.0389 0.0365 0.0344 LDC-MAX-ALL-Micro hltcoe_KB_XLING_3 0 0.3090 0.1084 0.1605 LDC-MAX-ALL-Micro hltcoe_KB_XLING_3 1 0.1176 0.0390 0.0586 LDC-MAX-ALL-Micro hltcoe_KB_XLING_3 ALL 0.2553 0.0881 0.1310 LDC-MAX-ALL-Macro hltcoe_KB_XLING_3 0 0.1213 0.1083 0.1039 LDC-MAX-ALL-Macro hltcoe_KB_XLING_3 1 0.0513 0.0500 0.0481 LDC-MAX-ALL-Macro hltcoe_KB_XLING_3 ALL 0.0908 0.0829 0.0796 LDC-MEAN-ALL-Macro hltcoe_KB_XLING_3 0 0.0517 0.0476 0.0449 LDC-MEAN-ALL-Macro hltcoe_KB_XLING_3 1 0.0208 0.0206 0.0196 LDC-MEAN-ALL-Macro hltcoe_KB_XLING_3 ALL 0.0383 0.0358 0.0339 *ALL-Macro Prec, Recall and F1 refer to mean-precision, mean-recall and mean-F1. --------------------- Composite KB Evaluation (SF-SLOTS only): Scores based on confidence values and Average Precision (AP); k=3 justifications allowed: Metric RunID Hop AP SF-ALL-Macro hltcoe_KB_XLING_3 0 0.0889 SF-ALL-Macro hltcoe_KB_XLING_3 1 0.0079 SF-ALL-Macro hltcoe_KB_XLING_3 ALL 0.0636 LDC-MEAN-ALL-Macro hltcoe_KB_XLING_3 0 0.0879 LDC-MEAN-ALL-Macro hltcoe_KB_XLING_3 1 0.0079 LDC-MEAN-ALL-Macro hltcoe_KB_XLING_3 ALL 0.0628 (PRIMARY METRIC) *ALL-Macro AP refer to mean of corresponding AP values. Scores based on confidence values and Average Precision (AP); only k=1 justification allowed: Metric RunID Hop AP SF-ALL-Macro hltcoe_KB_XLING_3 0 0.1028 SF-ALL-Macro hltcoe_KB_XLING_3 1 0.0064 SF-ALL-Macro hltcoe_KB_XLING_3 ALL 0.0742 LDC-MEAN-ALL-Macro hltcoe_KB_XLING_3 0 0.1016 LDC-MEAN-ALL-Macro hltcoe_KB_XLING_3 1 0.0064 LDC-MEAN-ALL-Macro hltcoe_KB_XLING_3 ALL 0.0734 *ALL-Macro AP refer to mean of corresponding AP values. Scores based on P/R/F1; only k=1 justification allowed: Metric RunID Hop Prec Recall F1 SF-ALL-Micro hltcoe_KB_XLING_3 0 0.3054 0.1046 0.1559 SF-ALL-Micro hltcoe_KB_XLING_3 1 0.0382 0.0432 0.0405 SF-ALL-Micro hltcoe_KB_XLING_3 ALL 0.1647 0.0891 0.1157 SF-ALL-Macro hltcoe_KB_XLING_3 0 0.1261 0.1160 0.1093 SF-ALL-Macro hltcoe_KB_XLING_3 1 0.0412 0.0407 0.0388 SF-ALL-Macro hltcoe_KB_XLING_3 ALL 0.0848 0.0794 0.0750 LDC-MAX-ALL-Micro hltcoe_KB_XLING_3 0 0.3176 0.2441 0.2761 LDC-MAX-ALL-Micro hltcoe_KB_XLING_3 1 0.1241 0.1079 0.1154 LDC-MAX-ALL-Micro hltcoe_KB_XLING_3 ALL 0.2643 0.2099 0.2340 LDC-MAX-ALL-Macro hltcoe_KB_XLING_3 0 0.2921 0.2608 0.2503 LDC-MAX-ALL-Macro hltcoe_KB_XLING_3 1 0.1015 0.0990 0.0951 LDC-MAX-ALL-Macro hltcoe_KB_XLING_3 ALL 0.1999 0.1825 0.1752 LDC-MEAN-ALL-Macro hltcoe_KB_XLING_3 0 0.1246 0.1147 0.1080 LDC-MEAN-ALL-Macro hltcoe_KB_XLING_3 1 0.0412 0.0407 0.0388 LDC-MEAN-ALL-Macro hltcoe_KB_XLING_3 ALL 0.0843 0.0789 0.0745 *ALL-Macro Prec, Recall and F1 refer to mean-precision, mean-recall and mean-F1. --------------------- Composite KB Evaluation (EVENT-SLOTS only): Scores based on confidence values and Average Precision (AP); k=3 justifications allowed: SUMMARY: This section provides summary of AP scores Metric RunID Hop AP SF-ALL-Macro hltcoe_KB_XLING_3 0 0.0000 SF-ALL-Macro hltcoe_KB_XLING_3 ALL 0.0000 LDC-MEAN-ALL-Macro hltcoe_KB_XLING_3 0 0.0000 LDC-MEAN-ALL-Macro hltcoe_KB_XLING_3 ALL 0.0000 (PRIMARY METRIC) *ALL-Macro AP refer to mean of corresponding AP values. Scores based on confidence values and Average Precision (AP); only k=1 justification allowed: SUMMARY: This section provides summary of AP scores Metric RunID Hop AP SF-ALL-Macro hltcoe_KB_XLING_3 0 0.0000 SF-ALL-Macro hltcoe_KB_XLING_3 ALL 0.0000 LDC-MEAN-ALL-Macro hltcoe_KB_XLING_3 0 0.0000 LDC-MEAN-ALL-Macro hltcoe_KB_XLING_3 ALL 0.0000 *ALL-Macro AP refer to mean of corresponding AP values. Scores based on P/R/F1; only k=1 justification allowed: Metric RunID Hop Prec Recall F1 SF-ALL-Micro hltcoe_KB_XLING_3 0 0.0000 0.0000 0.0000 SF-ALL-Micro hltcoe_KB_XLING_3 ALL 0.0000 0.0000 0.0000 SF-ALL-Macro hltcoe_KB_XLING_3 0 0.0000 0.0000 0.0000 SF-ALL-Macro hltcoe_KB_XLING_3 ALL 0.0000 0.0000 0.0000 LDC-MAX-ALL-Micro hltcoe_KB_XLING_3 0 0.0000 0.0000 0.0000 LDC-MAX-ALL-Micro hltcoe_KB_XLING_3 ALL 0.0000 0.0000 0.0000 LDC-MAX-ALL-Macro hltcoe_KB_XLING_3 0 0.0000 0.0000 0.0000 LDC-MAX-ALL-Macro hltcoe_KB_XLING_3 ALL 0.0000 0.0000 0.0000 LDC-MEAN-ALL-Macro hltcoe_KB_XLING_3 0 0.0000 0.0000 0.0000 LDC-MEAN-ALL-Macro hltcoe_KB_XLING_3 ALL 0.0000 0.0000 0.0000 *ALL-Macro Prec, Recall and F1 refer to mean-precision, mean-recall and mean-F1. --------------------- Composite KB Evaluation (SENTIMENT-SLOTS only): Scores based on confidence values and Average Precision (AP); k=3 justifications allowed: Metric RunID Hop AP SF-ALL-Macro hltcoe_KB_XLING_3 0 0.0000 SF-ALL-Macro hltcoe_KB_XLING_3 1 0.0000 SF-ALL-Macro hltcoe_KB_XLING_3 ALL 0.0000 LDC-MEAN-ALL-Macro hltcoe_KB_XLING_3 0 0.0000 LDC-MEAN-ALL-Macro hltcoe_KB_XLING_3 1 0.0000 LDC-MEAN-ALL-Macro hltcoe_KB_XLING_3 ALL 0.0000 (PRIMARY METRIC) *ALL-Macro AP refer to mean of corresponding AP values. Scores based on confidence values and Average Precision (AP); only k=1 justification allowed: Metric RunID Hop AP SF-ALL-Macro hltcoe_KB_XLING_3 0 0.0000 SF-ALL-Macro hltcoe_KB_XLING_3 1 0.0000 SF-ALL-Macro hltcoe_KB_XLING_3 ALL 0.0000 LDC-MEAN-ALL-Macro hltcoe_KB_XLING_3 0 0.0000 LDC-MEAN-ALL-Macro hltcoe_KB_XLING_3 1 0.0000 LDC-MEAN-ALL-Macro hltcoe_KB_XLING_3 ALL 0.0000 *ALL-Macro AP refer to mean of corresponding AP values. Scores based on P/R/F1; only k=1 justification allowed: Metric RunID Hop Prec Recall F1 SF-ALL-Micro hltcoe_KB_XLING_3 0 0.0000 0.0000 0.0000 SF-ALL-Micro hltcoe_KB_XLING_3 1 0.0000 0.0000 0.0000 SF-ALL-Micro hltcoe_KB_XLING_3 ALL 0.0000 0.0000 0.0000 SF-ALL-Macro hltcoe_KB_XLING_3 0 0.0000 0.0000 0.0000 SF-ALL-Macro hltcoe_KB_XLING_3 1 0.0000 0.0000 0.0000 SF-ALL-Macro hltcoe_KB_XLING_3 ALL 0.0000 0.0000 0.0000 LDC-MAX-ALL-Micro hltcoe_KB_XLING_3 0 0.0000 0.0000 0.0000 LDC-MAX-ALL-Micro hltcoe_KB_XLING_3 1 0.0000 0.0000 0.0000 LDC-MAX-ALL-Micro hltcoe_KB_XLING_3 ALL 0.0000 0.0000 0.0000 LDC-MAX-ALL-Macro hltcoe_KB_XLING_3 0 0.0000 0.0000 0.0000 LDC-MAX-ALL-Macro hltcoe_KB_XLING_3 1 0.0000 0.0000 0.0000 LDC-MAX-ALL-Macro hltcoe_KB_XLING_3 ALL 0.0000 0.0000 0.0000 LDC-MEAN-ALL-Macro hltcoe_KB_XLING_3 0 0.0000 0.0000 0.0000 LDC-MEAN-ALL-Macro hltcoe_KB_XLING_3 1 0.0000 0.0000 0.0000 LDC-MEAN-ALL-Macro hltcoe_KB_XLING_3 ALL 0.0000 0.0000 0.0000 *ALL-Macro Prec, Recall and F1 refer to mean-precision, mean-recall and mean-F1. ************************************************************* Run ID: hltcoe_KB_XLING_4 Did the run access the live Web during the evaluation window: No Did the run extract relations from the Cold Start source corpus: Yes Did the run generate meaningful confidence values: Yes Run number of the English CSKB system that is most closely configured to the English component of this run: 2 Run number of the Spanish CSKB system that is most closely configured to the Spanish component of this run: NA Run number of the Chinese CSKB system that is most closely configured to the Chinese component of this run: 1 Did the run attempt the full Entity Discovery and Linking task: Yes Did the run attempt the full Event Nugget Detection and Coreference task: No Did the run include SF relations: Yes Did the run attempt the Event Argument Extraction and Linking task: No Did the run include Sentiment relations: No --------------------- Composite KB Evaluation (All slot types): Scores based on confidence values and Average Precision (AP); k=3 justifications allowed: Metric RunID Hop AP SF-ALL-Macro hltcoe_KB_XLING_4 0 0.0381 SF-ALL-Macro hltcoe_KB_XLING_4 1 0.0033 SF-ALL-Macro hltcoe_KB_XLING_4 ALL 0.0275 LDC-MEAN-ALL-Macro hltcoe_KB_XLING_4 0 0.0373 LDC-MEAN-ALL-Macro hltcoe_KB_XLING_4 1 0.0032 LDC-MEAN-ALL-Macro hltcoe_KB_XLING_4 ALL 0.0270 (PRIMARY METRIC) *ALL-Macro AP refer to mean of corresponding AP values. Scores based on confidence values and Average Precision (AP); only k=1 justification allowed: Metric RunID Hop AP SF-ALL-Macro hltcoe_KB_XLING_4 0 0.0434 SF-ALL-Macro hltcoe_KB_XLING_4 1 0.0022 SF-ALL-Macro hltcoe_KB_XLING_4 ALL 0.0314 LDC-MEAN-ALL-Macro hltcoe_KB_XLING_4 0 0.0425 LDC-MEAN-ALL-Macro hltcoe_KB_XLING_4 1 0.0021 LDC-MEAN-ALL-Macro hltcoe_KB_XLING_4 ALL 0.0307 *ALL-Macro AP refer to mean of corresponding AP values. Scores based on P/R/F1; only k=1 justification allowed: Metric RunID Hop Prec Recall F1 SF-ALL-Micro hltcoe_KB_XLING_4 0 0.3298 0.0468 0.0820 SF-ALL-Micro hltcoe_KB_XLING_4 1 0.0378 0.0152 0.0217 SF-ALL-Micro hltcoe_KB_XLING_4 ALL 0.1723 0.0376 0.0617 SF-ALL-Macro hltcoe_KB_XLING_4 0 0.0533 0.0493 0.0464 SF-ALL-Macro hltcoe_KB_XLING_4 1 0.0192 0.0200 0.0187 SF-ALL-Macro hltcoe_KB_XLING_4 ALL 0.0383 0.0365 0.0343 LDC-MAX-ALL-Micro hltcoe_KB_XLING_4 0 0.3127 0.1098 0.1625 LDC-MAX-ALL-Micro hltcoe_KB_XLING_4 1 0.1627 0.0390 0.0629 LDC-MAX-ALL-Micro hltcoe_KB_XLING_4 ALL 0.2797 0.0891 0.1352 LDC-MAX-ALL-Macro hltcoe_KB_XLING_4 0 0.1226 0.1097 0.1052 LDC-MAX-ALL-Macro hltcoe_KB_XLING_4 1 0.0492 0.0500 0.0474 LDC-MAX-ALL-Macro hltcoe_KB_XLING_4 ALL 0.0907 0.0837 0.0801 LDC-MEAN-ALL-Macro hltcoe_KB_XLING_4 0 0.0521 0.0482 0.0454 LDC-MEAN-ALL-Macro hltcoe_KB_XLING_4 1 0.0190 0.0198 0.0185 LDC-MEAN-ALL-Macro hltcoe_KB_XLING_4 ALL 0.0377 0.0359 0.0337 *ALL-Macro Prec, Recall and F1 refer to mean-precision, mean-recall and mean-F1. --------------------- Composite KB Evaluation (SF-SLOTS only): Scores based on confidence values and Average Precision (AP); k=3 justifications allowed: Metric RunID Hop AP SF-ALL-Macro hltcoe_KB_XLING_4 0 0.0909 SF-ALL-Macro hltcoe_KB_XLING_4 1 0.0092 SF-ALL-Macro hltcoe_KB_XLING_4 ALL 0.0657 LDC-MEAN-ALL-Macro hltcoe_KB_XLING_4 0 0.0899 LDC-MEAN-ALL-Macro hltcoe_KB_XLING_4 1 0.0092 LDC-MEAN-ALL-Macro hltcoe_KB_XLING_4 ALL 0.0650 (PRIMARY METRIC) *ALL-Macro AP refer to mean of corresponding AP values. Scores based on confidence values and Average Precision (AP); only k=1 justification allowed: Metric RunID Hop AP SF-ALL-Macro hltcoe_KB_XLING_4 0 0.1037 SF-ALL-Macro hltcoe_KB_XLING_4 1 0.0062 SF-ALL-Macro hltcoe_KB_XLING_4 ALL 0.0748 LDC-MEAN-ALL-Macro hltcoe_KB_XLING_4 0 0.1025 LDC-MEAN-ALL-Macro hltcoe_KB_XLING_4 1 0.0062 LDC-MEAN-ALL-Macro hltcoe_KB_XLING_4 ALL 0.0740 *ALL-Macro AP refer to mean of corresponding AP values. Scores based on P/R/F1; only k=1 justification allowed: Metric RunID Hop Prec Recall F1 SF-ALL-Micro hltcoe_KB_XLING_4 0 0.3298 0.1053 0.1596 SF-ALL-Micro hltcoe_KB_XLING_4 1 0.0378 0.0419 0.0397 SF-ALL-Micro hltcoe_KB_XLING_4 ALL 0.1723 0.0893 0.1176 SF-ALL-Macro hltcoe_KB_XLING_4 0 0.1271 0.1175 0.1107 SF-ALL-Macro hltcoe_KB_XLING_4 1 0.0375 0.0392 0.0366 SF-ALL-Macro hltcoe_KB_XLING_4 ALL 0.0835 0.0794 0.0746 LDC-MAX-ALL-Micro hltcoe_KB_XLING_4 0 0.3135 0.2473 0.2765 LDC-MAX-ALL-Micro hltcoe_KB_XLING_4 1 0.1635 0.1079 0.1300 LDC-MAX-ALL-Micro hltcoe_KB_XLING_4 ALL 0.2806 0.2123 0.2417 LDC-MAX-ALL-Macro hltcoe_KB_XLING_4 0 0.2953 0.2642 0.2535 LDC-MAX-ALL-Macro hltcoe_KB_XLING_4 1 0.0974 0.0990 0.0939 LDC-MAX-ALL-Macro hltcoe_KB_XLING_4 ALL 0.1996 0.1843 0.1762 LDC-MEAN-ALL-Macro hltcoe_KB_XLING_4 0 0.1256 0.1162 0.1094 LDC-MEAN-ALL-Macro hltcoe_KB_XLING_4 1 0.0375 0.0392 0.0366 LDC-MEAN-ALL-Macro hltcoe_KB_XLING_4 ALL 0.0830 0.0789 0.0742 *ALL-Macro Prec, Recall and F1 refer to mean-precision, mean-recall and mean-F1. --------------------- Composite KB Evaluation (EVENT-SLOTS only): Scores based on confidence values and Average Precision (AP); k=3 justifications allowed: SUMMARY: This section provides summary of AP scores Metric RunID Hop AP SF-ALL-Macro hltcoe_KB_XLING_4 0 0.0000 SF-ALL-Macro hltcoe_KB_XLING_4 ALL 0.0000 LDC-MEAN-ALL-Macro hltcoe_KB_XLING_4 0 0.0000 LDC-MEAN-ALL-Macro hltcoe_KB_XLING_4 ALL 0.0000 (PRIMARY METRIC) *ALL-Macro AP refer to mean of corresponding AP values. Scores based on confidence values and Average Precision (AP); only k=1 justification allowed: SUMMARY: This section provides summary of AP scores Metric RunID Hop AP SF-ALL-Macro hltcoe_KB_XLING_4 0 0.0000 SF-ALL-Macro hltcoe_KB_XLING_4 ALL 0.0000 LDC-MEAN-ALL-Macro hltcoe_KB_XLING_4 0 0.0000 LDC-MEAN-ALL-Macro hltcoe_KB_XLING_4 ALL 0.0000 *ALL-Macro AP refer to mean of corresponding AP values. Scores based on P/R/F1; only k=1 justification allowed: Metric RunID Hop Prec Recall F1 SF-ALL-Micro hltcoe_KB_XLING_4 0 0.0000 0.0000 0.0000 SF-ALL-Micro hltcoe_KB_XLING_4 ALL 0.0000 0.0000 0.0000 SF-ALL-Macro hltcoe_KB_XLING_4 0 0.0000 0.0000 0.0000 SF-ALL-Macro hltcoe_KB_XLING_4 ALL 0.0000 0.0000 0.0000 LDC-MAX-ALL-Micro hltcoe_KB_XLING_4 0 0.0000 0.0000 0.0000 LDC-MAX-ALL-Micro hltcoe_KB_XLING_4 ALL 0.0000 0.0000 0.0000 LDC-MAX-ALL-Macro hltcoe_KB_XLING_4 0 0.0000 0.0000 0.0000 LDC-MAX-ALL-Macro hltcoe_KB_XLING_4 ALL 0.0000 0.0000 0.0000 LDC-MEAN-ALL-Macro hltcoe_KB_XLING_4 0 0.0000 0.0000 0.0000 LDC-MEAN-ALL-Macro hltcoe_KB_XLING_4 ALL 0.0000 0.0000 0.0000 *ALL-Macro Prec, Recall and F1 refer to mean-precision, mean-recall and mean-F1. --------------------- Composite KB Evaluation (SENTIMENT-SLOTS only): Scores based on confidence values and Average Precision (AP); k=3 justifications allowed: Metric RunID Hop AP SF-ALL-Macro hltcoe_KB_XLING_4 0 0.0000 SF-ALL-Macro hltcoe_KB_XLING_4 1 0.0000 SF-ALL-Macro hltcoe_KB_XLING_4 ALL 0.0000 LDC-MEAN-ALL-Macro hltcoe_KB_XLING_4 0 0.0000 LDC-MEAN-ALL-Macro hltcoe_KB_XLING_4 1 0.0000 LDC-MEAN-ALL-Macro hltcoe_KB_XLING_4 ALL 0.0000 (PRIMARY METRIC) *ALL-Macro AP refer to mean of corresponding AP values. Scores based on confidence values and Average Precision (AP); only k=1 justification allowed: Metric RunID Hop AP SF-ALL-Macro hltcoe_KB_XLING_4 0 0.0000 SF-ALL-Macro hltcoe_KB_XLING_4 1 0.0000 SF-ALL-Macro hltcoe_KB_XLING_4 ALL 0.0000 LDC-MEAN-ALL-Macro hltcoe_KB_XLING_4 0 0.0000 LDC-MEAN-ALL-Macro hltcoe_KB_XLING_4 1 0.0000 LDC-MEAN-ALL-Macro hltcoe_KB_XLING_4 ALL 0.0000 *ALL-Macro AP refer to mean of corresponding AP values. Scores based on P/R/F1; only k=1 justification allowed: Metric RunID Hop Prec Recall F1 SF-ALL-Micro hltcoe_KB_XLING_4 0 0.0000 0.0000 0.0000 SF-ALL-Micro hltcoe_KB_XLING_4 1 0.0000 0.0000 0.0000 SF-ALL-Micro hltcoe_KB_XLING_4 ALL 0.0000 0.0000 0.0000 SF-ALL-Macro hltcoe_KB_XLING_4 0 0.0000 0.0000 0.0000 SF-ALL-Macro hltcoe_KB_XLING_4 1 0.0000 0.0000 0.0000 SF-ALL-Macro hltcoe_KB_XLING_4 ALL 0.0000 0.0000 0.0000 LDC-MAX-ALL-Micro hltcoe_KB_XLING_4 0 0.0000 0.0000 0.0000 LDC-MAX-ALL-Micro hltcoe_KB_XLING_4 1 0.0000 0.0000 0.0000 LDC-MAX-ALL-Micro hltcoe_KB_XLING_4 ALL 0.0000 0.0000 0.0000 LDC-MAX-ALL-Macro hltcoe_KB_XLING_4 0 0.0000 0.0000 0.0000 LDC-MAX-ALL-Macro hltcoe_KB_XLING_4 1 0.0000 0.0000 0.0000 LDC-MAX-ALL-Macro hltcoe_KB_XLING_4 ALL 0.0000 0.0000 0.0000 LDC-MEAN-ALL-Macro hltcoe_KB_XLING_4 0 0.0000 0.0000 0.0000 LDC-MEAN-ALL-Macro hltcoe_KB_XLING_4 1 0.0000 0.0000 0.0000 LDC-MEAN-ALL-Macro hltcoe_KB_XLING_4 ALL 0.0000 0.0000 0.0000 *ALL-Macro Prec, Recall and F1 refer to mean-precision, mean-recall and mean-F1.