=========================================================================== TAC KBP 2017 CROSS-LINGUAL KB CONSTRUCTION: COMPOSITE KB EVALUATION RESULTS =========================================================================== Team ID: STANFORD Organization: Stanford University ************************************************************* Run ID: STANFORD_KB_XLING_1 Did the run access the live Web during the evaluation window: No Did the run extract relations from the Cold Start source corpus: Yes Did the run generate meaningful confidence values: Yes Run number of the English CSKB system that is most closely configured to the English component of this run: 1 Run number of the Spanish CSKB system that is most closely configured to the Spanish component of this run: 1 Run number of the Chinese CSKB system that is most closely configured to the Chinese component of this run: 1 Did the run attempt the full Entity Discovery and Linking task: No Did the run attempt the full Event Nugget Detection and Coreference task: No Did the run include SF relations: Yes Did the run attempt the Event Argument Extraction and Linking task: No Did the run include Sentiment relations: No --------------------- Composite KB Evaluation (All slot types): Scores based on confidence values and Average Precision (AP); k=3 justifications allowed: Metric RunID Hop AP SF-ALL-Macro STANFORD_KB_XLING_1 0 0.0476 SF-ALL-Macro STANFORD_KB_XLING_1 1 0.0069 SF-ALL-Macro STANFORD_KB_XLING_1 ALL 0.0305 LDC-MEAN-ALL-Macro STANFORD_KB_XLING_1 0 0.0472 LDC-MEAN-ALL-Macro STANFORD_KB_XLING_1 1 0.0068 LDC-MEAN-ALL-Macro STANFORD_KB_XLING_1 ALL 0.0302 (PRIMARY METRIC) *ALL-Macro AP refer to mean of corresponding AP values. Scores based on confidence values and Average Precision (AP); only k=1 justification allowed: Metric RunID Hop AP SF-ALL-Macro STANFORD_KB_XLING_1 0 0.0714 SF-ALL-Macro STANFORD_KB_XLING_1 1 0.0122 SF-ALL-Macro STANFORD_KB_XLING_1 ALL 0.0492 LDC-MEAN-ALL-Macro STANFORD_KB_XLING_1 0 0.0708 LDC-MEAN-ALL-Macro STANFORD_KB_XLING_1 1 0.0120 LDC-MEAN-ALL-Macro STANFORD_KB_XLING_1 ALL 0.0488 *ALL-Macro AP refer to mean of corresponding AP values. Scores based on P/R/F1; only k=1 justification allowed: Metric RunID Hop Prec Recall F1 SF-ALL-Micro STANFORD_KB_XLING_1 0 0.2161 0.0507 0.0822 SF-ALL-Micro STANFORD_KB_XLING_1 1 0.0410 0.0305 0.0350 SF-ALL-Micro STANFORD_KB_XLING_1 ALL 0.1169 0.0448 0.0648 SF-ALL-Macro STANFORD_KB_XLING_1 0 0.0745 0.0768 0.0667 SF-ALL-Macro STANFORD_KB_XLING_1 1 0.0389 0.0403 0.0363 SF-ALL-Macro STANFORD_KB_XLING_1 ALL 0.0589 0.0608 0.0534 LDC-MAX-ALL-Micro STANFORD_KB_XLING_1 0 0.3607 0.1060 0.1639 LDC-MAX-ALL-Micro STANFORD_KB_XLING_1 1 0.1897 0.0550 0.0853 LDC-MAX-ALL-Micro STANFORD_KB_XLING_1 ALL 0.3112 0.0911 0.1410 LDC-MAX-ALL-Macro STANFORD_KB_XLING_1 0 0.1561 0.1478 0.1413 LDC-MAX-ALL-Macro STANFORD_KB_XLING_1 1 0.0665 0.0664 0.0610 LDC-MAX-ALL-Macro STANFORD_KB_XLING_1 ALL 0.1171 0.1124 0.1064 LDC-MEAN-ALL-Macro STANFORD_KB_XLING_1 0 0.0740 0.0761 0.0662 LDC-MEAN-ALL-Macro STANFORD_KB_XLING_1 1 0.0384 0.0398 0.0359 LDC-MEAN-ALL-Macro STANFORD_KB_XLING_1 ALL 0.0585 0.0603 0.0530 *ALL-Macro Prec, Recall and F1 refer to mean-precision, mean-recall and mean-F1. --------------------- Composite KB Evaluation (SF-SLOTS only): Scores based on confidence values and Average Precision (AP); k=3 justifications allowed: Metric RunID Hop AP SF-ALL-Macro STANFORD_KB_XLING_1 0 0.1136 SF-ALL-Macro STANFORD_KB_XLING_1 1 0.0196 SF-ALL-Macro STANFORD_KB_XLING_1 ALL 0.0727 LDC-MEAN-ALL-Macro STANFORD_KB_XLING_1 0 0.1136 LDC-MEAN-ALL-Macro STANFORD_KB_XLING_1 1 0.0196 LDC-MEAN-ALL-Macro STANFORD_KB_XLING_1 ALL 0.0728 (PRIMARY METRIC) *ALL-Macro AP refer to mean of corresponding AP values. Scores based on confidence values and Average Precision (AP); only k=1 justification allowed: Metric RunID Hop AP SF-ALL-Macro STANFORD_KB_XLING_1 0 0.1704 SF-ALL-Macro STANFORD_KB_XLING_1 1 0.0345 SF-ALL-Macro STANFORD_KB_XLING_1 ALL 0.1175 LDC-MEAN-ALL-Macro STANFORD_KB_XLING_1 0 0.1706 LDC-MEAN-ALL-Macro STANFORD_KB_XLING_1 1 0.0345 LDC-MEAN-ALL-Macro STANFORD_KB_XLING_1 ALL 0.1176 *ALL-Macro AP refer to mean of corresponding AP values. Scores based on P/R/F1; only k=1 justification allowed: Metric RunID Hop Prec Recall F1 SF-ALL-Micro STANFORD_KB_XLING_1 0 0.2161 0.1140 0.1493 SF-ALL-Micro STANFORD_KB_XLING_1 1 0.0410 0.0838 0.0550 SF-ALL-Micro STANFORD_KB_XLING_1 ALL 0.1169 0.1064 0.1114 SF-ALL-Macro STANFORD_KB_XLING_1 0 0.1778 0.1833 0.1592 SF-ALL-Macro STANFORD_KB_XLING_1 1 0.0761 0.0789 0.0711 SF-ALL-Macro STANFORD_KB_XLING_1 ALL 0.1283 0.1325 0.1163 LDC-MAX-ALL-Micro STANFORD_KB_XLING_1 0 0.3578 0.2388 0.2864 LDC-MAX-ALL-Micro STANFORD_KB_XLING_1 1 0.1564 0.1524 0.1543 LDC-MAX-ALL-Micro STANFORD_KB_XLING_1 ALL 0.2915 0.2171 0.2489 LDC-MAX-ALL-Macro STANFORD_KB_XLING_1 0 0.3759 0.3560 0.3404 LDC-MAX-ALL-Macro STANFORD_KB_XLING_1 1 0.1316 0.1314 0.1207 LDC-MAX-ALL-Macro STANFORD_KB_XLING_1 ALL 0.2577 0.2473 0.2341 LDC-MEAN-ALL-Macro STANFORD_KB_XLING_1 0 0.1782 0.1834 0.1595 LDC-MEAN-ALL-Macro STANFORD_KB_XLING_1 1 0.0761 0.0789 0.0711 LDC-MEAN-ALL-Macro STANFORD_KB_XLING_1 ALL 0.1288 0.1328 0.1167 *ALL-Macro Prec, Recall and F1 refer to mean-precision, mean-recall and mean-F1. --------------------- Composite KB Evaluation (EVENT-SLOTS only): Scores based on confidence values and Average Precision (AP); k=3 justifications allowed: SUMMARY: This section provides summary of AP scores Metric RunID Hop AP SF-ALL-Macro STANFORD_KB_XLING_1 0 0.0000 SF-ALL-Macro STANFORD_KB_XLING_1 ALL 0.0000 LDC-MEAN-ALL-Macro STANFORD_KB_XLING_1 0 0.0000 LDC-MEAN-ALL-Macro STANFORD_KB_XLING_1 ALL 0.0000 (PRIMARY METRIC) *ALL-Macro AP refer to mean of corresponding AP values. Scores based on confidence values and Average Precision (AP); only k=1 justification allowed: SUMMARY: This section provides summary of AP scores Metric RunID Hop AP SF-ALL-Macro STANFORD_KB_XLING_1 0 0.0000 SF-ALL-Macro STANFORD_KB_XLING_1 ALL 0.0000 LDC-MEAN-ALL-Macro STANFORD_KB_XLING_1 0 0.0000 LDC-MEAN-ALL-Macro STANFORD_KB_XLING_1 ALL 0.0000 *ALL-Macro AP refer to mean of corresponding AP values. Scores based on P/R/F1; only k=1 justification allowed: Metric RunID Hop Prec Recall F1 SF-ALL-Micro STANFORD_KB_XLING_1 0 0.0000 0.0000 0.0000 SF-ALL-Micro STANFORD_KB_XLING_1 ALL 0.0000 0.0000 0.0000 SF-ALL-Macro STANFORD_KB_XLING_1 0 0.0000 0.0000 0.0000 SF-ALL-Macro STANFORD_KB_XLING_1 ALL 0.0000 0.0000 0.0000 LDC-MAX-ALL-Micro STANFORD_KB_XLING_1 0 0.0000 0.0000 0.0000 LDC-MAX-ALL-Micro STANFORD_KB_XLING_1 ALL 0.0000 0.0000 0.0000 LDC-MAX-ALL-Macro STANFORD_KB_XLING_1 0 0.0000 0.0000 0.0000 LDC-MAX-ALL-Macro STANFORD_KB_XLING_1 ALL 0.0000 0.0000 0.0000 LDC-MEAN-ALL-Macro STANFORD_KB_XLING_1 0 0.0000 0.0000 0.0000 LDC-MEAN-ALL-Macro STANFORD_KB_XLING_1 ALL 0.0000 0.0000 0.0000 *ALL-Macro Prec, Recall and F1 refer to mean-precision, mean-recall and mean-F1. --------------------- Composite KB Evaluation (SENTIMENT-SLOTS only): Scores based on confidence values and Average Precision (AP); k=3 justifications allowed: Metric RunID Hop AP SF-ALL-Macro STANFORD_KB_XLING_1 0 0.0000 SF-ALL-Macro STANFORD_KB_XLING_1 1 0.0000 SF-ALL-Macro STANFORD_KB_XLING_1 ALL 0.0000 LDC-MEAN-ALL-Macro STANFORD_KB_XLING_1 0 0.0000 LDC-MEAN-ALL-Macro STANFORD_KB_XLING_1 1 0.0000 LDC-MEAN-ALL-Macro STANFORD_KB_XLING_1 ALL 0.0000 (PRIMARY METRIC) *ALL-Macro AP refer to mean of corresponding AP values. Scores based on confidence values and Average Precision (AP); only k=1 justification allowed: Metric RunID Hop AP SF-ALL-Macro STANFORD_KB_XLING_1 0 0.0000 SF-ALL-Macro STANFORD_KB_XLING_1 1 0.0000 SF-ALL-Macro STANFORD_KB_XLING_1 ALL 0.0000 LDC-MEAN-ALL-Macro STANFORD_KB_XLING_1 0 0.0000 LDC-MEAN-ALL-Macro STANFORD_KB_XLING_1 1 0.0000 LDC-MEAN-ALL-Macro STANFORD_KB_XLING_1 ALL 0.0000 *ALL-Macro AP refer to mean of corresponding AP values. Scores based on P/R/F1; only k=1 justification allowed: Metric RunID Hop Prec Recall F1 SF-ALL-Micro STANFORD_KB_XLING_1 0 0.0000 0.0000 0.0000 SF-ALL-Micro STANFORD_KB_XLING_1 1 0.0000 0.0000 0.0000 SF-ALL-Micro STANFORD_KB_XLING_1 ALL 0.0000 0.0000 0.0000 SF-ALL-Macro STANFORD_KB_XLING_1 0 0.0000 0.0000 0.0000 SF-ALL-Macro STANFORD_KB_XLING_1 1 0.0000 0.0000 0.0000 SF-ALL-Macro STANFORD_KB_XLING_1 ALL 0.0000 0.0000 0.0000 LDC-MAX-ALL-Micro STANFORD_KB_XLING_1 0 0.0000 0.0000 0.0000 LDC-MAX-ALL-Micro STANFORD_KB_XLING_1 1 0.0000 0.0000 0.0000 LDC-MAX-ALL-Micro STANFORD_KB_XLING_1 ALL 0.0000 0.0000 0.0000 LDC-MAX-ALL-Macro STANFORD_KB_XLING_1 0 0.0000 0.0000 0.0000 LDC-MAX-ALL-Macro STANFORD_KB_XLING_1 1 0.0000 0.0000 0.0000 LDC-MAX-ALL-Macro STANFORD_KB_XLING_1 ALL 0.0000 0.0000 0.0000 LDC-MEAN-ALL-Macro STANFORD_KB_XLING_1 0 0.0000 0.0000 0.0000 LDC-MEAN-ALL-Macro STANFORD_KB_XLING_1 1 0.0000 0.0000 0.0000 LDC-MEAN-ALL-Macro STANFORD_KB_XLING_1 ALL 0.0000 0.0000 0.0000 *ALL-Macro Prec, Recall and F1 refer to mean-precision, mean-recall and mean-F1.