=========================================================================== TAC KBP 2017 CROSS-LINGUAL KB CONSTRUCTION: COMPOSITE KB EVALUATION RESULTS =========================================================================== Team ID: LDC Organization: Linguistic Data Consortium ************************************************************* Run ID: LDC_KB_XLING_1 Did the run access the live Web during the evaluation window: No Did the run extract relations from the Cold Start source corpus: Yes Did the run generate meaningful confidence values: Yes Run number of the English CSKB system that is most closely configured to the English component of this run: 1 Run number of the Spanish CSKB system that is most closely configured to the Spanish component of this run: 1 Run number of the Chinese CSKB system that is most closely configured to the Chinese component of this run: 1 Did the run attempt the full Entity Discovery and Linking task: Yes Did the run attempt the full Event Nugget Detection and Coreference task: Yes Did the run include SF relations: Yes Did the run attempt the Event Argument Extraction and Linking task: Yes Did the run include Sentiment relations: Yes --------------------- Composite KB Evaluation (All slot types): Scores based on confidence values and Average Precision (AP); k=3 justifications allowed: Metric RunID Hop AP SF-ALL-Macro LDC_KB_XLING_1 0 0.5976 SF-ALL-Macro LDC_KB_XLING_1 1 0.1576 SF-ALL-Macro LDC_KB_XLING_1 ALL 0.4944 LDC-MEAN-ALL-Macro LDC_KB_XLING_1 0 0.6008 LDC-MEAN-ALL-Macro LDC_KB_XLING_1 1 0.1664 LDC-MEAN-ALL-Macro LDC_KB_XLING_1 ALL 0.4986 (PRIMARY METRIC) *ALL-Macro AP refer to mean of corresponding AP values. Scores based on confidence values and Average Precision (AP); only k=1 justification allowed: Metric RunID Hop AP SF-ALL-Macro LDC_KB_XLING_1 0 0.6890 SF-ALL-Macro LDC_KB_XLING_1 1 0.1601 SF-ALL-Macro LDC_KB_XLING_1 ALL 0.5647 LDC-MEAN-ALL-Macro LDC_KB_XLING_1 0 0.6926 LDC-MEAN-ALL-Macro LDC_KB_XLING_1 1 0.1688 LDC-MEAN-ALL-Macro LDC_KB_XLING_1 ALL 0.5696 *ALL-Macro AP refer to mean of corresponding AP values. Scores based on P/R/F1; only k=1 justification allowed: Metric RunID Hop Prec Recall F1 SF-ALL-Micro LDC_KB_XLING_1 0 0.8961 0.3531 0.5066 SF-ALL-Micro LDC_KB_XLING_1 1 0.8493 0.0898 0.1625 SF-ALL-Micro LDC_KB_XLING_1 ALL 0.8915 0.2763 0.4218 SF-ALL-Macro LDC_KB_XLING_1 0 0.9356 0.7095 0.7547 SF-ALL-Macro LDC_KB_XLING_1 1 0.1509 0.1435 0.1446 SF-ALL-Macro LDC_KB_XLING_1 ALL 0.5923 0.4618 0.4878 LDC-MAX-ALL-Micro LDC_KB_XLING_1 0 0.8974 0.3559 0.5097 LDC-MAX-ALL-Micro LDC_KB_XLING_1 1 0.8963 0.4461 0.5957 LDC-MAX-ALL-Micro LDC_KB_XLING_1 ALL 0.8970 0.3822 0.5361 LDC-MAX-ALL-Macro LDC_KB_XLING_1 0 0.9353 0.7129 0.7576 LDC-MAX-ALL-Macro LDC_KB_XLING_1 1 0.7461 0.7092 0.7148 LDC-MAX-ALL-Macro LDC_KB_XLING_1 ALL 0.8530 0.7113 0.7389 LDC-MEAN-ALL-Macro LDC_KB_XLING_1 0 0.9353 0.7129 0.7576 LDC-MEAN-ALL-Macro LDC_KB_XLING_1 1 0.1560 0.1487 0.1498 LDC-MEAN-ALL-Macro LDC_KB_XLING_1 ALL 0.5962 0.4674 0.4931 *ALL-Macro Prec, Recall and F1 refer to mean-precision, mean-recall and mean-F1. --------------------- Composite KB Evaluation (SF-SLOTS only): Scores based on confidence values and Average Precision (AP); k=3 justifications allowed: Metric RunID Hop AP SF-ALL-Macro LDC_KB_XLING_1 0 0.5624 SF-ALL-Macro LDC_KB_XLING_1 1 0.1155 SF-ALL-Macro LDC_KB_XLING_1 ALL 0.4279 LDC-MEAN-ALL-Macro LDC_KB_XLING_1 0 0.5641 LDC-MEAN-ALL-Macro LDC_KB_XLING_1 1 0.1155 LDC-MEAN-ALL-Macro LDC_KB_XLING_1 ALL 0.4313 (PRIMARY METRIC) *ALL-Macro AP refer to mean of corresponding AP values. Scores based on confidence values and Average Precision (AP); only k=1 justification allowed: Metric RunID Hop AP SF-ALL-Macro LDC_KB_XLING_1 0 0.7339 SF-ALL-Macro LDC_KB_XLING_1 1 0.1218 SF-ALL-Macro LDC_KB_XLING_1 ALL 0.5523 LDC-MEAN-ALL-Macro LDC_KB_XLING_1 0 0.7350 LDC-MEAN-ALL-Macro LDC_KB_XLING_1 1 0.1218 LDC-MEAN-ALL-Macro LDC_KB_XLING_1 ALL 0.5556 *ALL-Macro AP refer to mean of corresponding AP values. Scores based on P/R/F1; only k=1 justification allowed: Metric RunID Hop Prec Recall F1 SF-ALL-Micro LDC_KB_XLING_1 0 0.9126 0.3353 0.4904 SF-ALL-Micro LDC_KB_XLING_1 1 0.8267 0.1181 0.2067 SF-ALL-Micro LDC_KB_XLING_1 ALL 0.9027 0.2805 0.4280 SF-ALL-Macro LDC_KB_XLING_1 0 0.9485 0.7501 0.7940 SF-ALL-Macro LDC_KB_XLING_1 1 0.1398 0.1306 0.1323 SF-ALL-Macro LDC_KB_XLING_1 ALL 0.5549 0.4485 0.4719 LDC-MAX-ALL-Micro LDC_KB_XLING_1 0 0.9133 0.3369 0.4922 LDC-MAX-ALL-Micro LDC_KB_XLING_1 1 0.8493 0.5905 0.6966 LDC-MAX-ALL-Micro LDC_KB_XLING_1 ALL 0.8885 0.4006 0.5523 LDC-MAX-ALL-Macro LDC_KB_XLING_1 0 0.9491 0.7515 0.7954 LDC-MAX-ALL-Macro LDC_KB_XLING_1 1 0.6988 0.6530 0.6614 LDC-MAX-ALL-Macro LDC_KB_XLING_1 ALL 0.8280 0.7038 0.7306 LDC-MEAN-ALL-Macro LDC_KB_XLING_1 0 0.9491 0.7515 0.7954 LDC-MEAN-ALL-Macro LDC_KB_XLING_1 1 0.1398 0.1306 0.1323 LDC-MEAN-ALL-Macro LDC_KB_XLING_1 ALL 0.5575 0.4511 0.4746 *ALL-Macro Prec, Recall and F1 refer to mean-precision, mean-recall and mean-F1. --------------------- Composite KB Evaluation (EVENT-SLOTS only): Scores based on confidence values and Average Precision (AP); k=3 justifications allowed: SUMMARY: This section provides summary of AP scores Metric RunID Hop AP SF-ALL-Macro LDC_KB_XLING_1 0 0.5827 SF-ALL-Macro LDC_KB_XLING_1 ALL 0.5827 LDC-MEAN-ALL-Macro LDC_KB_XLING_1 0 0.5825 LDC-MEAN-ALL-Macro LDC_KB_XLING_1 ALL 0.5825 (PRIMARY METRIC) *ALL-Macro AP refer to mean of corresponding AP values. Scores based on confidence values and Average Precision (AP); only k=1 justification allowed: SUMMARY: This section provides summary of AP scores Metric RunID Hop AP SF-ALL-Macro LDC_KB_XLING_1 0 0.6327 SF-ALL-Macro LDC_KB_XLING_1 ALL 0.6327 LDC-MEAN-ALL-Macro LDC_KB_XLING_1 0 0.6363 LDC-MEAN-ALL-Macro LDC_KB_XLING_1 ALL 0.6363 *ALL-Macro AP refer to mean of corresponding AP values. Scores based on P/R/F1; only k=1 justification allowed: Metric RunID Hop Prec Recall F1 SF-ALL-Micro LDC_KB_XLING_1 0 0.8606 0.3925 0.5391 SF-ALL-Micro LDC_KB_XLING_1 ALL 0.8606 0.3925 0.5391 SF-ALL-Macro LDC_KB_XLING_1 0 0.9128 0.6667 0.7126 SF-ALL-Macro LDC_KB_XLING_1 ALL 0.9128 0.6667 0.7126 LDC-MAX-ALL-Micro LDC_KB_XLING_1 0 0.8621 0.3947 0.5415 LDC-MAX-ALL-Micro LDC_KB_XLING_1 ALL 0.8621 0.3947 0.5415 LDC-MAX-ALL-Macro LDC_KB_XLING_1 0 0.9104 0.6694 0.7145 LDC-MAX-ALL-Macro LDC_KB_XLING_1 ALL 0.9104 0.6694 0.7145 LDC-MEAN-ALL-Macro LDC_KB_XLING_1 0 0.9104 0.6694 0.7145 LDC-MEAN-ALL-Macro LDC_KB_XLING_1 ALL 0.9104 0.6694 0.7145 *ALL-Macro Prec, Recall and F1 refer to mean-precision, mean-recall and mean-F1. --------------------- Composite KB Evaluation (SENTIMENT-SLOTS only): Scores based on confidence values and Average Precision (AP); k=3 justifications allowed: Metric RunID Hop AP SF-ALL-Macro LDC_KB_XLING_1 0 0.6813 SF-ALL-Macro LDC_KB_XLING_1 1 0.1806 SF-ALL-Macro LDC_KB_XLING_1 ALL 0.4842 LDC-MEAN-ALL-Macro LDC_KB_XLING_1 0 0.6909 LDC-MEAN-ALL-Macro LDC_KB_XLING_1 1 0.1964 LDC-MEAN-ALL-Macro LDC_KB_XLING_1 ALL 0.4942 (PRIMARY METRIC) *ALL-Macro AP refer to mean of corresponding AP values. Scores based on confidence values and Average Precision (AP); only k=1 justification allowed: Metric RunID Hop AP SF-ALL-Macro LDC_KB_XLING_1 0 0.6911 SF-ALL-Macro LDC_KB_XLING_1 1 0.1807 SF-ALL-Macro LDC_KB_XLING_1 ALL 0.4879 LDC-MEAN-ALL-Macro LDC_KB_XLING_1 0 0.7004 LDC-MEAN-ALL-Macro LDC_KB_XLING_1 1 0.1965 LDC-MEAN-ALL-Macro LDC_KB_XLING_1 ALL 0.4978 *ALL-Macro AP refer to mean of corresponding AP values. Scores based on P/R/F1; only k=1 justification allowed: Metric RunID Hop Prec Recall F1 SF-ALL-Micro LDC_KB_XLING_1 0 0.9429 0.3215 0.4795 SF-ALL-Micro LDC_KB_XLING_1 1 0.8983 0.0719 0.1332 SF-ALL-Micro LDC_KB_XLING_1 ALL 0.9395 0.2556 0.4019 SF-ALL-Macro LDC_KB_XLING_1 0 0.9456 0.6997 0.7464 SF-ALL-Macro LDC_KB_XLING_1 1 0.1250 0.1214 0.1220 SF-ALL-Macro LDC_KB_XLING_1 ALL 0.6113 0.4641 0.4921 LDC-MAX-ALL-Micro LDC_KB_XLING_1 0 0.9444 0.3277 0.4866 LDC-MAX-ALL-Micro LDC_KB_XLING_1 1 0.9815 0.3510 0.5171 LDC-MAX-ALL-Micro LDC_KB_XLING_1 ALL 0.9545 0.3339 0.4948 LDC-MAX-ALL-Macro LDC_KB_XLING_1 0 0.9472 0.7087 0.7541 LDC-MAX-ALL-Macro LDC_KB_XLING_1 1 0.6098 0.5923 0.5950 LDC-MAX-ALL-Macro LDC_KB_XLING_1 ALL 0.8102 0.6614 0.6895 LDC-MEAN-ALL-Macro LDC_KB_XLING_1 0 0.9472 0.7087 0.7541 LDC-MEAN-ALL-Macro LDC_KB_XLING_1 1 0.1358 0.1323 0.1328 LDC-MEAN-ALL-Macro LDC_KB_XLING_1 ALL 0.6178 0.4747 0.5019 *ALL-Macro Prec, Recall and F1 refer to mean-precision, mean-recall and mean-F1.