===================================================================== TAC KBP 2017 SPANISH KB CONSTRUCTION: COMPOSITE KB EVALUATION RESULTS ===================================================================== Team ID: LDC Organization: Linguistic Data Consortium ************************************************************* Run ID: LDC_KB_SPA_1 Did the run access the live Web during the evaluation window: No Did the run extract relations from the Cold Start source corpus: Yes Did the run generate meaningful confidence values: Yes Did the run attempt the full Entity Discovery and Linking task: Yes Did the run attempt the full Event Nugget Detection and Coreference task: Yes Did the run include SF relations: Yes Did the run attempt the Event Argument Extraction and Linking task: Yes Did the run include Sentiment relations: Yes --------------------- Composite KB Evaluation (All slot types): Scores based on confidence values and Average Precision (AP); k=3 justifications allowed: Metric RunID Hop AP SF-ALL-Macro LDC_KB_SPA_1 0 0.7852 SF-ALL-Macro LDC_KB_SPA_1 1 0.5153 SF-ALL-Macro LDC_KB_SPA_1 ALL 0.7096 LDC-MEAN-ALL-Macro LDC_KB_SPA_1 0 0.7990 LDC-MEAN-ALL-Macro LDC_KB_SPA_1 1 0.6096 LDC-MEAN-ALL-Macro LDC_KB_SPA_1 ALL 0.7395 (PRIMARY METRIC) *ALL-Macro AP refer to mean of corresponding AP values. Scores based on confidence values and Average Precision (AP); only k=1 justification allowed: Metric RunID Hop AP SF-ALL-Macro LDC_KB_SPA_1 0 0.8310 SF-ALL-Macro LDC_KB_SPA_1 1 0.5190 SF-ALL-Macro LDC_KB_SPA_1 ALL 0.7415 LDC-MEAN-ALL-Macro LDC_KB_SPA_1 0 0.8462 LDC-MEAN-ALL-Macro LDC_KB_SPA_1 1 0.6128 LDC-MEAN-ALL-Macro LDC_KB_SPA_1 ALL 0.7727 *ALL-Macro AP refer to mean of corresponding AP values. Scores based on P/R/F1; only k=1 justification allowed: Metric RunID Hop Prec Recall F1 SF-ALL-Micro LDC_KB_SPA_1 0 0.8976 0.6765 0.7715 SF-ALL-Micro LDC_KB_SPA_1 1 0.8400 0.5176 0.6405 SF-ALL-Micro LDC_KB_SPA_1 ALL 0.8803 0.6220 0.7289 SF-ALL-Macro LDC_KB_SPA_1 0 0.8962 0.8440 0.8528 SF-ALL-Macro LDC_KB_SPA_1 1 0.4899 0.4862 0.4867 SF-ALL-Macro LDC_KB_SPA_1 ALL 0.7445 0.7104 0.7161 LDC-MAX-ALL-Micro LDC_KB_SPA_1 0 0.8866 0.7226 0.7962 LDC-MAX-ALL-Micro LDC_KB_SPA_1 1 0.8698 0.8909 0.8802 LDC-MAX-ALL-Micro LDC_KB_SPA_1 ALL 0.8796 0.7834 0.8287 LDC-MAX-ALL-Macro LDC_KB_SPA_1 0 0.8972 0.8587 0.8661 LDC-MAX-ALL-Macro LDC_KB_SPA_1 1 0.8784 0.8716 0.8726 LDC-MAX-ALL-Macro LDC_KB_SPA_1 ALL 0.8904 0.8634 0.8685 LDC-MEAN-ALL-Macro LDC_KB_SPA_1 0 0.8972 0.8587 0.8661 LDC-MEAN-ALL-Macro LDC_KB_SPA_1 1 0.5761 0.5735 0.5737 LDC-MEAN-ALL-Macro LDC_KB_SPA_1 ALL 0.7800 0.7546 0.7593 *ALL-Macro Prec, Recall and F1 refer to mean-precision, mean-recall and mean-F1. --------------------- Composite KB Evaluation (SF-SLOTS only): Scores based on confidence values and Average Precision (AP); k=3 justifications allowed: Metric RunID Hop AP SF-ALL-Macro LDC_KB_SPA_1 0 0.7705 SF-ALL-Macro LDC_KB_SPA_1 1 0.4855 SF-ALL-Macro LDC_KB_SPA_1 ALL 0.6700 LDC-MEAN-ALL-Macro LDC_KB_SPA_1 0 0.7819 LDC-MEAN-ALL-Macro LDC_KB_SPA_1 1 0.5801 LDC-MEAN-ALL-Macro LDC_KB_SPA_1 ALL 0.7067 (PRIMARY METRIC) *ALL-Macro AP refer to mean of corresponding AP values. Scores based on confidence values and Average Precision (AP); only k=1 justification allowed: Metric RunID Hop AP SF-ALL-Macro LDC_KB_SPA_1 0 0.8514 SF-ALL-Macro LDC_KB_SPA_1 1 0.4933 SF-ALL-Macro LDC_KB_SPA_1 ALL 0.7258 LDC-MEAN-ALL-Macro LDC_KB_SPA_1 0 0.8629 LDC-MEAN-ALL-Macro LDC_KB_SPA_1 1 0.5866 LDC-MEAN-ALL-Macro LDC_KB_SPA_1 ALL 0.7633 *ALL-Macro AP refer to mean of corresponding AP values. Scores based on P/R/F1; only k=1 justification allowed: Metric RunID Hop Prec Recall F1 SF-ALL-Micro LDC_KB_SPA_1 0 0.9263 0.7701 0.8410 SF-ALL-Micro LDC_KB_SPA_1 1 0.8556 0.4968 0.6286 SF-ALL-Micro LDC_KB_SPA_1 ALL 0.9055 0.6683 0.7690 SF-ALL-Macro LDC_KB_SPA_1 0 0.8986 0.8560 0.8696 SF-ALL-Macro LDC_KB_SPA_1 1 0.4486 0.4393 0.4424 SF-ALL-Macro LDC_KB_SPA_1 ALL 0.7331 0.7027 0.7125 LDC-MAX-ALL-Micro LDC_KB_SPA_1 0 0.9219 0.7867 0.8489 LDC-MAX-ALL-Micro LDC_KB_SPA_1 1 0.8851 0.8280 0.8556 LDC-MAX-ALL-Micro LDC_KB_SPA_1 ALL 0.9070 0.8025 0.8515 LDC-MAX-ALL-Macro LDC_KB_SPA_1 0 0.9061 0.8667 0.8794 LDC-MAX-ALL-Macro LDC_KB_SPA_1 1 0.8000 0.7833 0.7889 LDC-MAX-ALL-Macro LDC_KB_SPA_1 ALL 0.8686 0.8373 0.8475 LDC-MEAN-ALL-Macro LDC_KB_SPA_1 0 0.9061 0.8667 0.8794 LDC-MEAN-ALL-Macro LDC_KB_SPA_1 1 0.5333 0.5264 0.5287 LDC-MEAN-ALL-Macro LDC_KB_SPA_1 ALL 0.7745 0.7466 0.7556 *ALL-Macro Prec, Recall and F1 refer to mean-precision, mean-recall and mean-F1. --------------------- Composite KB Evaluation (EVENT-SLOTS only): Scores based on confidence values and Average Precision (AP); k=3 justifications allowed: SUMMARY: This section provides summary of AP scores Metric RunID Hop AP SF-ALL-Macro LDC_KB_SPA_1 0 0.8800 SF-ALL-Macro LDC_KB_SPA_1 ALL 0.8800 LDC-MEAN-ALL-Macro LDC_KB_SPA_1 0 0.8871 LDC-MEAN-ALL-Macro LDC_KB_SPA_1 ALL 0.8871 (PRIMARY METRIC) *ALL-Macro AP refer to mean of corresponding AP values. Scores based on confidence values and Average Precision (AP); only k=1 justification allowed: SUMMARY: This section provides summary of AP scores Metric RunID Hop AP SF-ALL-Macro LDC_KB_SPA_1 0 0.8807 SF-ALL-Macro LDC_KB_SPA_1 ALL 0.8807 LDC-MEAN-ALL-Macro LDC_KB_SPA_1 0 0.8878 LDC-MEAN-ALL-Macro LDC_KB_SPA_1 ALL 0.8878 *ALL-Macro AP refer to mean of corresponding AP values. Scores based on P/R/F1; only k=1 justification allowed: Metric RunID Hop Prec Recall F1 SF-ALL-Micro LDC_KB_SPA_1 0 0.8548 0.9138 0.8833 SF-ALL-Micro LDC_KB_SPA_1 ALL 0.8548 0.9138 0.8833 SF-ALL-Macro LDC_KB_SPA_1 0 0.9205 0.9205 0.9091 SF-ALL-Macro LDC_KB_SPA_1 ALL 0.9205 0.9205 0.9091 LDC-MAX-ALL-Micro LDC_KB_SPA_1 0 0.8356 0.9242 0.8777 LDC-MAX-ALL-Micro LDC_KB_SPA_1 ALL 0.8356 0.9242 0.8777 LDC-MAX-ALL-Macro LDC_KB_SPA_1 0 0.9286 0.9286 0.9184 LDC-MAX-ALL-Macro LDC_KB_SPA_1 ALL 0.9286 0.9286 0.9184 LDC-MEAN-ALL-Macro LDC_KB_SPA_1 0 0.9286 0.9286 0.9184 LDC-MEAN-ALL-Macro LDC_KB_SPA_1 ALL 0.9286 0.9286 0.9184 *ALL-Macro Prec, Recall and F1 refer to mean-precision, mean-recall and mean-F1. --------------------- Composite KB Evaluation (SENTIMENT-SLOTS only): Scores based on confidence values and Average Precision (AP); k=3 justifications allowed: Metric RunID Hop AP SF-ALL-Macro LDC_KB_SPA_1 0 0.6943 SF-ALL-Macro LDC_KB_SPA_1 1 0.5200 SF-ALL-Macro LDC_KB_SPA_1 ALL 0.5856 LDC-MEAN-ALL-Macro LDC_KB_SPA_1 0 0.7276 LDC-MEAN-ALL-Macro LDC_KB_SPA_1 1 0.6821 LDC-MEAN-ALL-Macro LDC_KB_SPA_1 ALL 0.6329 (PRIMARY METRIC) *ALL-Macro AP refer to mean of corresponding AP values. Scores based on confidence values and Average Precision (AP); only k=1 justification allowed: Metric RunID Hop AP SF-ALL-Macro LDC_KB_SPA_1 0 0.6997 SF-ALL-Macro LDC_KB_SPA_1 1 0.5200 SF-ALL-Macro LDC_KB_SPA_1 ALL 0.5907 LDC-MEAN-ALL-Macro LDC_KB_SPA_1 0 0.7325 LDC-MEAN-ALL-Macro LDC_KB_SPA_1 1 0.6821 LDC-MEAN-ALL-Macro LDC_KB_SPA_1 ALL 0.6375 *ALL-Macro AP refer to mean of corresponding AP values. Scores based on P/R/F1; only k=1 justification allowed: Metric RunID Hop Prec Recall F1 SF-ALL-Micro LDC_KB_SPA_1 0 0.8841 0.3653 0.5169 SF-ALL-Micro LDC_KB_SPA_1 1 0.7500 0.5556 0.6383 SF-ALL-Micro LDC_KB_SPA_1 ALL 0.8539 0.3918 0.5371 SF-ALL-Macro LDC_KB_SPA_1 0 0.8548 0.6997 0.7230 SF-ALL-Macro LDC_KB_SPA_1 1 0.5200 0.5200 0.5200 SF-ALL-Macro LDC_KB_SPA_1 ALL 0.7586 0.6481 0.6647 LDC-MAX-ALL-Micro LDC_KB_SPA_1 0 0.8649 0.4211 0.5664 LDC-MAX-ALL-Micro LDC_KB_SPA_1 1 0.7500 1.0000 0.8571 LDC-MAX-ALL-Micro LDC_KB_SPA_1 ALL 0.8246 0.5165 0.6351 LDC-MAX-ALL-Macro LDC_KB_SPA_1 0 0.8235 0.7325 0.7476 LDC-MAX-ALL-Macro LDC_KB_SPA_1 1 1.0000 1.0000 1.0000 LDC-MAX-ALL-Macro LDC_KB_SPA_1 ALL 0.8723 0.8065 0.8174 LDC-MEAN-ALL-Macro LDC_KB_SPA_1 0 0.8235 0.7325 0.7476 LDC-MEAN-ALL-Macro LDC_KB_SPA_1 1 0.6821 0.6821 0.6821 LDC-MEAN-ALL-Macro LDC_KB_SPA_1 ALL 0.7844 0.7185 0.7295 *ALL-Macro Prec, Recall and F1 refer to mean-precision, mean-recall and mean-F1.