===================================================================== TAC KBP 2017 ENGLISH KB CONSTRUCTION: COMPOSITE KB EVALUATION RESULTS ===================================================================== Team ID: LDC Organization: Linguistic Data Consortium ************************************************************* Run ID: LDC_KB_ENG_1 Did the run access the live Web during the evaluation window: No Did the run extract relations from the Cold Start source corpus: Yes Did the run generate meaningful confidence values: Yes Did the run attempt the full Entity Discovery and Linking task: Yes Did the run attempt the full Event Nugget Detection and Coreference task: Yes Did the run include SF relations: Yes Did the run attempt the Event Argument Extraction and Linking task: Yes Did the run include Sentiment relations: Yes --------------------- Composite KB Evaluation (All slot types): Scores based on confidence values and Average Precision (AP); k=3 justifications allowed: Metric RunID Hop AP SF-ALL-Macro LDC_KB_ENG_1 0 0.5495 SF-ALL-Macro LDC_KB_ENG_1 1 0.2811 SF-ALL-Macro LDC_KB_ENG_1 ALL 0.4892 LDC-MEAN-ALL-Macro LDC_KB_ENG_1 0 0.5007 LDC-MEAN-ALL-Macro LDC_KB_ENG_1 1 0.3071 LDC-MEAN-ALL-Macro LDC_KB_ENG_1 ALL 0.4504 (PRIMARY METRIC) *ALL-Macro AP refer to mean of corresponding AP values. Scores based on confidence values and Average Precision (AP); only k=1 justification allowed: Metric RunID Hop AP SF-ALL-Macro LDC_KB_ENG_1 0 0.6628 SF-ALL-Macro LDC_KB_ENG_1 1 0.2881 SF-ALL-Macro LDC_KB_ENG_1 ALL 0.5786 LDC-MEAN-ALL-Macro LDC_KB_ENG_1 0 0.6109 LDC-MEAN-ALL-Macro LDC_KB_ENG_1 1 0.3201 LDC-MEAN-ALL-Macro LDC_KB_ENG_1 ALL 0.5393 *ALL-Macro AP refer to mean of corresponding AP values. Scores based on P/R/F1; only k=1 justification allowed: Metric RunID Hop Prec Recall F1 SF-ALL-Micro LDC_KB_ENG_1 0 0.9445 0.4300 0.5910 SF-ALL-Micro LDC_KB_ENG_1 1 0.9138 0.2311 0.3689 SF-ALL-Micro LDC_KB_ENG_1 ALL 0.9402 0.3848 0.5461 SF-ALL-Macro LDC_KB_ENG_1 0 0.8034 0.6675 0.7021 SF-ALL-Macro LDC_KB_ENG_1 1 0.2652 0.2516 0.2553 SF-ALL-Macro LDC_KB_ENG_1 ALL 0.6183 0.5244 0.5484 LDC-MAX-ALL-Micro LDC_KB_ENG_1 0 0.9490 0.3687 0.5310 LDC-MAX-ALL-Micro LDC_KB_ENG_1 1 0.9408 0.5719 0.7114 LDC-MAX-ALL-Micro LDC_KB_ENG_1 ALL 0.9465 0.4126 0.5747 LDC-MAX-ALL-Macro LDC_KB_ENG_1 0 0.7490 0.6156 0.6486 LDC-MAX-ALL-Macro LDC_KB_ENG_1 1 0.6850 0.6498 0.6595 LDC-MAX-ALL-Macro LDC_KB_ENG_1 ALL 0.7271 0.6273 0.6523 LDC-MEAN-ALL-Macro LDC_KB_ENG_1 0 0.7490 0.6156 0.6486 LDC-MEAN-ALL-Macro LDC_KB_ENG_1 1 0.2826 0.2673 0.2715 LDC-MEAN-ALL-Macro LDC_KB_ENG_1 ALL 0.5895 0.4966 0.5197 *ALL-Macro Prec, Recall and F1 refer to mean-precision, mean-recall and mean-F1. --------------------- Composite KB Evaluation (SF-SLOTS only): Scores based on confidence values and Average Precision (AP); k=3 justifications allowed: Metric RunID Hop AP SF-ALL-Macro LDC_KB_ENG_1 0 0.4074 SF-ALL-Macro LDC_KB_ENG_1 1 0.1846 SF-ALL-Macro LDC_KB_ENG_1 ALL 0.3473 LDC-MEAN-ALL-Macro LDC_KB_ENG_1 0 0.3600 LDC-MEAN-ALL-Macro LDC_KB_ENG_1 1 0.1907 LDC-MEAN-ALL-Macro LDC_KB_ENG_1 ALL 0.3138 (PRIMARY METRIC) *ALL-Macro AP refer to mean of corresponding AP values. Scores based on confidence values and Average Precision (AP); only k=1 justification allowed: Metric RunID Hop AP SF-ALL-Macro LDC_KB_ENG_1 0 0.6086 SF-ALL-Macro LDC_KB_ENG_1 1 0.2044 SF-ALL-Macro LDC_KB_ENG_1 ALL 0.4933 LDC-MEAN-ALL-Macro LDC_KB_ENG_1 0 0.5507 LDC-MEAN-ALL-Macro LDC_KB_ENG_1 1 0.2249 LDC-MEAN-ALL-Macro LDC_KB_ENG_1 ALL 0.4584 *ALL-Macro AP refer to mean of corresponding AP values. Scores based on P/R/F1; only k=1 justification allowed: Metric RunID Hop Prec Recall F1 SF-ALL-Micro LDC_KB_ENG_1 0 0.9606 0.3368 0.4987 SF-ALL-Micro LDC_KB_ENG_1 1 0.9103 0.1940 0.3198 SF-ALL-Micro LDC_KB_ENG_1 ALL 0.9525 0.3025 0.4592 SF-ALL-Macro LDC_KB_ENG_1 0 0.7260 0.6087 0.6382 SF-ALL-Macro LDC_KB_ENG_1 1 0.2015 0.1879 0.1920 SF-ALL-Macro LDC_KB_ENG_1 ALL 0.5181 0.4419 0.4614 LDC-MAX-ALL-Micro LDC_KB_ENG_1 0 0.9669 0.2786 0.4326 LDC-MAX-ALL-Micro LDC_KB_ENG_1 1 0.9467 0.4494 0.6094 LDC-MAX-ALL-Micro LDC_KB_ENG_1 ALL 0.9602 0.3182 0.4780 LDC-MAX-ALL-Macro LDC_KB_ENG_1 0 0.6580 0.5507 0.5766 LDC-MAX-ALL-Macro LDC_KB_ENG_1 1 0.4981 0.4645 0.4746 LDC-MAX-ALL-Macro LDC_KB_ENG_1 ALL 0.5959 0.5172 0.5370 LDC-MEAN-ALL-Macro LDC_KB_ENG_1 0 0.6580 0.5507 0.5766 LDC-MEAN-ALL-Macro LDC_KB_ENG_1 1 0.2063 0.1926 0.1968 LDC-MEAN-ALL-Macro LDC_KB_ENG_1 ALL 0.4826 0.4117 0.4291 *ALL-Macro Prec, Recall and F1 refer to mean-precision, mean-recall and mean-F1. --------------------- Composite KB Evaluation (EVENT-SLOTS only): Scores based on confidence values and Average Precision (AP); k=3 justifications allowed: SUMMARY: This section provides summary of AP scores Metric RunID Hop AP SF-ALL-Macro LDC_KB_ENG_1 0 0.5874 SF-ALL-Macro LDC_KB_ENG_1 ALL 0.5874 LDC-MEAN-ALL-Macro LDC_KB_ENG_1 0 0.5424 LDC-MEAN-ALL-Macro LDC_KB_ENG_1 ALL 0.5424 (PRIMARY METRIC) *ALL-Macro AP refer to mean of corresponding AP values. Scores based on confidence values and Average Precision (AP); only k=1 justification allowed: SUMMARY: This section provides summary of AP scores Metric RunID Hop AP SF-ALL-Macro LDC_KB_ENG_1 0 0.6563 SF-ALL-Macro LDC_KB_ENG_1 ALL 0.6563 LDC-MEAN-ALL-Macro LDC_KB_ENG_1 0 0.6030 LDC-MEAN-ALL-Macro LDC_KB_ENG_1 ALL 0.6030 *ALL-Macro AP refer to mean of corresponding AP values. Scores based on P/R/F1; only k=1 justification allowed: Metric RunID Hop Prec Recall F1 SF-ALL-Micro LDC_KB_ENG_1 0 0.9214 0.4901 0.6399 SF-ALL-Micro LDC_KB_ENG_1 ALL 0.9214 0.4901 0.6399 SF-ALL-Macro LDC_KB_ENG_1 0 0.8307 0.6688 0.7108 SF-ALL-Macro LDC_KB_ENG_1 ALL 0.8307 0.6688 0.7108 LDC-MAX-ALL-Micro LDC_KB_ENG_1 0 0.9240 0.4236 0.5809 LDC-MAX-ALL-Micro LDC_KB_ENG_1 ALL 0.9240 0.4236 0.5809 LDC-MAX-ALL-Macro LDC_KB_ENG_1 0 0.7886 0.6161 0.6592 LDC-MAX-ALL-Macro LDC_KB_ENG_1 ALL 0.7886 0.6161 0.6592 LDC-MEAN-ALL-Macro LDC_KB_ENG_1 0 0.7886 0.6161 0.6592 LDC-MEAN-ALL-Macro LDC_KB_ENG_1 ALL 0.7886 0.6161 0.6592 *ALL-Macro Prec, Recall and F1 refer to mean-precision, mean-recall and mean-F1. --------------------- Composite KB Evaluation (SENTIMENT-SLOTS only): Scores based on confidence values and Average Precision (AP); k=3 justifications allowed: Metric RunID Hop AP SF-ALL-Macro LDC_KB_ENG_1 0 0.7987 SF-ALL-Macro LDC_KB_ENG_1 1 0.3448 SF-ALL-Macro LDC_KB_ENG_1 ALL 0.6182 LDC-MEAN-ALL-Macro LDC_KB_ENG_1 0 0.7835 LDC-MEAN-ALL-Macro LDC_KB_ENG_1 1 0.3651 LDC-MEAN-ALL-Macro LDC_KB_ENG_1 ALL 0.6213 (PRIMARY METRIC) *ALL-Macro AP refer to mean of corresponding AP values. Scores based on confidence values and Average Precision (AP); only k=1 justification allowed: Metric RunID Hop AP SF-ALL-Macro LDC_KB_ENG_1 0 0.7992 SF-ALL-Macro LDC_KB_ENG_1 1 0.3448 SF-ALL-Macro LDC_KB_ENG_1 ALL 0.6187 LDC-MEAN-ALL-Macro LDC_KB_ENG_1 0 0.7840 LDC-MEAN-ALL-Macro LDC_KB_ENG_1 1 0.3651 LDC-MEAN-ALL-Macro LDC_KB_ENG_1 ALL 0.6217 *ALL-Macro AP refer to mean of corresponding AP values. Scores based on P/R/F1; only k=1 justification allowed: Metric RunID Hop Prec Recall F1 SF-ALL-Micro LDC_KB_ENG_1 0 0.9650 0.6069 0.7452 SF-ALL-Micro LDC_KB_ENG_1 1 0.9600 0.3000 0.4571 SF-ALL-Micro LDC_KB_ENG_1 ALL 0.9644 0.5452 0.6966 SF-ALL-Macro LDC_KB_ENG_1 0 0.9257 0.7992 0.8303 SF-ALL-Macro LDC_KB_ENG_1 1 0.3151 0.3151 0.3151 SF-ALL-Macro LDC_KB_ENG_1 ALL 0.7460 0.6567 0.6787 LDC-MAX-ALL-Micro LDC_KB_ENG_1 0 0.9714 0.6071 0.7473 LDC-MAX-ALL-Micro LDC_KB_ENG_1 1 1.0000 0.8000 0.8889 LDC-MAX-ALL-Micro LDC_KB_ENG_1 ALL 0.9787 0.6479 0.7797 LDC-MAX-ALL-Macro LDC_KB_ENG_1 0 0.9062 0.7840 0.8149 LDC-MAX-ALL-Macro LDC_KB_ENG_1 1 0.8519 0.8519 0.8519 LDC-MAX-ALL-Macro LDC_KB_ENG_1 ALL 0.8901 0.8041 0.8259 LDC-MEAN-ALL-Macro LDC_KB_ENG_1 0 0.9062 0.7840 0.8149 LDC-MEAN-ALL-Macro LDC_KB_ENG_1 1 0.3210 0.3210 0.3210 LDC-MEAN-ALL-Macro LDC_KB_ENG_1 ALL 0.7326 0.6466 0.6684 *ALL-Macro Prec, Recall and F1 refer to mean-precision, mean-recall and mean-F1.