===================================================================== TAC KBP 2017 CHINESE KB CONSTRUCTION: COMPOSITE KB EVALUATION RESULTS ===================================================================== Team ID: LDC Organization: Linguistic Data Consortium ************************************************************* Run ID: LDC_KB_CMN_1 Did the run access the live Web during the evaluation window: No Did the run extract relations from the Cold Start source corpus: Yes Did the run generate meaningful confidence values: Yes Did the run attempt the full Entity Discovery and Linking task: Yes Did the run attempt the full Event Nugget Detection and Coreference task: Yes Did the run include SF relations: Yes Did the run attempt the Event Argument Extraction and Linking task: Yes Did the run include Sentiment relations: Yes --------------------- Composite KB Evaluation (All slot types): Scores based on confidence values and Average Precision (AP); k=3 justifications allowed: Metric RunID Hop AP SF-ALL-Macro LDC_KB_CMN_1 0 0.5542 SF-ALL-Macro LDC_KB_CMN_1 1 0.4292 SF-ALL-Macro LDC_KB_CMN_1 ALL 0.5157 LDC-MEAN-ALL-Macro LDC_KB_CMN_1 0 0.5183 LDC-MEAN-ALL-Macro LDC_KB_CMN_1 1 0.5473 LDC-MEAN-ALL-Macro LDC_KB_CMN_1 ALL 0.4980 (PRIMARY METRIC) *ALL-Macro AP refer to mean of corresponding AP values. Scores based on confidence values and Average Precision (AP); only k=1 justification allowed: Metric RunID Hop AP SF-ALL-Macro LDC_KB_CMN_1 0 0.6377 SF-ALL-Macro LDC_KB_CMN_1 1 0.4391 SF-ALL-Macro LDC_KB_CMN_1 ALL 0.5807 LDC-MEAN-ALL-Macro LDC_KB_CMN_1 0 0.5988 LDC-MEAN-ALL-Macro LDC_KB_CMN_1 1 0.5580 LDC-MEAN-ALL-Macro LDC_KB_CMN_1 ALL 0.5624 *ALL-Macro AP refer to mean of corresponding AP values. Scores based on P/R/F1; only k=1 justification allowed: Metric RunID Hop Prec Recall F1 SF-ALL-Micro LDC_KB_CMN_1 0 0.8841 0.2766 0.4213 SF-ALL-Micro LDC_KB_CMN_1 1 0.8120 0.1469 0.2488 SF-ALL-Micro LDC_KB_CMN_1 ALL 0.8692 0.2362 0.3715 SF-ALL-Macro LDC_KB_CMN_1 0 0.8262 0.6470 0.6800 SF-ALL-Macro LDC_KB_CMN_1 1 0.3962 0.3593 0.3649 SF-ALL-Macro LDC_KB_CMN_1 ALL 0.6662 0.5399 0.5627 LDC-MAX-ALL-Micro LDC_KB_CMN_1 0 0.8968 0.2694 0.4143 LDC-MAX-ALL-Micro LDC_KB_CMN_1 1 0.8438 0.2250 0.3553 LDC-MAX-ALL-Micro LDC_KB_CMN_1 ALL 0.8813 0.2553 0.3959 LDC-MAX-ALL-Macro LDC_KB_CMN_1 0 0.7765 0.6035 0.6376 LDC-MAX-ALL-Macro LDC_KB_CMN_1 1 0.6954 0.6306 0.6404 LDC-MAX-ALL-Macro LDC_KB_CMN_1 ALL 0.7482 0.6129 0.6386 LDC-MEAN-ALL-Macro LDC_KB_CMN_1 0 0.7765 0.6035 0.6376 LDC-MEAN-ALL-Macro LDC_KB_CMN_1 1 0.5472 0.5019 0.5079 LDC-MEAN-ALL-Macro LDC_KB_CMN_1 ALL 0.6964 0.5680 0.5923 *ALL-Macro Prec, Recall and F1 refer to mean-precision, mean-recall and mean-F1. --------------------- Composite KB Evaluation (SF-SLOTS only): Scores based on confidence values and Average Precision (AP); k=3 justifications allowed: Metric RunID Hop AP SF-ALL-Macro LDC_KB_CMN_1 0 0.6254 SF-ALL-Macro LDC_KB_CMN_1 1 0.4066 SF-ALL-Macro LDC_KB_CMN_1 ALL 0.5729 LDC-MEAN-ALL-Macro LDC_KB_CMN_1 0 0.5783 LDC-MEAN-ALL-Macro LDC_KB_CMN_1 1 0.5370 LDC-MEAN-ALL-Macro LDC_KB_CMN_1 ALL 0.5503 (PRIMARY METRIC) *ALL-Macro AP refer to mean of corresponding AP values. Scores based on confidence values and Average Precision (AP); only k=1 justification allowed: Metric RunID Hop AP SF-ALL-Macro LDC_KB_CMN_1 0 0.7544 SF-ALL-Macro LDC_KB_CMN_1 1 0.4237 SF-ALL-Macro LDC_KB_CMN_1 ALL 0.6692 LDC-MEAN-ALL-Macro LDC_KB_CMN_1 0 0.7072 LDC-MEAN-ALL-Macro LDC_KB_CMN_1 1 0.5542 LDC-MEAN-ALL-Macro LDC_KB_CMN_1 ALL 0.6509 *ALL-Macro AP refer to mean of corresponding AP values. Scores based on P/R/F1; only k=1 justification allowed: Metric RunID Hop Prec Recall F1 SF-ALL-Micro LDC_KB_CMN_1 0 0.8943 0.3543 0.5075 SF-ALL-Micro LDC_KB_CMN_1 1 0.7738 0.3591 0.4906 SF-ALL-Micro LDC_KB_CMN_1 ALL 0.8636 0.3554 0.5035 SF-ALL-Macro LDC_KB_CMN_1 0 0.8794 0.7544 0.7821 SF-ALL-Macro LDC_KB_CMN_1 1 0.5676 0.5248 0.5338 SF-ALL-Macro LDC_KB_CMN_1 ALL 0.7732 0.6762 0.6976 LDC-MAX-ALL-Micro LDC_KB_CMN_1 0 0.9062 0.3208 0.4739 LDC-MAX-ALL-Micro LDC_KB_CMN_1 1 0.8025 0.5702 0.6667 LDC-MAX-ALL-Micro LDC_KB_CMN_1 ALL 0.8714 0.3710 0.5204 LDC-MAX-ALL-Macro LDC_KB_CMN_1 0 0.8390 0.7072 0.7355 LDC-MAX-ALL-Macro LDC_KB_CMN_1 1 0.8182 0.7565 0.7695 LDC-MAX-ALL-Macro LDC_KB_CMN_1 ALL 0.8316 0.7248 0.7476 LDC-MEAN-ALL-Macro LDC_KB_CMN_1 0 0.8390 0.7072 0.7355 LDC-MEAN-ALL-Macro LDC_KB_CMN_1 1 0.6719 0.6379 0.6452 LDC-MEAN-ALL-Macro LDC_KB_CMN_1 ALL 0.7792 0.6824 0.7032 *ALL-Macro Prec, Recall and F1 refer to mean-precision, mean-recall and mean-F1. --------------------- Composite KB Evaluation (EVENT-SLOTS only): Scores based on confidence values and Average Precision (AP); k=3 justifications allowed: SUMMARY: This section provides summary of AP scores Metric RunID Hop AP SF-ALL-Macro LDC_KB_CMN_1 0 0.4808 SF-ALL-Macro LDC_KB_CMN_1 ALL 0.4808 LDC-MEAN-ALL-Macro LDC_KB_CMN_1 0 0.4461 LDC-MEAN-ALL-Macro LDC_KB_CMN_1 ALL 0.4461 (PRIMARY METRIC) *ALL-Macro AP refer to mean of corresponding AP values. Scores based on confidence values and Average Precision (AP); only k=1 justification allowed: SUMMARY: This section provides summary of AP scores Metric RunID Hop AP SF-ALL-Macro LDC_KB_CMN_1 0 0.5358 SF-ALL-Macro LDC_KB_CMN_1 ALL 0.5358 LDC-MEAN-ALL-Macro LDC_KB_CMN_1 0 0.4909 LDC-MEAN-ALL-Macro LDC_KB_CMN_1 ALL 0.4909 *ALL-Macro AP refer to mean of corresponding AP values. Scores based on P/R/F1; only k=1 justification allowed: Metric RunID Hop Prec Recall F1 SF-ALL-Micro LDC_KB_CMN_1 0 0.8351 0.2898 0.4303 SF-ALL-Micro LDC_KB_CMN_1 ALL 0.8351 0.2898 0.4303 SF-ALL-Macro LDC_KB_CMN_1 0 0.7683 0.5634 0.5971 SF-ALL-Macro LDC_KB_CMN_1 ALL 0.7683 0.5634 0.5971 LDC-MAX-ALL-Micro LDC_KB_CMN_1 0 0.8559 0.2730 0.4139 LDC-MAX-ALL-Micro LDC_KB_CMN_1 ALL 0.8559 0.2730 0.4139 LDC-MAX-ALL-Macro LDC_KB_CMN_1 0 0.7142 0.5049 0.5450 LDC-MAX-ALL-Macro LDC_KB_CMN_1 ALL 0.7142 0.5049 0.5450 LDC-MEAN-ALL-Macro LDC_KB_CMN_1 0 0.7142 0.5049 0.5450 LDC-MEAN-ALL-Macro LDC_KB_CMN_1 ALL 0.7142 0.5049 0.5450 *ALL-Macro Prec, Recall and F1 refer to mean-precision, mean-recall and mean-F1. --------------------- Composite KB Evaluation (SENTIMENT-SLOTS only): Scores based on confidence values and Average Precision (AP); k=3 justifications allowed: Metric RunID Hop AP SF-ALL-Macro LDC_KB_CMN_1 0 0.5018 SF-ALL-Macro LDC_KB_CMN_1 1 0.4277 SF-ALL-Macro LDC_KB_CMN_1 ALL 0.4295 LDC-MEAN-ALL-Macro LDC_KB_CMN_1 0 0.4872 LDC-MEAN-ALL-Macro LDC_KB_CMN_1 1 0.5062 LDC-MEAN-ALL-Macro LDC_KB_CMN_1 ALL 0.4501 (PRIMARY METRIC) *ALL-Macro AP refer to mean of corresponding AP values. Scores based on confidence values and Average Precision (AP); only k=1 justification allowed: Metric RunID Hop AP SF-ALL-Macro LDC_KB_CMN_1 0 0.5181 SF-ALL-Macro LDC_KB_CMN_1 1 0.4289 SF-ALL-Macro LDC_KB_CMN_1 ALL 0.4310 LDC-MEAN-ALL-Macro LDC_KB_CMN_1 0 0.5006 LDC-MEAN-ALL-Macro LDC_KB_CMN_1 1 0.5071 LDC-MEAN-ALL-Macro LDC_KB_CMN_1 ALL 0.4513 *ALL-Macro AP refer to mean of corresponding AP values. Scores based on P/R/F1; only k=1 justification allowed: Metric RunID Hop Prec Recall F1 SF-ALL-Micro LDC_KB_CMN_1 0 0.9855 0.1521 0.2636 SF-ALL-Micro LDC_KB_CMN_1 1 1.0000 0.0520 0.0989 SF-ALL-Micro LDC_KB_CMN_1 ALL 0.9880 0.1145 0.2053 SF-ALL-Macro LDC_KB_CMN_1 0 0.7927 0.5181 0.5640 SF-ALL-Macro LDC_KB_CMN_1 1 0.1414 0.1269 0.1292 SF-ALL-Macro LDC_KB_CMN_1 ALL 0.4365 0.3041 0.3262 LDC-MAX-ALL-Micro LDC_KB_CMN_1 0 0.9744 0.1638 0.2804 LDC-MAX-ALL-Micro LDC_KB_CMN_1 1 1.0000 0.1321 0.2333 LDC-MAX-ALL-Micro LDC_KB_CMN_1 ALL 0.9811 0.1538 0.2660 LDC-MAX-ALL-Macro LDC_KB_CMN_1 0 0.7200 0.5006 0.5399 LDC-MAX-ALL-Macro LDC_KB_CMN_1 1 0.3333 0.2992 0.3046 LDC-MAX-ALL-Macro LDC_KB_CMN_1 ALL 0.5435 0.4086 0.4325 LDC-MEAN-ALL-Macro LDC_KB_CMN_1 0 0.7200 0.5006 0.5399 LDC-MEAN-ALL-Macro LDC_KB_CMN_1 1 0.2302 0.2151 0.2171 LDC-MEAN-ALL-Macro LDC_KB_CMN_1 ALL 0.4964 0.3702 0.3925 *ALL-Macro Prec, Recall and F1 refer to mean-precision, mean-recall and mean-F1.