===================================================================== TAC KBP 2017 CHINESE KB CONSTRUCTION: COMPOSITE KB EVALUATION RESULTS ===================================================================== Team ID: hltcoe Organization: Human Language Technology Center of Excellence ************************************************************* Run ID: hltcoe_KB_CMN_1 Did the run access the live Web during the evaluation window: No Did the run extract relations from the Cold Start source corpus: Yes Did the run generate meaningful confidence values: Yes Did the run attempt the full Entity Discovery and Linking task: Yes Did the run attempt the full Event Nugget Detection and Coreference task: No Did the run include SF relations: Yes Did the run attempt the Event Argument Extraction and Linking task: No Did the run include Sentiment relations: No --------------------- Composite KB Evaluation (All slot types): Scores based on confidence values and Average Precision (AP); k=3 justifications allowed: Metric RunID Hop AP SF-ALL-Macro hltcoe_KB_CMN_1 0 0.0430 SF-ALL-Macro hltcoe_KB_CMN_1 1 0.0029 SF-ALL-Macro hltcoe_KB_CMN_1 ALL 0.0327 LDC-MEAN-ALL-Macro hltcoe_KB_CMN_1 0 0.0581 LDC-MEAN-ALL-Macro hltcoe_KB_CMN_1 1 0.0031 LDC-MEAN-ALL-Macro hltcoe_KB_CMN_1 ALL 0.0448 (PRIMARY METRIC) *ALL-Macro AP refer to mean of corresponding AP values. Scores based on confidence values and Average Precision (AP); only k=1 justification allowed: Metric RunID Hop AP SF-ALL-Macro hltcoe_KB_CMN_1 0 0.0467 SF-ALL-Macro hltcoe_KB_CMN_1 1 0.0071 SF-ALL-Macro hltcoe_KB_CMN_1 ALL 0.0363 LDC-MEAN-ALL-Macro hltcoe_KB_CMN_1 0 0.0611 LDC-MEAN-ALL-Macro hltcoe_KB_CMN_1 1 0.0064 LDC-MEAN-ALL-Macro hltcoe_KB_CMN_1 ALL 0.0477 *ALL-Macro AP refer to mean of corresponding AP values. Scores based on P/R/F1; only k=1 justification allowed: Metric RunID Hop Prec Recall F1 SF-ALL-Micro hltcoe_KB_CMN_1 0 0.4700 0.0627 0.1106 SF-ALL-Micro hltcoe_KB_CMN_1 1 0.0714 0.0027 0.0052 SF-ALL-Micro hltcoe_KB_CMN_1 ALL 0.4245 0.0440 0.0798 SF-ALL-Macro hltcoe_KB_CMN_1 0 0.0543 0.0501 0.0494 SF-ALL-Macro hltcoe_KB_CMN_1 1 0.0057 0.0075 0.0063 SF-ALL-Macro hltcoe_KB_CMN_1 ALL 0.0362 0.0343 0.0334 LDC-MAX-ALL-Micro hltcoe_KB_CMN_1 0 0.4545 0.0775 0.1325 LDC-MAX-ALL-Micro hltcoe_KB_CMN_1 1 0.0714 0.0042 0.0079 LDC-MAX-ALL-Micro hltcoe_KB_CMN_1 ALL 0.4020 0.0542 0.0956 LDC-MAX-ALL-Macro hltcoe_KB_CMN_1 0 0.0785 0.0760 0.0736 LDC-MAX-ALL-Macro hltcoe_KB_CMN_1 1 0.0099 0.0132 0.0110 LDC-MAX-ALL-Macro hltcoe_KB_CMN_1 ALL 0.0546 0.0540 0.0517 LDC-MEAN-ALL-Macro hltcoe_KB_CMN_1 0 0.0673 0.0663 0.0635 LDC-MEAN-ALL-Macro hltcoe_KB_CMN_1 1 0.0066 0.0099 0.0077 LDC-MEAN-ALL-Macro hltcoe_KB_CMN_1 ALL 0.0461 0.0466 0.0440 *ALL-Macro Prec, Recall and F1 refer to mean-precision, mean-recall and mean-F1. --------------------- Composite KB Evaluation (SF-SLOTS only): Scores based on confidence values and Average Precision (AP); k=3 justifications allowed: Metric RunID Hop AP SF-ALL-Macro hltcoe_KB_CMN_1 0 0.0893 SF-ALL-Macro hltcoe_KB_CMN_1 1 0.0058 SF-ALL-Macro hltcoe_KB_CMN_1 ALL 0.0679 LDC-MEAN-ALL-Macro hltcoe_KB_CMN_1 0 0.1183 LDC-MEAN-ALL-Macro hltcoe_KB_CMN_1 1 0.0060 LDC-MEAN-ALL-Macro hltcoe_KB_CMN_1 ALL 0.0913 (PRIMARY METRIC) *ALL-Macro AP refer to mean of corresponding AP values. Scores based on confidence values and Average Precision (AP); only k=1 justification allowed: Metric RunID Hop AP SF-ALL-Macro hltcoe_KB_CMN_1 0 0.0970 SF-ALL-Macro hltcoe_KB_CMN_1 1 0.0141 SF-ALL-Macro hltcoe_KB_CMN_1 ALL 0.0755 LDC-MEAN-ALL-Macro hltcoe_KB_CMN_1 0 0.1244 LDC-MEAN-ALL-Macro hltcoe_KB_CMN_1 1 0.0127 LDC-MEAN-ALL-Macro hltcoe_KB_CMN_1 ALL 0.0972 *ALL-Macro AP refer to mean of corresponding AP values. Scores based on P/R/F1; only k=1 justification allowed: Metric RunID Hop Prec Recall F1 SF-ALL-Micro hltcoe_KB_CMN_1 0 0.4700 0.1643 0.2434 SF-ALL-Micro hltcoe_KB_CMN_1 1 0.0714 0.0110 0.0191 SF-ALL-Micro hltcoe_KB_CMN_1 ALL 0.4245 0.1297 0.1987 SF-ALL-Macro hltcoe_KB_CMN_1 0 0.1129 0.1042 0.1028 SF-ALL-Macro hltcoe_KB_CMN_1 1 0.0135 0.0180 0.0150 SF-ALL-Macro hltcoe_KB_CMN_1 ALL 0.0791 0.0748 0.0729 LDC-MAX-ALL-Micro hltcoe_KB_CMN_1 0 0.4545 0.1770 0.2548 LDC-MAX-ALL-Micro hltcoe_KB_CMN_1 1 0.0714 0.0175 0.0282 LDC-MAX-ALL-Micro hltcoe_KB_CMN_1 ALL 0.4020 0.1449 0.2130 LDC-MAX-ALL-Macro hltcoe_KB_CMN_1 0 0.1599 0.1547 0.1498 LDC-MAX-ALL-Macro hltcoe_KB_CMN_1 1 0.0195 0.0260 0.0216 LDC-MAX-ALL-Macro hltcoe_KB_CMN_1 ALL 0.1096 0.1086 0.1039 LDC-MEAN-ALL-Macro hltcoe_KB_CMN_1 0 0.1370 0.1349 0.1292 LDC-MEAN-ALL-Macro hltcoe_KB_CMN_1 1 0.0130 0.0195 0.0152 LDC-MEAN-ALL-Macro hltcoe_KB_CMN_1 ALL 0.0926 0.0936 0.0884 *ALL-Macro Prec, Recall and F1 refer to mean-precision, mean-recall and mean-F1. --------------------- Composite KB Evaluation (EVENT-SLOTS only): Scores based on confidence values and Average Precision (AP); k=3 justifications allowed: SUMMARY: This section provides summary of AP scores Metric RunID Hop AP SF-ALL-Macro hltcoe_KB_CMN_1 0 0.0000 SF-ALL-Macro hltcoe_KB_CMN_1 ALL 0.0000 LDC-MEAN-ALL-Macro hltcoe_KB_CMN_1 0 0.0000 LDC-MEAN-ALL-Macro hltcoe_KB_CMN_1 ALL 0.0000 (PRIMARY METRIC) *ALL-Macro AP refer to mean of corresponding AP values. Scores based on confidence values and Average Precision (AP); only k=1 justification allowed: SUMMARY: This section provides summary of AP scores Metric RunID Hop AP SF-ALL-Macro hltcoe_KB_CMN_1 0 0.0000 SF-ALL-Macro hltcoe_KB_CMN_1 ALL 0.0000 LDC-MEAN-ALL-Macro hltcoe_KB_CMN_1 0 0.0000 LDC-MEAN-ALL-Macro hltcoe_KB_CMN_1 ALL 0.0000 *ALL-Macro AP refer to mean of corresponding AP values. Scores based on P/R/F1; only k=1 justification allowed: Metric RunID Hop Prec Recall F1 SF-ALL-Micro hltcoe_KB_CMN_1 0 0.0000 0.0000 0.0000 SF-ALL-Micro hltcoe_KB_CMN_1 ALL 0.0000 0.0000 0.0000 SF-ALL-Macro hltcoe_KB_CMN_1 0 0.0000 0.0000 0.0000 SF-ALL-Macro hltcoe_KB_CMN_1 ALL 0.0000 0.0000 0.0000 LDC-MAX-ALL-Micro hltcoe_KB_CMN_1 0 0.0000 0.0000 0.0000 LDC-MAX-ALL-Micro hltcoe_KB_CMN_1 ALL 0.0000 0.0000 0.0000 LDC-MAX-ALL-Macro hltcoe_KB_CMN_1 0 0.0000 0.0000 0.0000 LDC-MAX-ALL-Macro hltcoe_KB_CMN_1 ALL 0.0000 0.0000 0.0000 LDC-MEAN-ALL-Macro hltcoe_KB_CMN_1 0 0.0000 0.0000 0.0000 LDC-MEAN-ALL-Macro hltcoe_KB_CMN_1 ALL 0.0000 0.0000 0.0000 *ALL-Macro Prec, Recall and F1 refer to mean-precision, mean-recall and mean-F1. --------------------- Composite KB Evaluation (SENTIMENT-SLOTS only): Scores based on confidence values and Average Precision (AP); k=3 justifications allowed: Metric RunID Hop AP SF-ALL-Macro hltcoe_KB_CMN_1 0 0.0000 SF-ALL-Macro hltcoe_KB_CMN_1 1 0.0000 SF-ALL-Macro hltcoe_KB_CMN_1 ALL 0.0000 LDC-MEAN-ALL-Macro hltcoe_KB_CMN_1 0 0.0000 LDC-MEAN-ALL-Macro hltcoe_KB_CMN_1 1 0.0000 LDC-MEAN-ALL-Macro hltcoe_KB_CMN_1 ALL 0.0000 (PRIMARY METRIC) *ALL-Macro AP refer to mean of corresponding AP values. Scores based on confidence values and Average Precision (AP); only k=1 justification allowed: Metric RunID Hop AP SF-ALL-Macro hltcoe_KB_CMN_1 0 0.0000 SF-ALL-Macro hltcoe_KB_CMN_1 1 0.0000 SF-ALL-Macro hltcoe_KB_CMN_1 ALL 0.0000 LDC-MEAN-ALL-Macro hltcoe_KB_CMN_1 0 0.0000 LDC-MEAN-ALL-Macro hltcoe_KB_CMN_1 1 0.0000 LDC-MEAN-ALL-Macro hltcoe_KB_CMN_1 ALL 0.0000 *ALL-Macro AP refer to mean of corresponding AP values. Scores based on P/R/F1; only k=1 justification allowed: Metric RunID Hop Prec Recall F1 SF-ALL-Micro hltcoe_KB_CMN_1 0 0.0000 0.0000 0.0000 SF-ALL-Micro hltcoe_KB_CMN_1 1 0.0000 0.0000 0.0000 SF-ALL-Micro hltcoe_KB_CMN_1 ALL 0.0000 0.0000 0.0000 SF-ALL-Macro hltcoe_KB_CMN_1 0 0.0000 0.0000 0.0000 SF-ALL-Macro hltcoe_KB_CMN_1 1 0.0000 0.0000 0.0000 SF-ALL-Macro hltcoe_KB_CMN_1 ALL 0.0000 0.0000 0.0000 LDC-MAX-ALL-Micro hltcoe_KB_CMN_1 0 0.0000 0.0000 0.0000 LDC-MAX-ALL-Micro hltcoe_KB_CMN_1 1 0.0000 0.0000 0.0000 LDC-MAX-ALL-Micro hltcoe_KB_CMN_1 ALL 0.0000 0.0000 0.0000 LDC-MAX-ALL-Macro hltcoe_KB_CMN_1 0 0.0000 0.0000 0.0000 LDC-MAX-ALL-Macro hltcoe_KB_CMN_1 1 0.0000 0.0000 0.0000 LDC-MAX-ALL-Macro hltcoe_KB_CMN_1 ALL 0.0000 0.0000 0.0000 LDC-MEAN-ALL-Macro hltcoe_KB_CMN_1 0 0.0000 0.0000 0.0000 LDC-MEAN-ALL-Macro hltcoe_KB_CMN_1 1 0.0000 0.0000 0.0000 LDC-MEAN-ALL-Macro hltcoe_KB_CMN_1 ALL 0.0000 0.0000 0.0000 *ALL-Macro Prec, Recall and F1 refer to mean-precision, mean-recall and mean-F1. ************************************************************* Run ID: hltcoe_KB_CMN_2 Did the run access the live Web during the evaluation window: No Did the run extract relations from the Cold Start source corpus: Yes Did the run generate meaningful confidence values: Yes Did the run attempt the full Entity Discovery and Linking task: Yes Did the run attempt the full Event Nugget Detection and Coreference task: No Did the run include SF relations: No Did the run attempt the Event Argument Extraction and Linking task: No Did the run include Sentiment relations: No --------------------- Composite KB Evaluation (All slot types): Scores based on confidence values and Average Precision (AP); k=3 justifications allowed: Metric RunID Hop AP SF-ALL-Macro hltcoe_KB_CMN_2 0 0.0430 SF-ALL-Macro hltcoe_KB_CMN_2 1 0.0029 SF-ALL-Macro hltcoe_KB_CMN_2 ALL 0.0327 LDC-MEAN-ALL-Macro hltcoe_KB_CMN_2 0 0.0581 LDC-MEAN-ALL-Macro hltcoe_KB_CMN_2 1 0.0031 LDC-MEAN-ALL-Macro hltcoe_KB_CMN_2 ALL 0.0449 (PRIMARY METRIC) *ALL-Macro AP refer to mean of corresponding AP values. Scores based on confidence values and Average Precision (AP); only k=1 justification allowed: Metric RunID Hop AP SF-ALL-Macro hltcoe_KB_CMN_2 0 0.0467 SF-ALL-Macro hltcoe_KB_CMN_2 1 0.0071 SF-ALL-Macro hltcoe_KB_CMN_2 ALL 0.0364 LDC-MEAN-ALL-Macro hltcoe_KB_CMN_2 0 0.0611 LDC-MEAN-ALL-Macro hltcoe_KB_CMN_2 1 0.0064 LDC-MEAN-ALL-Macro hltcoe_KB_CMN_2 ALL 0.0478 *ALL-Macro AP refer to mean of corresponding AP values. Scores based on P/R/F1; only k=1 justification allowed: Metric RunID Hop Prec Recall F1 SF-ALL-Micro hltcoe_KB_CMN_2 0 0.4658 0.0627 0.1105 SF-ALL-Micro hltcoe_KB_CMN_2 1 0.0714 0.0027 0.0052 SF-ALL-Micro hltcoe_KB_CMN_2 ALL 0.4211 0.0440 0.0797 SF-ALL-Macro hltcoe_KB_CMN_2 0 0.0543 0.0501 0.0494 SF-ALL-Macro hltcoe_KB_CMN_2 1 0.0057 0.0075 0.0063 SF-ALL-Macro hltcoe_KB_CMN_2 ALL 0.0362 0.0343 0.0334 LDC-MAX-ALL-Micro hltcoe_KB_CMN_2 0 0.4520 0.0775 0.1323 LDC-MAX-ALL-Micro hltcoe_KB_CMN_2 1 0.0714 0.0042 0.0079 LDC-MAX-ALL-Micro hltcoe_KB_CMN_2 ALL 0.4000 0.0542 0.0955 LDC-MAX-ALL-Macro hltcoe_KB_CMN_2 0 0.0785 0.0760 0.0736 LDC-MAX-ALL-Macro hltcoe_KB_CMN_2 1 0.0099 0.0132 0.0110 LDC-MAX-ALL-Macro hltcoe_KB_CMN_2 ALL 0.0546 0.0540 0.0517 LDC-MEAN-ALL-Macro hltcoe_KB_CMN_2 0 0.0673 0.0663 0.0635 LDC-MEAN-ALL-Macro hltcoe_KB_CMN_2 1 0.0066 0.0099 0.0077 LDC-MEAN-ALL-Macro hltcoe_KB_CMN_2 ALL 0.0461 0.0466 0.0440 *ALL-Macro Prec, Recall and F1 refer to mean-precision, mean-recall and mean-F1. --------------------- Composite KB Evaluation (SF-SLOTS only): Scores based on confidence values and Average Precision (AP); k=3 justifications allowed: Metric RunID Hop AP SF-ALL-Macro hltcoe_KB_CMN_2 0 0.0894 SF-ALL-Macro hltcoe_KB_CMN_2 1 0.0058 SF-ALL-Macro hltcoe_KB_CMN_2 ALL 0.0680 LDC-MEAN-ALL-Macro hltcoe_KB_CMN_2 0 0.1184 LDC-MEAN-ALL-Macro hltcoe_KB_CMN_2 1 0.0060 LDC-MEAN-ALL-Macro hltcoe_KB_CMN_2 ALL 0.0914 (PRIMARY METRIC) *ALL-Macro AP refer to mean of corresponding AP values. Scores based on confidence values and Average Precision (AP); only k=1 justification allowed: Metric RunID Hop AP SF-ALL-Macro hltcoe_KB_CMN_2 0 0.0971 SF-ALL-Macro hltcoe_KB_CMN_2 1 0.0141 SF-ALL-Macro hltcoe_KB_CMN_2 ALL 0.0756 LDC-MEAN-ALL-Macro hltcoe_KB_CMN_2 0 0.1245 LDC-MEAN-ALL-Macro hltcoe_KB_CMN_2 1 0.0127 LDC-MEAN-ALL-Macro hltcoe_KB_CMN_2 ALL 0.0973 *ALL-Macro AP refer to mean of corresponding AP values. Scores based on P/R/F1; only k=1 justification allowed: Metric RunID Hop Prec Recall F1 SF-ALL-Micro hltcoe_KB_CMN_2 0 0.4658 0.1643 0.2429 SF-ALL-Micro hltcoe_KB_CMN_2 1 0.0714 0.0110 0.0191 SF-ALL-Micro hltcoe_KB_CMN_2 ALL 0.4211 0.1297 0.1983 SF-ALL-Macro hltcoe_KB_CMN_2 0 0.1129 0.1042 0.1028 SF-ALL-Macro hltcoe_KB_CMN_2 1 0.0135 0.0180 0.0150 SF-ALL-Macro hltcoe_KB_CMN_2 ALL 0.0791 0.0748 0.0729 LDC-MAX-ALL-Micro hltcoe_KB_CMN_2 0 0.4444 0.1770 0.2532 LDC-MAX-ALL-Micro hltcoe_KB_CMN_2 1 0.0714 0.0175 0.0282 LDC-MAX-ALL-Micro hltcoe_KB_CMN_2 ALL 0.3942 0.1449 0.2119 LDC-MAX-ALL-Macro hltcoe_KB_CMN_2 0 0.1599 0.1547 0.1498 LDC-MAX-ALL-Macro hltcoe_KB_CMN_2 1 0.0195 0.0260 0.0216 LDC-MAX-ALL-Macro hltcoe_KB_CMN_2 ALL 0.1096 0.1086 0.1039 LDC-MEAN-ALL-Macro hltcoe_KB_CMN_2 0 0.1370 0.1349 0.1292 LDC-MEAN-ALL-Macro hltcoe_KB_CMN_2 1 0.0130 0.0195 0.0152 LDC-MEAN-ALL-Macro hltcoe_KB_CMN_2 ALL 0.0926 0.0936 0.0884 *ALL-Macro Prec, Recall and F1 refer to mean-precision, mean-recall and mean-F1. --------------------- Composite KB Evaluation (EVENT-SLOTS only): Scores based on confidence values and Average Precision (AP); k=3 justifications allowed: SUMMARY: This section provides summary of AP scores Metric RunID Hop AP SF-ALL-Macro hltcoe_KB_CMN_2 0 0.0000 SF-ALL-Macro hltcoe_KB_CMN_2 ALL 0.0000 LDC-MEAN-ALL-Macro hltcoe_KB_CMN_2 0 0.0000 LDC-MEAN-ALL-Macro hltcoe_KB_CMN_2 ALL 0.0000 (PRIMARY METRIC) *ALL-Macro AP refer to mean of corresponding AP values. Scores based on confidence values and Average Precision (AP); only k=1 justification allowed: SUMMARY: This section provides summary of AP scores Metric RunID Hop AP SF-ALL-Macro hltcoe_KB_CMN_2 0 0.0000 SF-ALL-Macro hltcoe_KB_CMN_2 ALL 0.0000 LDC-MEAN-ALL-Macro hltcoe_KB_CMN_2 0 0.0000 LDC-MEAN-ALL-Macro hltcoe_KB_CMN_2 ALL 0.0000 *ALL-Macro AP refer to mean of corresponding AP values. Scores based on P/R/F1; only k=1 justification allowed: Metric RunID Hop Prec Recall F1 SF-ALL-Micro hltcoe_KB_CMN_2 0 0.0000 0.0000 0.0000 SF-ALL-Micro hltcoe_KB_CMN_2 ALL 0.0000 0.0000 0.0000 SF-ALL-Macro hltcoe_KB_CMN_2 0 0.0000 0.0000 0.0000 SF-ALL-Macro hltcoe_KB_CMN_2 ALL 0.0000 0.0000 0.0000 LDC-MAX-ALL-Micro hltcoe_KB_CMN_2 0 0.0000 0.0000 0.0000 LDC-MAX-ALL-Micro hltcoe_KB_CMN_2 ALL 0.0000 0.0000 0.0000 LDC-MAX-ALL-Macro hltcoe_KB_CMN_2 0 0.0000 0.0000 0.0000 LDC-MAX-ALL-Macro hltcoe_KB_CMN_2 ALL 0.0000 0.0000 0.0000 LDC-MEAN-ALL-Macro hltcoe_KB_CMN_2 0 0.0000 0.0000 0.0000 LDC-MEAN-ALL-Macro hltcoe_KB_CMN_2 ALL 0.0000 0.0000 0.0000 *ALL-Macro Prec, Recall and F1 refer to mean-precision, mean-recall and mean-F1. --------------------- Composite KB Evaluation (SENTIMENT-SLOTS only): Scores based on confidence values and Average Precision (AP); k=3 justifications allowed: Metric RunID Hop AP SF-ALL-Macro hltcoe_KB_CMN_2 0 0.0000 SF-ALL-Macro hltcoe_KB_CMN_2 1 0.0000 SF-ALL-Macro hltcoe_KB_CMN_2 ALL 0.0000 LDC-MEAN-ALL-Macro hltcoe_KB_CMN_2 0 0.0000 LDC-MEAN-ALL-Macro hltcoe_KB_CMN_2 1 0.0000 LDC-MEAN-ALL-Macro hltcoe_KB_CMN_2 ALL 0.0000 (PRIMARY METRIC) *ALL-Macro AP refer to mean of corresponding AP values. Scores based on confidence values and Average Precision (AP); only k=1 justification allowed: Metric RunID Hop AP SF-ALL-Macro hltcoe_KB_CMN_2 0 0.0000 SF-ALL-Macro hltcoe_KB_CMN_2 1 0.0000 SF-ALL-Macro hltcoe_KB_CMN_2 ALL 0.0000 LDC-MEAN-ALL-Macro hltcoe_KB_CMN_2 0 0.0000 LDC-MEAN-ALL-Macro hltcoe_KB_CMN_2 1 0.0000 LDC-MEAN-ALL-Macro hltcoe_KB_CMN_2 ALL 0.0000 *ALL-Macro AP refer to mean of corresponding AP values. Scores based on P/R/F1; only k=1 justification allowed: Metric RunID Hop Prec Recall F1 SF-ALL-Micro hltcoe_KB_CMN_2 0 0.0000 0.0000 0.0000 SF-ALL-Micro hltcoe_KB_CMN_2 1 0.0000 0.0000 0.0000 SF-ALL-Micro hltcoe_KB_CMN_2 ALL 0.0000 0.0000 0.0000 SF-ALL-Macro hltcoe_KB_CMN_2 0 0.0000 0.0000 0.0000 SF-ALL-Macro hltcoe_KB_CMN_2 1 0.0000 0.0000 0.0000 SF-ALL-Macro hltcoe_KB_CMN_2 ALL 0.0000 0.0000 0.0000 LDC-MAX-ALL-Micro hltcoe_KB_CMN_2 0 0.0000 0.0000 0.0000 LDC-MAX-ALL-Micro hltcoe_KB_CMN_2 1 0.0000 0.0000 0.0000 LDC-MAX-ALL-Micro hltcoe_KB_CMN_2 ALL 0.0000 0.0000 0.0000 LDC-MAX-ALL-Macro hltcoe_KB_CMN_2 0 0.0000 0.0000 0.0000 LDC-MAX-ALL-Macro hltcoe_KB_CMN_2 1 0.0000 0.0000 0.0000 LDC-MAX-ALL-Macro hltcoe_KB_CMN_2 ALL 0.0000 0.0000 0.0000 LDC-MEAN-ALL-Macro hltcoe_KB_CMN_2 0 0.0000 0.0000 0.0000 LDC-MEAN-ALL-Macro hltcoe_KB_CMN_2 1 0.0000 0.0000 0.0000 LDC-MEAN-ALL-Macro hltcoe_KB_CMN_2 ALL 0.0000 0.0000 0.0000 *ALL-Macro Prec, Recall and F1 refer to mean-precision, mean-recall and mean-F1. ************************************************************* Run ID: hltcoe_KB_CMN_3 Did the run access the live Web during the evaluation window: No Did the run extract relations from the Cold Start source corpus: Yes Did the run generate meaningful confidence values: Yes Did the run attempt the full Entity Discovery and Linking task: Yes Did the run attempt the full Event Nugget Detection and Coreference task: No Did the run include SF relations: No Did the run attempt the Event Argument Extraction and Linking task: No Did the run include Sentiment relations: No --------------------- Composite KB Evaluation (All slot types): Scores based on confidence values and Average Precision (AP); k=3 justifications allowed: Metric RunID Hop AP SF-ALL-Macro hltcoe_KB_CMN_3 0 0.0430 SF-ALL-Macro hltcoe_KB_CMN_3 1 0.0029 SF-ALL-Macro hltcoe_KB_CMN_3 ALL 0.0327 LDC-MEAN-ALL-Macro hltcoe_KB_CMN_3 0 0.0581 LDC-MEAN-ALL-Macro hltcoe_KB_CMN_3 1 0.0031 LDC-MEAN-ALL-Macro hltcoe_KB_CMN_3 ALL 0.0449 (PRIMARY METRIC) *ALL-Macro AP refer to mean of corresponding AP values. Scores based on confidence values and Average Precision (AP); only k=1 justification allowed: Metric RunID Hop AP SF-ALL-Macro hltcoe_KB_CMN_3 0 0.0467 SF-ALL-Macro hltcoe_KB_CMN_3 1 0.0071 SF-ALL-Macro hltcoe_KB_CMN_3 ALL 0.0364 LDC-MEAN-ALL-Macro hltcoe_KB_CMN_3 0 0.0611 LDC-MEAN-ALL-Macro hltcoe_KB_CMN_3 1 0.0064 LDC-MEAN-ALL-Macro hltcoe_KB_CMN_3 ALL 0.0478 *ALL-Macro AP refer to mean of corresponding AP values. Scores based on P/R/F1; only k=1 justification allowed: Metric RunID Hop Prec Recall F1 SF-ALL-Micro hltcoe_KB_CMN_3 0 0.4658 0.0627 0.1105 SF-ALL-Micro hltcoe_KB_CMN_3 1 0.0714 0.0027 0.0052 SF-ALL-Micro hltcoe_KB_CMN_3 ALL 0.4211 0.0440 0.0797 SF-ALL-Macro hltcoe_KB_CMN_3 0 0.0543 0.0501 0.0494 SF-ALL-Macro hltcoe_KB_CMN_3 1 0.0057 0.0075 0.0063 SF-ALL-Macro hltcoe_KB_CMN_3 ALL 0.0362 0.0343 0.0334 LDC-MAX-ALL-Micro hltcoe_KB_CMN_3 0 0.4469 0.0775 0.1321 LDC-MAX-ALL-Micro hltcoe_KB_CMN_3 1 0.0714 0.0042 0.0079 LDC-MAX-ALL-Micro hltcoe_KB_CMN_3 ALL 0.3961 0.0542 0.0954 LDC-MAX-ALL-Macro hltcoe_KB_CMN_3 0 0.0785 0.0760 0.0736 LDC-MAX-ALL-Macro hltcoe_KB_CMN_3 1 0.0099 0.0132 0.0110 LDC-MAX-ALL-Macro hltcoe_KB_CMN_3 ALL 0.0546 0.0540 0.0517 LDC-MEAN-ALL-Macro hltcoe_KB_CMN_3 0 0.0673 0.0663 0.0635 LDC-MEAN-ALL-Macro hltcoe_KB_CMN_3 1 0.0066 0.0099 0.0077 LDC-MEAN-ALL-Macro hltcoe_KB_CMN_3 ALL 0.0461 0.0466 0.0440 *ALL-Macro Prec, Recall and F1 refer to mean-precision, mean-recall and mean-F1. --------------------- Composite KB Evaluation (SF-SLOTS only): Scores based on confidence values and Average Precision (AP); k=3 justifications allowed: Metric RunID Hop AP SF-ALL-Macro hltcoe_KB_CMN_3 0 0.0893 SF-ALL-Macro hltcoe_KB_CMN_3 1 0.0058 SF-ALL-Macro hltcoe_KB_CMN_3 ALL 0.0680 LDC-MEAN-ALL-Macro hltcoe_KB_CMN_3 0 0.1183 LDC-MEAN-ALL-Macro hltcoe_KB_CMN_3 1 0.0060 LDC-MEAN-ALL-Macro hltcoe_KB_CMN_3 ALL 0.0914 (PRIMARY METRIC) *ALL-Macro AP refer to mean of corresponding AP values. Scores based on confidence values and Average Precision (AP); only k=1 justification allowed: Metric RunID Hop AP SF-ALL-Macro hltcoe_KB_CMN_3 0 0.0971 SF-ALL-Macro hltcoe_KB_CMN_3 1 0.0141 SF-ALL-Macro hltcoe_KB_CMN_3 ALL 0.0756 LDC-MEAN-ALL-Macro hltcoe_KB_CMN_3 0 0.1244 LDC-MEAN-ALL-Macro hltcoe_KB_CMN_3 1 0.0127 LDC-MEAN-ALL-Macro hltcoe_KB_CMN_3 ALL 0.0973 *ALL-Macro AP refer to mean of corresponding AP values. Scores based on P/R/F1; only k=1 justification allowed: Metric RunID Hop Prec Recall F1 SF-ALL-Micro hltcoe_KB_CMN_3 0 0.4658 0.1643 0.2429 SF-ALL-Micro hltcoe_KB_CMN_3 1 0.0714 0.0110 0.0191 SF-ALL-Micro hltcoe_KB_CMN_3 ALL 0.4211 0.1297 0.1983 SF-ALL-Macro hltcoe_KB_CMN_3 0 0.1129 0.1042 0.1028 SF-ALL-Macro hltcoe_KB_CMN_3 1 0.0135 0.0180 0.0150 SF-ALL-Macro hltcoe_KB_CMN_3 ALL 0.0791 0.0748 0.0729 LDC-MAX-ALL-Micro hltcoe_KB_CMN_3 0 0.4469 0.1770 0.2536 LDC-MAX-ALL-Micro hltcoe_KB_CMN_3 1 0.0714 0.0175 0.0282 LDC-MAX-ALL-Micro hltcoe_KB_CMN_3 ALL 0.3961 0.1449 0.2122 LDC-MAX-ALL-Macro hltcoe_KB_CMN_3 0 0.1599 0.1547 0.1498 LDC-MAX-ALL-Macro hltcoe_KB_CMN_3 1 0.0195 0.0260 0.0216 LDC-MAX-ALL-Macro hltcoe_KB_CMN_3 ALL 0.1096 0.1086 0.1039 LDC-MEAN-ALL-Macro hltcoe_KB_CMN_3 0 0.1370 0.1349 0.1292 LDC-MEAN-ALL-Macro hltcoe_KB_CMN_3 1 0.0130 0.0195 0.0152 LDC-MEAN-ALL-Macro hltcoe_KB_CMN_3 ALL 0.0926 0.0936 0.0884 *ALL-Macro Prec, Recall and F1 refer to mean-precision, mean-recall and mean-F1. --------------------- Composite KB Evaluation (EVENT-SLOTS only): Scores based on confidence values and Average Precision (AP); k=3 justifications allowed: SUMMARY: This section provides summary of AP scores Metric RunID Hop AP SF-ALL-Macro hltcoe_KB_CMN_3 0 0.0000 SF-ALL-Macro hltcoe_KB_CMN_3 ALL 0.0000 LDC-MEAN-ALL-Macro hltcoe_KB_CMN_3 0 0.0000 LDC-MEAN-ALL-Macro hltcoe_KB_CMN_3 ALL 0.0000 (PRIMARY METRIC) *ALL-Macro AP refer to mean of corresponding AP values. Scores based on confidence values and Average Precision (AP); only k=1 justification allowed: SUMMARY: This section provides summary of AP scores Metric RunID Hop AP SF-ALL-Macro hltcoe_KB_CMN_3 0 0.0000 SF-ALL-Macro hltcoe_KB_CMN_3 ALL 0.0000 LDC-MEAN-ALL-Macro hltcoe_KB_CMN_3 0 0.0000 LDC-MEAN-ALL-Macro hltcoe_KB_CMN_3 ALL 0.0000 *ALL-Macro AP refer to mean of corresponding AP values. Scores based on P/R/F1; only k=1 justification allowed: Metric RunID Hop Prec Recall F1 SF-ALL-Micro hltcoe_KB_CMN_3 0 0.0000 0.0000 0.0000 SF-ALL-Micro hltcoe_KB_CMN_3 ALL 0.0000 0.0000 0.0000 SF-ALL-Macro hltcoe_KB_CMN_3 0 0.0000 0.0000 0.0000 SF-ALL-Macro hltcoe_KB_CMN_3 ALL 0.0000 0.0000 0.0000 LDC-MAX-ALL-Micro hltcoe_KB_CMN_3 0 0.0000 0.0000 0.0000 LDC-MAX-ALL-Micro hltcoe_KB_CMN_3 ALL 0.0000 0.0000 0.0000 LDC-MAX-ALL-Macro hltcoe_KB_CMN_3 0 0.0000 0.0000 0.0000 LDC-MAX-ALL-Macro hltcoe_KB_CMN_3 ALL 0.0000 0.0000 0.0000 LDC-MEAN-ALL-Macro hltcoe_KB_CMN_3 0 0.0000 0.0000 0.0000 LDC-MEAN-ALL-Macro hltcoe_KB_CMN_3 ALL 0.0000 0.0000 0.0000 *ALL-Macro Prec, Recall and F1 refer to mean-precision, mean-recall and mean-F1. --------------------- Composite KB Evaluation (SENTIMENT-SLOTS only): Scores based on confidence values and Average Precision (AP); k=3 justifications allowed: Metric RunID Hop AP SF-ALL-Macro hltcoe_KB_CMN_3 0 0.0000 SF-ALL-Macro hltcoe_KB_CMN_3 1 0.0000 SF-ALL-Macro hltcoe_KB_CMN_3 ALL 0.0000 LDC-MEAN-ALL-Macro hltcoe_KB_CMN_3 0 0.0000 LDC-MEAN-ALL-Macro hltcoe_KB_CMN_3 1 0.0000 LDC-MEAN-ALL-Macro hltcoe_KB_CMN_3 ALL 0.0000 (PRIMARY METRIC) *ALL-Macro AP refer to mean of corresponding AP values. Scores based on confidence values and Average Precision (AP); only k=1 justification allowed: Metric RunID Hop AP SF-ALL-Macro hltcoe_KB_CMN_3 0 0.0000 SF-ALL-Macro hltcoe_KB_CMN_3 1 0.0000 SF-ALL-Macro hltcoe_KB_CMN_3 ALL 0.0000 LDC-MEAN-ALL-Macro hltcoe_KB_CMN_3 0 0.0000 LDC-MEAN-ALL-Macro hltcoe_KB_CMN_3 1 0.0000 LDC-MEAN-ALL-Macro hltcoe_KB_CMN_3 ALL 0.0000 *ALL-Macro AP refer to mean of corresponding AP values. Scores based on P/R/F1; only k=1 justification allowed: Metric RunID Hop Prec Recall F1 SF-ALL-Micro hltcoe_KB_CMN_3 0 0.0000 0.0000 0.0000 SF-ALL-Micro hltcoe_KB_CMN_3 1 0.0000 0.0000 0.0000 SF-ALL-Micro hltcoe_KB_CMN_3 ALL 0.0000 0.0000 0.0000 SF-ALL-Macro hltcoe_KB_CMN_3 0 0.0000 0.0000 0.0000 SF-ALL-Macro hltcoe_KB_CMN_3 1 0.0000 0.0000 0.0000 SF-ALL-Macro hltcoe_KB_CMN_3 ALL 0.0000 0.0000 0.0000 LDC-MAX-ALL-Micro hltcoe_KB_CMN_3 0 0.0000 0.0000 0.0000 LDC-MAX-ALL-Micro hltcoe_KB_CMN_3 1 0.0000 0.0000 0.0000 LDC-MAX-ALL-Micro hltcoe_KB_CMN_3 ALL 0.0000 0.0000 0.0000 LDC-MAX-ALL-Macro hltcoe_KB_CMN_3 0 0.0000 0.0000 0.0000 LDC-MAX-ALL-Macro hltcoe_KB_CMN_3 1 0.0000 0.0000 0.0000 LDC-MAX-ALL-Macro hltcoe_KB_CMN_3 ALL 0.0000 0.0000 0.0000 LDC-MEAN-ALL-Macro hltcoe_KB_CMN_3 0 0.0000 0.0000 0.0000 LDC-MEAN-ALL-Macro hltcoe_KB_CMN_3 1 0.0000 0.0000 0.0000 LDC-MEAN-ALL-Macro hltcoe_KB_CMN_3 ALL 0.0000 0.0000 0.0000 *ALL-Macro Prec, Recall and F1 refer to mean-precision, mean-recall and mean-F1.