===================================================================== TAC KBP 2017 SPANISH KB CONSTRUCTION: COMPOSITE KB EVALUATION RESULTS ===================================================================== Team ID: TinkerBell Organization: RPI, UIUC, Stanford, Columbia, Cornell, JHU, UPenn ************************************************************* Run ID: TinkerBell_KB_SPA_1 Did the run access the live Web during the evaluation window: No Did the run extract relations from the Cold Start source corpus: Yes Did the run generate meaningful confidence values: Yes Did the run attempt the full Entity Discovery and Linking task: Yes Did the run attempt the full Event Nugget Detection and Coreference task: Yes Did the run include SF relations: Yes Did the run attempt the Event Argument Extraction and Linking task: Yes Did the run include Sentiment relations: Yes --------------------- Composite KB Evaluation (All slot types): Scores based on confidence values and Average Precision (AP); k=3 justifications allowed: Metric RunID Hop AP SF-ALL-Macro TinkerBell_KB_SPA_1 0 0.1018 SF-ALL-Macro TinkerBell_KB_SPA_1 1 0.0122 SF-ALL-Macro TinkerBell_KB_SPA_1 ALL 0.0705 LDC-MEAN-ALL-Macro TinkerBell_KB_SPA_1 0 0.1104 LDC-MEAN-ALL-Macro TinkerBell_KB_SPA_1 1 0.0106 LDC-MEAN-ALL-Macro TinkerBell_KB_SPA_1 ALL 0.0755 (PRIMARY METRIC) *ALL-Macro AP refer to mean of corresponding AP values. Scores based on confidence values and Average Precision (AP); only k=1 justification allowed: Metric RunID Hop AP SF-ALL-Macro TinkerBell_KB_SPA_1 0 0.1271 SF-ALL-Macro TinkerBell_KB_SPA_1 1 0.0147 SF-ALL-Macro TinkerBell_KB_SPA_1 ALL 0.0892 LDC-MEAN-ALL-Macro TinkerBell_KB_SPA_1 0 0.1409 LDC-MEAN-ALL-Macro TinkerBell_KB_SPA_1 1 0.0128 LDC-MEAN-ALL-Macro TinkerBell_KB_SPA_1 ALL 0.0975 *ALL-Macro AP refer to mean of corresponding AP values. Scores based on P/R/F1; only k=1 justification allowed: Metric RunID Hop Prec Recall F1 SF-ALL-Micro TinkerBell_KB_SPA_1 0 0.1086 0.2500 0.1514 SF-ALL-Micro TinkerBell_KB_SPA_1 1 0.0091 0.0493 0.0154 SF-ALL-Micro TinkerBell_KB_SPA_1 ALL 0.0537 0.1812 0.0829 SF-ALL-Macro TinkerBell_KB_SPA_1 0 0.1311 0.1721 0.1269 SF-ALL-Macro TinkerBell_KB_SPA_1 1 0.0553 0.0553 0.0536 SF-ALL-Macro TinkerBell_KB_SPA_1 ALL 0.1028 0.1285 0.0995 LDC-MAX-ALL-Micro TinkerBell_KB_SPA_1 0 0.1050 0.2877 0.1538 LDC-MAX-ALL-Micro TinkerBell_KB_SPA_1 1 0.0052 0.0424 0.0092 LDC-MAX-ALL-Micro TinkerBell_KB_SPA_1 ALL 0.0423 0.1991 0.0698 LDC-MAX-ALL-Macro TinkerBell_KB_SPA_1 0 0.1560 0.2003 0.1535 LDC-MAX-ALL-Macro TinkerBell_KB_SPA_1 1 0.0495 0.0495 0.0480 LDC-MAX-ALL-Macro TinkerBell_KB_SPA_1 ALL 0.1171 0.1453 0.1150 LDC-MEAN-ALL-Macro TinkerBell_KB_SPA_1 0 0.1457 0.1825 0.1402 LDC-MEAN-ALL-Macro TinkerBell_KB_SPA_1 1 0.0495 0.0495 0.0480 LDC-MEAN-ALL-Macro TinkerBell_KB_SPA_1 ALL 0.1106 0.1340 0.1066 *ALL-Macro Prec, Recall and F1 refer to mean-precision, mean-recall and mean-F1. --------------------- Composite KB Evaluation (SF-SLOTS only): Scores based on confidence values and Average Precision (AP); k=3 justifications allowed: Metric RunID Hop AP SF-ALL-Macro TinkerBell_KB_SPA_1 0 0.1714 SF-ALL-Macro TinkerBell_KB_SPA_1 1 0.0257 SF-ALL-Macro TinkerBell_KB_SPA_1 ALL 0.1155 LDC-MEAN-ALL-Macro TinkerBell_KB_SPA_1 0 0.1797 LDC-MEAN-ALL-Macro TinkerBell_KB_SPA_1 1 0.0217 LDC-MEAN-ALL-Macro TinkerBell_KB_SPA_1 ALL 0.1190 (PRIMARY METRIC) *ALL-Macro AP refer to mean of corresponding AP values. Scores based on confidence values and Average Precision (AP); only k=1 justification allowed: Metric RunID Hop AP SF-ALL-Macro TinkerBell_KB_SPA_1 0 0.2167 SF-ALL-Macro TinkerBell_KB_SPA_1 1 0.0309 SF-ALL-Macro TinkerBell_KB_SPA_1 ALL 0.1488 LDC-MEAN-ALL-Macro TinkerBell_KB_SPA_1 0 0.2327 LDC-MEAN-ALL-Macro TinkerBell_KB_SPA_1 1 0.0261 LDC-MEAN-ALL-Macro TinkerBell_KB_SPA_1 ALL 0.1573 *ALL-Macro AP refer to mean of corresponding AP values. Scores based on P/R/F1; only k=1 justification allowed: Metric RunID Hop Prec Recall F1 SF-ALL-Micro TinkerBell_KB_SPA_1 0 0.2215 0.2682 0.2426 SF-ALL-Micro TinkerBell_KB_SPA_1 1 0.3684 0.0903 0.1451 SF-ALL-Micro TinkerBell_KB_SPA_1 ALL 0.2373 0.2019 0.2182 SF-ALL-Macro TinkerBell_KB_SPA_1 0 0.2209 0.2312 0.2123 SF-ALL-Macro TinkerBell_KB_SPA_1 1 0.1028 0.1028 0.0997 SF-ALL-Macro TinkerBell_KB_SPA_1 ALL 0.1775 0.1840 0.1709 LDC-MAX-ALL-Micro TinkerBell_KB_SPA_1 0 0.1847 0.3067 0.2306 LDC-MAX-ALL-Micro TinkerBell_KB_SPA_1 1 0.2800 0.0753 0.1186 LDC-MAX-ALL-Micro TinkerBell_KB_SPA_1 ALL 0.1934 0.2181 0.2050 LDC-MAX-ALL-Macro TinkerBell_KB_SPA_1 0 0.2552 0.2692 0.2497 LDC-MAX-ALL-Macro TinkerBell_KB_SPA_1 1 0.0917 0.0917 0.0889 LDC-MAX-ALL-Macro TinkerBell_KB_SPA_1 ALL 0.1975 0.2065 0.1929 LDC-MEAN-ALL-Macro TinkerBell_KB_SPA_1 0 0.2370 0.2449 0.2283 LDC-MEAN-ALL-Macro TinkerBell_KB_SPA_1 1 0.0917 0.0917 0.0889 LDC-MEAN-ALL-Macro TinkerBell_KB_SPA_1 ALL 0.1857 0.1908 0.1791 *ALL-Macro Prec, Recall and F1 refer to mean-precision, mean-recall and mean-F1. --------------------- Composite KB Evaluation (EVENT-SLOTS only): Scores based on confidence values and Average Precision (AP); k=3 justifications allowed: SUMMARY: This section provides summary of AP scores Metric RunID Hop AP SF-ALL-Macro TinkerBell_KB_SPA_1 0 0.0028 SF-ALL-Macro TinkerBell_KB_SPA_1 ALL 0.0028 LDC-MEAN-ALL-Macro TinkerBell_KB_SPA_1 0 0.0051 LDC-MEAN-ALL-Macro TinkerBell_KB_SPA_1 ALL 0.0051 (PRIMARY METRIC) *ALL-Macro AP refer to mean of corresponding AP values. Scores based on confidence values and Average Precision (AP); only k=1 justification allowed: SUMMARY: This section provides summary of AP scores Metric RunID Hop AP SF-ALL-Macro TinkerBell_KB_SPA_1 0 0.0028 SF-ALL-Macro TinkerBell_KB_SPA_1 ALL 0.0028 LDC-MEAN-ALL-Macro TinkerBell_KB_SPA_1 0 0.0051 LDC-MEAN-ALL-Macro TinkerBell_KB_SPA_1 ALL 0.0051 *ALL-Macro AP refer to mean of corresponding AP values. Scores based on P/R/F1; only k=1 justification allowed: Metric RunID Hop Prec Recall F1 SF-ALL-Micro TinkerBell_KB_SPA_1 0 0.2500 0.0086 0.0167 SF-ALL-Micro TinkerBell_KB_SPA_1 ALL 0.2500 0.0086 0.0167 SF-ALL-Macro TinkerBell_KB_SPA_1 0 0.0114 0.0028 0.0045 SF-ALL-Macro TinkerBell_KB_SPA_1 ALL 0.0114 0.0028 0.0045 LDC-MAX-ALL-Micro TinkerBell_KB_SPA_1 0 0.3333 0.0152 0.0290 LDC-MAX-ALL-Micro TinkerBell_KB_SPA_1 ALL 0.3333 0.0152 0.0290 LDC-MAX-ALL-Macro TinkerBell_KB_SPA_1 0 0.0204 0.0051 0.0082 LDC-MAX-ALL-Macro TinkerBell_KB_SPA_1 ALL 0.0204 0.0051 0.0082 LDC-MEAN-ALL-Macro TinkerBell_KB_SPA_1 0 0.0204 0.0051 0.0082 LDC-MEAN-ALL-Macro TinkerBell_KB_SPA_1 ALL 0.0204 0.0051 0.0082 *ALL-Macro Prec, Recall and F1 refer to mean-precision, mean-recall and mean-F1. --------------------- Composite KB Evaluation (SENTIMENT-SLOTS only): Scores based on confidence values and Average Precision (AP); k=3 justifications allowed: Metric RunID Hop AP SF-ALL-Macro TinkerBell_KB_SPA_1 0 0.0358 SF-ALL-Macro TinkerBell_KB_SPA_1 1 0.0000 SF-ALL-Macro TinkerBell_KB_SPA_1 ALL 0.0333 LDC-MEAN-ALL-Macro TinkerBell_KB_SPA_1 0 0.0380 LDC-MEAN-ALL-Macro TinkerBell_KB_SPA_1 1 0.0000 LDC-MEAN-ALL-Macro TinkerBell_KB_SPA_1 ALL 0.0364 (PRIMARY METRIC) *ALL-Macro AP refer to mean of corresponding AP values. Scores based on confidence values and Average Precision (AP); only k=1 justification allowed: Metric RunID Hop AP SF-ALL-Macro TinkerBell_KB_SPA_1 0 0.0375 SF-ALL-Macro TinkerBell_KB_SPA_1 1 0.0000 SF-ALL-Macro TinkerBell_KB_SPA_1 ALL 0.0347 LDC-MEAN-ALL-Macro TinkerBell_KB_SPA_1 0 0.0394 LDC-MEAN-ALL-Macro TinkerBell_KB_SPA_1 1 0.0000 LDC-MEAN-ALL-Macro TinkerBell_KB_SPA_1 ALL 0.0376 *ALL-Macro AP refer to mean of corresponding AP values. Scores based on P/R/F1; only k=1 justification allowed: Metric RunID Hop Prec Recall F1 SF-ALL-Micro TinkerBell_KB_SPA_1 0 0.0697 0.3892 0.1183 SF-ALL-Micro TinkerBell_KB_SPA_1 1 0.0000 0.0000 0.0000 SF-ALL-Micro TinkerBell_KB_SPA_1 ALL 0.0643 0.3351 0.1079 SF-ALL-Macro TinkerBell_KB_SPA_1 0 0.0343 0.2367 0.0473 SF-ALL-Macro TinkerBell_KB_SPA_1 1 0.0000 0.0000 0.0000 SF-ALL-Macro TinkerBell_KB_SPA_1 ALL 0.0245 0.1687 0.0337 LDC-MAX-ALL-Micro TinkerBell_KB_SPA_1 0 0.0678 0.4868 0.1190 LDC-MAX-ALL-Micro TinkerBell_KB_SPA_1 1 0.0000 0.0000 0.0000 LDC-MAX-ALL-Micro TinkerBell_KB_SPA_1 ALL 0.0632 0.4066 0.1095 LDC-MAX-ALL-Macro TinkerBell_KB_SPA_1 0 0.0304 0.2589 0.0519 LDC-MAX-ALL-Macro TinkerBell_KB_SPA_1 1 0.0000 0.0000 0.0000 LDC-MAX-ALL-Macro TinkerBell_KB_SPA_1 ALL 0.0220 0.1873 0.0376 LDC-MEAN-ALL-Macro TinkerBell_KB_SPA_1 0 0.0309 0.2365 0.0458 LDC-MEAN-ALL-Macro TinkerBell_KB_SPA_1 1 0.0000 0.0000 0.0000 LDC-MEAN-ALL-Macro TinkerBell_KB_SPA_1 ALL 0.0224 0.1711 0.0331 *ALL-Macro Prec, Recall and F1 refer to mean-precision, mean-recall and mean-F1. ************************************************************* Run ID: TinkerBell_KB_SPA_2 Did the run access the live Web during the evaluation window: No Did the run extract relations from the Cold Start source corpus: Yes Did the run generate meaningful confidence values: Yes Did the run attempt the full Entity Discovery and Linking task: Yes Did the run attempt the full Event Nugget Detection and Coreference task: Yes Did the run include SF relations: Yes Did the run attempt the Event Argument Extraction and Linking task: Yes Did the run include Sentiment relations: Yes --------------------- Composite KB Evaluation (All slot types): Scores based on confidence values and Average Precision (AP); k=3 justifications allowed: Metric RunID Hop AP SF-ALL-Macro TinkerBell_KB_SPA_2 0 0.1018 SF-ALL-Macro TinkerBell_KB_SPA_2 1 0.0122 SF-ALL-Macro TinkerBell_KB_SPA_2 ALL 0.0705 LDC-MEAN-ALL-Macro TinkerBell_KB_SPA_2 0 0.1104 LDC-MEAN-ALL-Macro TinkerBell_KB_SPA_2 1 0.0106 LDC-MEAN-ALL-Macro TinkerBell_KB_SPA_2 ALL 0.0755 (PRIMARY METRIC) *ALL-Macro AP refer to mean of corresponding AP values. Scores based on confidence values and Average Precision (AP); only k=1 justification allowed: Metric RunID Hop AP SF-ALL-Macro TinkerBell_KB_SPA_2 0 0.1271 SF-ALL-Macro TinkerBell_KB_SPA_2 1 0.0147 SF-ALL-Macro TinkerBell_KB_SPA_2 ALL 0.0892 LDC-MEAN-ALL-Macro TinkerBell_KB_SPA_2 0 0.1409 LDC-MEAN-ALL-Macro TinkerBell_KB_SPA_2 1 0.0128 LDC-MEAN-ALL-Macro TinkerBell_KB_SPA_2 ALL 0.0975 *ALL-Macro AP refer to mean of corresponding AP values. Scores based on P/R/F1; only k=1 justification allowed: Metric RunID Hop Prec Recall F1 SF-ALL-Micro TinkerBell_KB_SPA_2 0 0.1088 0.2500 0.1516 SF-ALL-Micro TinkerBell_KB_SPA_2 1 0.0091 0.0493 0.0154 SF-ALL-Micro TinkerBell_KB_SPA_2 ALL 0.0538 0.1812 0.0829 SF-ALL-Macro TinkerBell_KB_SPA_2 0 0.1311 0.1721 0.1269 SF-ALL-Macro TinkerBell_KB_SPA_2 1 0.0553 0.0553 0.0536 SF-ALL-Macro TinkerBell_KB_SPA_2 ALL 0.1028 0.1285 0.0995 LDC-MAX-ALL-Micro TinkerBell_KB_SPA_2 0 0.1053 0.2877 0.1541 LDC-MAX-ALL-Micro TinkerBell_KB_SPA_2 1 0.0052 0.0424 0.0092 LDC-MAX-ALL-Micro TinkerBell_KB_SPA_2 ALL 0.0423 0.1991 0.0698 LDC-MAX-ALL-Macro TinkerBell_KB_SPA_2 0 0.1560 0.2003 0.1535 LDC-MAX-ALL-Macro TinkerBell_KB_SPA_2 1 0.0495 0.0495 0.0480 LDC-MAX-ALL-Macro TinkerBell_KB_SPA_2 ALL 0.1171 0.1453 0.1150 LDC-MEAN-ALL-Macro TinkerBell_KB_SPA_2 0 0.1457 0.1825 0.1403 LDC-MEAN-ALL-Macro TinkerBell_KB_SPA_2 1 0.0495 0.0495 0.0480 LDC-MEAN-ALL-Macro TinkerBell_KB_SPA_2 ALL 0.1106 0.1340 0.1066 *ALL-Macro Prec, Recall and F1 refer to mean-precision, mean-recall and mean-F1. --------------------- Composite KB Evaluation (SF-SLOTS only): Scores based on confidence values and Average Precision (AP); k=3 justifications allowed: Metric RunID Hop AP SF-ALL-Macro TinkerBell_KB_SPA_2 0 0.1714 SF-ALL-Macro TinkerBell_KB_SPA_2 1 0.0257 SF-ALL-Macro TinkerBell_KB_SPA_2 ALL 0.1155 LDC-MEAN-ALL-Macro TinkerBell_KB_SPA_2 0 0.1797 LDC-MEAN-ALL-Macro TinkerBell_KB_SPA_2 1 0.0217 LDC-MEAN-ALL-Macro TinkerBell_KB_SPA_2 ALL 0.1190 (PRIMARY METRIC) *ALL-Macro AP refer to mean of corresponding AP values. Scores based on confidence values and Average Precision (AP); only k=1 justification allowed: Metric RunID Hop AP SF-ALL-Macro TinkerBell_KB_SPA_2 0 0.2167 SF-ALL-Macro TinkerBell_KB_SPA_2 1 0.0309 SF-ALL-Macro TinkerBell_KB_SPA_2 ALL 0.1488 LDC-MEAN-ALL-Macro TinkerBell_KB_SPA_2 0 0.2327 LDC-MEAN-ALL-Macro TinkerBell_KB_SPA_2 1 0.0261 LDC-MEAN-ALL-Macro TinkerBell_KB_SPA_2 ALL 0.1573 *ALL-Macro AP refer to mean of corresponding AP values. Scores based on P/R/F1; only k=1 justification allowed: Metric RunID Hop Prec Recall F1 SF-ALL-Micro TinkerBell_KB_SPA_2 0 0.2215 0.2682 0.2426 SF-ALL-Micro TinkerBell_KB_SPA_2 1 0.3684 0.0903 0.1451 SF-ALL-Micro TinkerBell_KB_SPA_2 ALL 0.2373 0.2019 0.2182 SF-ALL-Macro TinkerBell_KB_SPA_2 0 0.2209 0.2312 0.2123 SF-ALL-Macro TinkerBell_KB_SPA_2 1 0.1028 0.1028 0.0997 SF-ALL-Macro TinkerBell_KB_SPA_2 ALL 0.1775 0.1840 0.1709 LDC-MAX-ALL-Micro TinkerBell_KB_SPA_2 0 0.1840 0.3067 0.2300 LDC-MAX-ALL-Micro TinkerBell_KB_SPA_2 1 0.2800 0.0753 0.1186 LDC-MAX-ALL-Micro TinkerBell_KB_SPA_2 ALL 0.1927 0.2181 0.2046 LDC-MAX-ALL-Macro TinkerBell_KB_SPA_2 0 0.2552 0.2692 0.2497 LDC-MAX-ALL-Macro TinkerBell_KB_SPA_2 1 0.0917 0.0917 0.0889 LDC-MAX-ALL-Macro TinkerBell_KB_SPA_2 ALL 0.1975 0.2065 0.1929 LDC-MEAN-ALL-Macro TinkerBell_KB_SPA_2 0 0.2370 0.2449 0.2283 LDC-MEAN-ALL-Macro TinkerBell_KB_SPA_2 1 0.0917 0.0917 0.0889 LDC-MEAN-ALL-Macro TinkerBell_KB_SPA_2 ALL 0.1857 0.1908 0.1791 *ALL-Macro Prec, Recall and F1 refer to mean-precision, mean-recall and mean-F1. --------------------- Composite KB Evaluation (EVENT-SLOTS only): Scores based on confidence values and Average Precision (AP); k=3 justifications allowed: SUMMARY: This section provides summary of AP scores Metric RunID Hop AP SF-ALL-Macro TinkerBell_KB_SPA_2 0 0.0028 SF-ALL-Macro TinkerBell_KB_SPA_2 ALL 0.0028 LDC-MEAN-ALL-Macro TinkerBell_KB_SPA_2 0 0.0051 LDC-MEAN-ALL-Macro TinkerBell_KB_SPA_2 ALL 0.0051 (PRIMARY METRIC) *ALL-Macro AP refer to mean of corresponding AP values. Scores based on confidence values and Average Precision (AP); only k=1 justification allowed: SUMMARY: This section provides summary of AP scores Metric RunID Hop AP SF-ALL-Macro TinkerBell_KB_SPA_2 0 0.0028 SF-ALL-Macro TinkerBell_KB_SPA_2 ALL 0.0028 LDC-MEAN-ALL-Macro TinkerBell_KB_SPA_2 0 0.0051 LDC-MEAN-ALL-Macro TinkerBell_KB_SPA_2 ALL 0.0051 *ALL-Macro AP refer to mean of corresponding AP values. Scores based on P/R/F1; only k=1 justification allowed: Metric RunID Hop Prec Recall F1 SF-ALL-Micro TinkerBell_KB_SPA_2 0 0.2500 0.0086 0.0167 SF-ALL-Micro TinkerBell_KB_SPA_2 ALL 0.2500 0.0086 0.0167 SF-ALL-Macro TinkerBell_KB_SPA_2 0 0.0114 0.0028 0.0045 SF-ALL-Macro TinkerBell_KB_SPA_2 ALL 0.0114 0.0028 0.0045 LDC-MAX-ALL-Micro TinkerBell_KB_SPA_2 0 0.3333 0.0152 0.0290 LDC-MAX-ALL-Micro TinkerBell_KB_SPA_2 ALL 0.3333 0.0152 0.0290 LDC-MAX-ALL-Macro TinkerBell_KB_SPA_2 0 0.0204 0.0051 0.0082 LDC-MAX-ALL-Macro TinkerBell_KB_SPA_2 ALL 0.0204 0.0051 0.0082 LDC-MEAN-ALL-Macro TinkerBell_KB_SPA_2 0 0.0204 0.0051 0.0082 LDC-MEAN-ALL-Macro TinkerBell_KB_SPA_2 ALL 0.0204 0.0051 0.0082 *ALL-Macro Prec, Recall and F1 refer to mean-precision, mean-recall and mean-F1. --------------------- Composite KB Evaluation (SENTIMENT-SLOTS only): Scores based on confidence values and Average Precision (AP); k=3 justifications allowed: Metric RunID Hop AP SF-ALL-Macro TinkerBell_KB_SPA_2 0 0.0358 SF-ALL-Macro TinkerBell_KB_SPA_2 1 0.0000 SF-ALL-Macro TinkerBell_KB_SPA_2 ALL 0.0333 LDC-MEAN-ALL-Macro TinkerBell_KB_SPA_2 0 0.0380 LDC-MEAN-ALL-Macro TinkerBell_KB_SPA_2 1 0.0000 LDC-MEAN-ALL-Macro TinkerBell_KB_SPA_2 ALL 0.0364 (PRIMARY METRIC) *ALL-Macro AP refer to mean of corresponding AP values. Scores based on confidence values and Average Precision (AP); only k=1 justification allowed: Metric RunID Hop AP SF-ALL-Macro TinkerBell_KB_SPA_2 0 0.0375 SF-ALL-Macro TinkerBell_KB_SPA_2 1 0.0000 SF-ALL-Macro TinkerBell_KB_SPA_2 ALL 0.0347 LDC-MEAN-ALL-Macro TinkerBell_KB_SPA_2 0 0.0394 LDC-MEAN-ALL-Macro TinkerBell_KB_SPA_2 1 0.0000 LDC-MEAN-ALL-Macro TinkerBell_KB_SPA_2 ALL 0.0376 *ALL-Macro AP refer to mean of corresponding AP values. Scores based on P/R/F1; only k=1 justification allowed: Metric RunID Hop Prec Recall F1 SF-ALL-Micro TinkerBell_KB_SPA_2 0 0.0699 0.3892 0.1185 SF-ALL-Micro TinkerBell_KB_SPA_2 1 0.0000 0.0000 0.0000 SF-ALL-Micro TinkerBell_KB_SPA_2 ALL 0.0644 0.3351 0.1081 SF-ALL-Macro TinkerBell_KB_SPA_2 0 0.0343 0.2367 0.0473 SF-ALL-Macro TinkerBell_KB_SPA_2 1 0.0000 0.0000 0.0000 SF-ALL-Macro TinkerBell_KB_SPA_2 ALL 0.0245 0.1687 0.0337 LDC-MAX-ALL-Micro TinkerBell_KB_SPA_2 0 0.0679 0.4868 0.1192 LDC-MAX-ALL-Micro TinkerBell_KB_SPA_2 1 0.0000 0.0000 0.0000 LDC-MAX-ALL-Micro TinkerBell_KB_SPA_2 ALL 0.0634 0.4066 0.1096 LDC-MAX-ALL-Macro TinkerBell_KB_SPA_2 0 0.0304 0.2589 0.0519 LDC-MAX-ALL-Macro TinkerBell_KB_SPA_2 1 0.0000 0.0000 0.0000 LDC-MAX-ALL-Macro TinkerBell_KB_SPA_2 ALL 0.0220 0.1873 0.0376 LDC-MEAN-ALL-Macro TinkerBell_KB_SPA_2 0 0.0309 0.2365 0.0458 LDC-MEAN-ALL-Macro TinkerBell_KB_SPA_2 1 0.0000 0.0000 0.0000 LDC-MEAN-ALL-Macro TinkerBell_KB_SPA_2 ALL 0.0224 0.1711 0.0331 *ALL-Macro Prec, Recall and F1 refer to mean-precision, mean-recall and mean-F1. ************************************************************* Run ID: TinkerBell_KB_SPA_3 Did the run access the live Web during the evaluation window: No Did the run extract relations from the Cold Start source corpus: Yes Did the run generate meaningful confidence values: Yes Did the run attempt the full Entity Discovery and Linking task: Yes Did the run attempt the full Event Nugget Detection and Coreference task: Yes Did the run include SF relations: Yes Did the run attempt the Event Argument Extraction and Linking task: Yes Did the run include Sentiment relations: Yes --------------------- Composite KB Evaluation (All slot types): Scores based on confidence values and Average Precision (AP); k=3 justifications allowed: Metric RunID Hop AP SF-ALL-Macro TinkerBell_KB_SPA_3 0 0.1052 SF-ALL-Macro TinkerBell_KB_SPA_3 1 0.0122 SF-ALL-Macro TinkerBell_KB_SPA_3 ALL 0.0739 LDC-MEAN-ALL-Macro TinkerBell_KB_SPA_3 0 0.1145 LDC-MEAN-ALL-Macro TinkerBell_KB_SPA_3 1 0.0106 LDC-MEAN-ALL-Macro TinkerBell_KB_SPA_3 ALL 0.0795 (PRIMARY METRIC) *ALL-Macro AP refer to mean of corresponding AP values. Scores based on confidence values and Average Precision (AP); only k=1 justification allowed: Metric RunID Hop AP SF-ALL-Macro TinkerBell_KB_SPA_3 0 0.1305 SF-ALL-Macro TinkerBell_KB_SPA_3 1 0.0147 SF-ALL-Macro TinkerBell_KB_SPA_3 ALL 0.0925 LDC-MEAN-ALL-Macro TinkerBell_KB_SPA_3 0 0.1450 LDC-MEAN-ALL-Macro TinkerBell_KB_SPA_3 1 0.0128 LDC-MEAN-ALL-Macro TinkerBell_KB_SPA_3 ALL 0.1016 *ALL-Macro AP refer to mean of corresponding AP values. Scores based on P/R/F1; only k=1 justification allowed: Metric RunID Hop Prec Recall F1 SF-ALL-Micro TinkerBell_KB_SPA_3 0 0.1200 0.2757 0.1672 SF-ALL-Micro TinkerBell_KB_SPA_3 1 0.0091 0.0493 0.0154 SF-ALL-Micro TinkerBell_KB_SPA_3 ALL 0.0588 0.1981 0.0907 SF-ALL-Macro TinkerBell_KB_SPA_3 0 0.1318 0.1767 0.1281 SF-ALL-Macro TinkerBell_KB_SPA_3 1 0.0553 0.0553 0.0536 SF-ALL-Macro TinkerBell_KB_SPA_3 ALL 0.1032 0.1314 0.1003 LDC-MAX-ALL-Micro TinkerBell_KB_SPA_3 0 0.1139 0.3116 0.1668 LDC-MAX-ALL-Micro TinkerBell_KB_SPA_3 1 0.0052 0.0424 0.0092 LDC-MAX-ALL-Micro TinkerBell_KB_SPA_3 ALL 0.0456 0.2144 0.0752 LDC-MAX-ALL-Macro TinkerBell_KB_SPA_3 0 0.1566 0.2043 0.1546 LDC-MAX-ALL-Macro TinkerBell_KB_SPA_3 1 0.0495 0.0495 0.0480 LDC-MAX-ALL-Macro TinkerBell_KB_SPA_3 ALL 0.1175 0.1478 0.1157 LDC-MEAN-ALL-Macro TinkerBell_KB_SPA_3 0 0.1463 0.1866 0.1413 LDC-MEAN-ALL-Macro TinkerBell_KB_SPA_3 1 0.0495 0.0495 0.0480 LDC-MEAN-ALL-Macro TinkerBell_KB_SPA_3 ALL 0.1110 0.1365 0.1073 *ALL-Macro Prec, Recall and F1 refer to mean-precision, mean-recall and mean-F1. --------------------- Composite KB Evaluation (SF-SLOTS only): Scores based on confidence values and Average Precision (AP); k=3 justifications allowed: Metric RunID Hop AP SF-ALL-Macro TinkerBell_KB_SPA_3 0 0.1714 SF-ALL-Macro TinkerBell_KB_SPA_3 1 0.0257 SF-ALL-Macro TinkerBell_KB_SPA_3 ALL 0.1155 LDC-MEAN-ALL-Macro TinkerBell_KB_SPA_3 0 0.1797 LDC-MEAN-ALL-Macro TinkerBell_KB_SPA_3 1 0.0217 LDC-MEAN-ALL-Macro TinkerBell_KB_SPA_3 ALL 0.1190 (PRIMARY METRIC) *ALL-Macro AP refer to mean of corresponding AP values. Scores based on confidence values and Average Precision (AP); only k=1 justification allowed: Metric RunID Hop AP SF-ALL-Macro TinkerBell_KB_SPA_3 0 0.2167 SF-ALL-Macro TinkerBell_KB_SPA_3 1 0.0309 SF-ALL-Macro TinkerBell_KB_SPA_3 ALL 0.1488 LDC-MEAN-ALL-Macro TinkerBell_KB_SPA_3 0 0.2327 LDC-MEAN-ALL-Macro TinkerBell_KB_SPA_3 1 0.0261 LDC-MEAN-ALL-Macro TinkerBell_KB_SPA_3 ALL 0.1573 *ALL-Macro AP refer to mean of corresponding AP values. Scores based on P/R/F1; only k=1 justification allowed: Metric RunID Hop Prec Recall F1 SF-ALL-Micro TinkerBell_KB_SPA_3 0 0.2215 0.2682 0.2426 SF-ALL-Micro TinkerBell_KB_SPA_3 1 0.3684 0.0903 0.1451 SF-ALL-Micro TinkerBell_KB_SPA_3 ALL 0.2373 0.2019 0.2182 SF-ALL-Macro TinkerBell_KB_SPA_3 0 0.2209 0.2312 0.2123 SF-ALL-Macro TinkerBell_KB_SPA_3 1 0.1028 0.1028 0.0997 SF-ALL-Macro TinkerBell_KB_SPA_3 ALL 0.1775 0.1840 0.1709 LDC-MAX-ALL-Micro TinkerBell_KB_SPA_3 0 0.1847 0.3067 0.2306 LDC-MAX-ALL-Micro TinkerBell_KB_SPA_3 1 0.2800 0.0753 0.1186 LDC-MAX-ALL-Micro TinkerBell_KB_SPA_3 ALL 0.1934 0.2181 0.2050 LDC-MAX-ALL-Macro TinkerBell_KB_SPA_3 0 0.2552 0.2692 0.2497 LDC-MAX-ALL-Macro TinkerBell_KB_SPA_3 1 0.0917 0.0917 0.0889 LDC-MAX-ALL-Macro TinkerBell_KB_SPA_3 ALL 0.1975 0.2065 0.1929 LDC-MEAN-ALL-Macro TinkerBell_KB_SPA_3 0 0.2370 0.2449 0.2283 LDC-MEAN-ALL-Macro TinkerBell_KB_SPA_3 1 0.0917 0.0917 0.0889 LDC-MEAN-ALL-Macro TinkerBell_KB_SPA_3 ALL 0.1857 0.1908 0.1791 *ALL-Macro Prec, Recall and F1 refer to mean-precision, mean-recall and mean-F1. --------------------- Composite KB Evaluation (EVENT-SLOTS only): Scores based on confidence values and Average Precision (AP); k=3 justifications allowed: SUMMARY: This section provides summary of AP scores Metric RunID Hop AP SF-ALL-Macro TinkerBell_KB_SPA_3 0 0.0028 SF-ALL-Macro TinkerBell_KB_SPA_3 ALL 0.0028 LDC-MEAN-ALL-Macro TinkerBell_KB_SPA_3 0 0.0051 LDC-MEAN-ALL-Macro TinkerBell_KB_SPA_3 ALL 0.0051 (PRIMARY METRIC) *ALL-Macro AP refer to mean of corresponding AP values. Scores based on confidence values and Average Precision (AP); only k=1 justification allowed: SUMMARY: This section provides summary of AP scores Metric RunID Hop AP SF-ALL-Macro TinkerBell_KB_SPA_3 0 0.0028 SF-ALL-Macro TinkerBell_KB_SPA_3 ALL 0.0028 LDC-MEAN-ALL-Macro TinkerBell_KB_SPA_3 0 0.0051 LDC-MEAN-ALL-Macro TinkerBell_KB_SPA_3 ALL 0.0051 *ALL-Macro AP refer to mean of corresponding AP values. Scores based on P/R/F1; only k=1 justification allowed: Metric RunID Hop Prec Recall F1 SF-ALL-Micro TinkerBell_KB_SPA_3 0 0.2500 0.0086 0.0167 SF-ALL-Micro TinkerBell_KB_SPA_3 ALL 0.2500 0.0086 0.0167 SF-ALL-Macro TinkerBell_KB_SPA_3 0 0.0114 0.0028 0.0045 SF-ALL-Macro TinkerBell_KB_SPA_3 ALL 0.0114 0.0028 0.0045 LDC-MAX-ALL-Micro TinkerBell_KB_SPA_3 0 0.3333 0.0152 0.0290 LDC-MAX-ALL-Micro TinkerBell_KB_SPA_3 ALL 0.3333 0.0152 0.0290 LDC-MAX-ALL-Macro TinkerBell_KB_SPA_3 0 0.0204 0.0051 0.0082 LDC-MAX-ALL-Macro TinkerBell_KB_SPA_3 ALL 0.0204 0.0051 0.0082 LDC-MEAN-ALL-Macro TinkerBell_KB_SPA_3 0 0.0204 0.0051 0.0082 LDC-MEAN-ALL-Macro TinkerBell_KB_SPA_3 ALL 0.0204 0.0051 0.0082 *ALL-Macro Prec, Recall and F1 refer to mean-precision, mean-recall and mean-F1. --------------------- Composite KB Evaluation (SENTIMENT-SLOTS only): Scores based on confidence values and Average Precision (AP); k=3 justifications allowed: Metric RunID Hop AP SF-ALL-Macro TinkerBell_KB_SPA_3 0 0.0539 SF-ALL-Macro TinkerBell_KB_SPA_3 1 0.0000 SF-ALL-Macro TinkerBell_KB_SPA_3 ALL 0.0513 LDC-MEAN-ALL-Macro TinkerBell_KB_SPA_3 0 0.0609 LDC-MEAN-ALL-Macro TinkerBell_KB_SPA_3 1 0.0000 LDC-MEAN-ALL-Macro TinkerBell_KB_SPA_3 ALL 0.0592 (PRIMARY METRIC) *ALL-Macro AP refer to mean of corresponding AP values. Scores based on confidence values and Average Precision (AP); only k=1 justification allowed: Metric RunID Hop AP SF-ALL-Macro TinkerBell_KB_SPA_3 0 0.0559 SF-ALL-Macro TinkerBell_KB_SPA_3 1 0.0000 SF-ALL-Macro TinkerBell_KB_SPA_3 ALL 0.0530 LDC-MEAN-ALL-Macro TinkerBell_KB_SPA_3 0 0.0625 LDC-MEAN-ALL-Macro TinkerBell_KB_SPA_3 1 0.0000 LDC-MEAN-ALL-Macro TinkerBell_KB_SPA_3 ALL 0.0606 *ALL-Macro AP refer to mean of corresponding AP values. Scores based on P/R/F1; only k=1 justification allowed: Metric RunID Hop Prec Recall F1 SF-ALL-Micro TinkerBell_KB_SPA_3 0 0.0849 0.4731 0.1440 SF-ALL-Micro TinkerBell_KB_SPA_3 1 0.0000 0.0000 0.0000 SF-ALL-Micro TinkerBell_KB_SPA_3 ALL 0.0783 0.4072 0.1313 SF-ALL-Macro TinkerBell_KB_SPA_3 0 0.0382 0.2618 0.0539 SF-ALL-Macro TinkerBell_KB_SPA_3 1 0.0000 0.0000 0.0000 SF-ALL-Macro TinkerBell_KB_SPA_3 ALL 0.0272 0.1866 0.0384 LDC-MAX-ALL-Micro TinkerBell_KB_SPA_3 0 0.0807 0.5789 0.1417 LDC-MAX-ALL-Micro TinkerBell_KB_SPA_3 1 0.0000 0.0000 0.0000 LDC-MAX-ALL-Micro TinkerBell_KB_SPA_3 ALL 0.0753 0.4835 0.1304 LDC-MAX-ALL-Macro TinkerBell_KB_SPA_3 0 0.0339 0.2817 0.0580 LDC-MAX-ALL-Macro TinkerBell_KB_SPA_3 1 0.0000 0.0000 0.0000 LDC-MAX-ALL-Macro TinkerBell_KB_SPA_3 ALL 0.0245 0.2038 0.0419 LDC-MEAN-ALL-Macro TinkerBell_KB_SPA_3 0 0.0344 0.2593 0.0518 LDC-MEAN-ALL-Macro TinkerBell_KB_SPA_3 1 0.0000 0.0000 0.0000 LDC-MEAN-ALL-Macro TinkerBell_KB_SPA_3 ALL 0.0249 0.1876 0.0375 *ALL-Macro Prec, Recall and F1 refer to mean-precision, mean-recall and mean-F1. ************************************************************* Run ID: TinkerBell_KB_SPA_4 Did the run access the live Web during the evaluation window: No Did the run extract relations from the Cold Start source corpus: Yes Did the run generate meaningful confidence values: Yes Did the run attempt the full Entity Discovery and Linking task: Yes Did the run attempt the full Event Nugget Detection and Coreference task: Yes Did the run include SF relations: Yes Did the run attempt the Event Argument Extraction and Linking task: Yes Did the run include Sentiment relations: Yes --------------------- Composite KB Evaluation (All slot types): Scores based on confidence values and Average Precision (AP); k=3 justifications allowed: Metric RunID Hop AP SF-ALL-Macro TinkerBell_KB_SPA_4 0 0.1018 SF-ALL-Macro TinkerBell_KB_SPA_4 1 0.0122 SF-ALL-Macro TinkerBell_KB_SPA_4 ALL 0.0705 LDC-MEAN-ALL-Macro TinkerBell_KB_SPA_4 0 0.1104 LDC-MEAN-ALL-Macro TinkerBell_KB_SPA_4 1 0.0106 LDC-MEAN-ALL-Macro TinkerBell_KB_SPA_4 ALL 0.0755 (PRIMARY METRIC) *ALL-Macro AP refer to mean of corresponding AP values. Scores based on confidence values and Average Precision (AP); only k=1 justification allowed: Metric RunID Hop AP SF-ALL-Macro TinkerBell_KB_SPA_4 0 0.1271 SF-ALL-Macro TinkerBell_KB_SPA_4 1 0.0147 SF-ALL-Macro TinkerBell_KB_SPA_4 ALL 0.0892 LDC-MEAN-ALL-Macro TinkerBell_KB_SPA_4 0 0.1409 LDC-MEAN-ALL-Macro TinkerBell_KB_SPA_4 1 0.0128 LDC-MEAN-ALL-Macro TinkerBell_KB_SPA_4 ALL 0.0975 *ALL-Macro AP refer to mean of corresponding AP values. Scores based on P/R/F1; only k=1 justification allowed: Metric RunID Hop Prec Recall F1 SF-ALL-Micro TinkerBell_KB_SPA_4 0 0.1086 0.2500 0.1514 SF-ALL-Micro TinkerBell_KB_SPA_4 1 0.0091 0.0493 0.0154 SF-ALL-Micro TinkerBell_KB_SPA_4 ALL 0.0537 0.1812 0.0829 SF-ALL-Macro TinkerBell_KB_SPA_4 0 0.1311 0.1721 0.1269 SF-ALL-Macro TinkerBell_KB_SPA_4 1 0.0553 0.0553 0.0536 SF-ALL-Macro TinkerBell_KB_SPA_4 ALL 0.1028 0.1285 0.0995 LDC-MAX-ALL-Micro TinkerBell_KB_SPA_4 0 0.1051 0.2877 0.1540 LDC-MAX-ALL-Micro TinkerBell_KB_SPA_4 1 0.0052 0.0424 0.0092 LDC-MAX-ALL-Micro TinkerBell_KB_SPA_4 ALL 0.0423 0.1991 0.0698 LDC-MAX-ALL-Macro TinkerBell_KB_SPA_4 0 0.1560 0.2003 0.1535 LDC-MAX-ALL-Macro TinkerBell_KB_SPA_4 1 0.0495 0.0495 0.0480 LDC-MAX-ALL-Macro TinkerBell_KB_SPA_4 ALL 0.1171 0.1453 0.1150 LDC-MEAN-ALL-Macro TinkerBell_KB_SPA_4 0 0.1457 0.1825 0.1402 LDC-MEAN-ALL-Macro TinkerBell_KB_SPA_4 1 0.0495 0.0495 0.0480 LDC-MEAN-ALL-Macro TinkerBell_KB_SPA_4 ALL 0.1106 0.1340 0.1066 *ALL-Macro Prec, Recall and F1 refer to mean-precision, mean-recall and mean-F1. --------------------- Composite KB Evaluation (SF-SLOTS only): Scores based on confidence values and Average Precision (AP); k=3 justifications allowed: Metric RunID Hop AP SF-ALL-Macro TinkerBell_KB_SPA_4 0 0.1714 SF-ALL-Macro TinkerBell_KB_SPA_4 1 0.0257 SF-ALL-Macro TinkerBell_KB_SPA_4 ALL 0.1155 LDC-MEAN-ALL-Macro TinkerBell_KB_SPA_4 0 0.1797 LDC-MEAN-ALL-Macro TinkerBell_KB_SPA_4 1 0.0217 LDC-MEAN-ALL-Macro TinkerBell_KB_SPA_4 ALL 0.1190 (PRIMARY METRIC) *ALL-Macro AP refer to mean of corresponding AP values. Scores based on confidence values and Average Precision (AP); only k=1 justification allowed: Metric RunID Hop AP SF-ALL-Macro TinkerBell_KB_SPA_4 0 0.2167 SF-ALL-Macro TinkerBell_KB_SPA_4 1 0.0309 SF-ALL-Macro TinkerBell_KB_SPA_4 ALL 0.1488 LDC-MEAN-ALL-Macro TinkerBell_KB_SPA_4 0 0.2327 LDC-MEAN-ALL-Macro TinkerBell_KB_SPA_4 1 0.0261 LDC-MEAN-ALL-Macro TinkerBell_KB_SPA_4 ALL 0.1573 *ALL-Macro AP refer to mean of corresponding AP values. Scores based on P/R/F1; only k=1 justification allowed: Metric RunID Hop Prec Recall F1 SF-ALL-Micro TinkerBell_KB_SPA_4 0 0.2215 0.2682 0.2426 SF-ALL-Micro TinkerBell_KB_SPA_4 1 0.3684 0.0903 0.1451 SF-ALL-Micro TinkerBell_KB_SPA_4 ALL 0.2373 0.2019 0.2182 SF-ALL-Macro TinkerBell_KB_SPA_4 0 0.2209 0.2312 0.2123 SF-ALL-Macro TinkerBell_KB_SPA_4 1 0.1028 0.1028 0.0997 SF-ALL-Macro TinkerBell_KB_SPA_4 ALL 0.1775 0.1840 0.1709 LDC-MAX-ALL-Micro TinkerBell_KB_SPA_4 0 0.1840 0.3067 0.2300 LDC-MAX-ALL-Micro TinkerBell_KB_SPA_4 1 0.2800 0.0753 0.1186 LDC-MAX-ALL-Micro TinkerBell_KB_SPA_4 ALL 0.1927 0.2181 0.2046 LDC-MAX-ALL-Macro TinkerBell_KB_SPA_4 0 0.2552 0.2692 0.2497 LDC-MAX-ALL-Macro TinkerBell_KB_SPA_4 1 0.0917 0.0917 0.0889 LDC-MAX-ALL-Macro TinkerBell_KB_SPA_4 ALL 0.1975 0.2065 0.1929 LDC-MEAN-ALL-Macro TinkerBell_KB_SPA_4 0 0.2370 0.2449 0.2283 LDC-MEAN-ALL-Macro TinkerBell_KB_SPA_4 1 0.0917 0.0917 0.0889 LDC-MEAN-ALL-Macro TinkerBell_KB_SPA_4 ALL 0.1857 0.1908 0.1791 *ALL-Macro Prec, Recall and F1 refer to mean-precision, mean-recall and mean-F1. --------------------- Composite KB Evaluation (EVENT-SLOTS only): Scores based on confidence values and Average Precision (AP); k=3 justifications allowed: SUMMARY: This section provides summary of AP scores Metric RunID Hop AP SF-ALL-Macro TinkerBell_KB_SPA_4 0 0.0028 SF-ALL-Macro TinkerBell_KB_SPA_4 ALL 0.0028 LDC-MEAN-ALL-Macro TinkerBell_KB_SPA_4 0 0.0051 LDC-MEAN-ALL-Macro TinkerBell_KB_SPA_4 ALL 0.0051 (PRIMARY METRIC) *ALL-Macro AP refer to mean of corresponding AP values. Scores based on confidence values and Average Precision (AP); only k=1 justification allowed: SUMMARY: This section provides summary of AP scores Metric RunID Hop AP SF-ALL-Macro TinkerBell_KB_SPA_4 0 0.0028 SF-ALL-Macro TinkerBell_KB_SPA_4 ALL 0.0028 LDC-MEAN-ALL-Macro TinkerBell_KB_SPA_4 0 0.0051 LDC-MEAN-ALL-Macro TinkerBell_KB_SPA_4 ALL 0.0051 *ALL-Macro AP refer to mean of corresponding AP values. Scores based on P/R/F1; only k=1 justification allowed: Metric RunID Hop Prec Recall F1 SF-ALL-Micro TinkerBell_KB_SPA_4 0 0.2500 0.0086 0.0167 SF-ALL-Micro TinkerBell_KB_SPA_4 ALL 0.2500 0.0086 0.0167 SF-ALL-Macro TinkerBell_KB_SPA_4 0 0.0114 0.0028 0.0045 SF-ALL-Macro TinkerBell_KB_SPA_4 ALL 0.0114 0.0028 0.0045 LDC-MAX-ALL-Micro TinkerBell_KB_SPA_4 0 0.3333 0.0152 0.0290 LDC-MAX-ALL-Micro TinkerBell_KB_SPA_4 ALL 0.3333 0.0152 0.0290 LDC-MAX-ALL-Macro TinkerBell_KB_SPA_4 0 0.0204 0.0051 0.0082 LDC-MAX-ALL-Macro TinkerBell_KB_SPA_4 ALL 0.0204 0.0051 0.0082 LDC-MEAN-ALL-Macro TinkerBell_KB_SPA_4 0 0.0204 0.0051 0.0082 LDC-MEAN-ALL-Macro TinkerBell_KB_SPA_4 ALL 0.0204 0.0051 0.0082 *ALL-Macro Prec, Recall and F1 refer to mean-precision, mean-recall and mean-F1. --------------------- Composite KB Evaluation (SENTIMENT-SLOTS only): Scores based on confidence values and Average Precision (AP); k=3 justifications allowed: Metric RunID Hop AP SF-ALL-Macro TinkerBell_KB_SPA_4 0 0.0358 SF-ALL-Macro TinkerBell_KB_SPA_4 1 0.0000 SF-ALL-Macro TinkerBell_KB_SPA_4 ALL 0.0333 LDC-MEAN-ALL-Macro TinkerBell_KB_SPA_4 0 0.0380 LDC-MEAN-ALL-Macro TinkerBell_KB_SPA_4 1 0.0000 LDC-MEAN-ALL-Macro TinkerBell_KB_SPA_4 ALL 0.0364 (PRIMARY METRIC) *ALL-Macro AP refer to mean of corresponding AP values. Scores based on confidence values and Average Precision (AP); only k=1 justification allowed: Metric RunID Hop AP SF-ALL-Macro TinkerBell_KB_SPA_4 0 0.0375 SF-ALL-Macro TinkerBell_KB_SPA_4 1 0.0000 SF-ALL-Macro TinkerBell_KB_SPA_4 ALL 0.0347 LDC-MEAN-ALL-Macro TinkerBell_KB_SPA_4 0 0.0394 LDC-MEAN-ALL-Macro TinkerBell_KB_SPA_4 1 0.0000 LDC-MEAN-ALL-Macro TinkerBell_KB_SPA_4 ALL 0.0376 *ALL-Macro AP refer to mean of corresponding AP values. Scores based on P/R/F1; only k=1 justification allowed: Metric RunID Hop Prec Recall F1 SF-ALL-Micro TinkerBell_KB_SPA_4 0 0.0697 0.3892 0.1183 SF-ALL-Micro TinkerBell_KB_SPA_4 1 0.0000 0.0000 0.0000 SF-ALL-Micro TinkerBell_KB_SPA_4 ALL 0.0643 0.3351 0.1079 SF-ALL-Macro TinkerBell_KB_SPA_4 0 0.0343 0.2367 0.0473 SF-ALL-Macro TinkerBell_KB_SPA_4 1 0.0000 0.0000 0.0000 SF-ALL-Macro TinkerBell_KB_SPA_4 ALL 0.0245 0.1687 0.0337 LDC-MAX-ALL-Micro TinkerBell_KB_SPA_4 0 0.0676 0.4868 0.1188 LDC-MAX-ALL-Micro TinkerBell_KB_SPA_4 1 0.0000 0.0000 0.0000 LDC-MAX-ALL-Micro TinkerBell_KB_SPA_4 ALL 0.0631 0.4066 0.1093 LDC-MAX-ALL-Macro TinkerBell_KB_SPA_4 0 0.0304 0.2589 0.0519 LDC-MAX-ALL-Macro TinkerBell_KB_SPA_4 1 0.0000 0.0000 0.0000 LDC-MAX-ALL-Macro TinkerBell_KB_SPA_4 ALL 0.0220 0.1873 0.0376 LDC-MEAN-ALL-Macro TinkerBell_KB_SPA_4 0 0.0309 0.2365 0.0458 LDC-MEAN-ALL-Macro TinkerBell_KB_SPA_4 1 0.0000 0.0000 0.0000 LDC-MEAN-ALL-Macro TinkerBell_KB_SPA_4 ALL 0.0224 0.1711 0.0331 *ALL-Macro Prec, Recall and F1 refer to mean-precision, mean-recall and mean-F1. ************************************************************* Run ID: TinkerBell_KB_SPA_5 Did the run access the live Web during the evaluation window: No Did the run extract relations from the Cold Start source corpus: Yes Did the run generate meaningful confidence values: Yes Did the run attempt the full Entity Discovery and Linking task: Yes Did the run attempt the full Event Nugget Detection and Coreference task: Yes Did the run include SF relations: Yes Did the run attempt the Event Argument Extraction and Linking task: Yes Did the run include Sentiment relations: Yes --------------------- Composite KB Evaluation (All slot types): Scores based on confidence values and Average Precision (AP); k=3 justifications allowed: Metric RunID Hop AP SF-ALL-Macro TinkerBell_KB_SPA_5 0 0.1018 SF-ALL-Macro TinkerBell_KB_SPA_5 1 0.0122 SF-ALL-Macro TinkerBell_KB_SPA_5 ALL 0.0705 LDC-MEAN-ALL-Macro TinkerBell_KB_SPA_5 0 0.1104 LDC-MEAN-ALL-Macro TinkerBell_KB_SPA_5 1 0.0106 LDC-MEAN-ALL-Macro TinkerBell_KB_SPA_5 ALL 0.0755 (PRIMARY METRIC) *ALL-Macro AP refer to mean of corresponding AP values. Scores based on confidence values and Average Precision (AP); only k=1 justification allowed: Metric RunID Hop AP SF-ALL-Macro TinkerBell_KB_SPA_5 0 0.1271 SF-ALL-Macro TinkerBell_KB_SPA_5 1 0.0147 SF-ALL-Macro TinkerBell_KB_SPA_5 ALL 0.0892 LDC-MEAN-ALL-Macro TinkerBell_KB_SPA_5 0 0.1409 LDC-MEAN-ALL-Macro TinkerBell_KB_SPA_5 1 0.0128 LDC-MEAN-ALL-Macro TinkerBell_KB_SPA_5 ALL 0.0975 *ALL-Macro AP refer to mean of corresponding AP values. Scores based on P/R/F1; only k=1 justification allowed: Metric RunID Hop Prec Recall F1 SF-ALL-Micro TinkerBell_KB_SPA_5 0 0.1088 0.2500 0.1516 SF-ALL-Micro TinkerBell_KB_SPA_5 1 0.0091 0.0493 0.0154 SF-ALL-Micro TinkerBell_KB_SPA_5 ALL 0.0538 0.1812 0.0829 SF-ALL-Macro TinkerBell_KB_SPA_5 0 0.1311 0.1721 0.1269 SF-ALL-Macro TinkerBell_KB_SPA_5 1 0.0553 0.0553 0.0536 SF-ALL-Macro TinkerBell_KB_SPA_5 ALL 0.1028 0.1285 0.0995 LDC-MAX-ALL-Micro TinkerBell_KB_SPA_5 0 0.1051 0.2877 0.1540 LDC-MAX-ALL-Micro TinkerBell_KB_SPA_5 1 0.0052 0.0424 0.0092 LDC-MAX-ALL-Micro TinkerBell_KB_SPA_5 ALL 0.0423 0.1991 0.0698 LDC-MAX-ALL-Macro TinkerBell_KB_SPA_5 0 0.1560 0.2003 0.1535 LDC-MAX-ALL-Macro TinkerBell_KB_SPA_5 1 0.0495 0.0495 0.0480 LDC-MAX-ALL-Macro TinkerBell_KB_SPA_5 ALL 0.1171 0.1453 0.1150 LDC-MEAN-ALL-Macro TinkerBell_KB_SPA_5 0 0.1457 0.1825 0.1403 LDC-MEAN-ALL-Macro TinkerBell_KB_SPA_5 1 0.0495 0.0495 0.0480 LDC-MEAN-ALL-Macro TinkerBell_KB_SPA_5 ALL 0.1106 0.1340 0.1066 *ALL-Macro Prec, Recall and F1 refer to mean-precision, mean-recall and mean-F1. --------------------- Composite KB Evaluation (SF-SLOTS only): Scores based on confidence values and Average Precision (AP); k=3 justifications allowed: Metric RunID Hop AP SF-ALL-Macro TinkerBell_KB_SPA_5 0 0.1714 SF-ALL-Macro TinkerBell_KB_SPA_5 1 0.0257 SF-ALL-Macro TinkerBell_KB_SPA_5 ALL 0.1155 LDC-MEAN-ALL-Macro TinkerBell_KB_SPA_5 0 0.1797 LDC-MEAN-ALL-Macro TinkerBell_KB_SPA_5 1 0.0217 LDC-MEAN-ALL-Macro TinkerBell_KB_SPA_5 ALL 0.1190 (PRIMARY METRIC) *ALL-Macro AP refer to mean of corresponding AP values. Scores based on confidence values and Average Precision (AP); only k=1 justification allowed: Metric RunID Hop AP SF-ALL-Macro TinkerBell_KB_SPA_5 0 0.2167 SF-ALL-Macro TinkerBell_KB_SPA_5 1 0.0309 SF-ALL-Macro TinkerBell_KB_SPA_5 ALL 0.1488 LDC-MEAN-ALL-Macro TinkerBell_KB_SPA_5 0 0.2327 LDC-MEAN-ALL-Macro TinkerBell_KB_SPA_5 1 0.0261 LDC-MEAN-ALL-Macro TinkerBell_KB_SPA_5 ALL 0.1573 *ALL-Macro AP refer to mean of corresponding AP values. Scores based on P/R/F1; only k=1 justification allowed: Metric RunID Hop Prec Recall F1 SF-ALL-Micro TinkerBell_KB_SPA_5 0 0.2215 0.2682 0.2426 SF-ALL-Micro TinkerBell_KB_SPA_5 1 0.3684 0.0903 0.1451 SF-ALL-Micro TinkerBell_KB_SPA_5 ALL 0.2373 0.2019 0.2182 SF-ALL-Macro TinkerBell_KB_SPA_5 0 0.2209 0.2312 0.2123 SF-ALL-Macro TinkerBell_KB_SPA_5 1 0.1028 0.1028 0.0997 SF-ALL-Macro TinkerBell_KB_SPA_5 ALL 0.1775 0.1840 0.1709 LDC-MAX-ALL-Micro TinkerBell_KB_SPA_5 0 0.1847 0.3067 0.2306 LDC-MAX-ALL-Micro TinkerBell_KB_SPA_5 1 0.2800 0.0753 0.1186 LDC-MAX-ALL-Micro TinkerBell_KB_SPA_5 ALL 0.1934 0.2181 0.2050 LDC-MAX-ALL-Macro TinkerBell_KB_SPA_5 0 0.2552 0.2692 0.2497 LDC-MAX-ALL-Macro TinkerBell_KB_SPA_5 1 0.0917 0.0917 0.0889 LDC-MAX-ALL-Macro TinkerBell_KB_SPA_5 ALL 0.1975 0.2065 0.1929 LDC-MEAN-ALL-Macro TinkerBell_KB_SPA_5 0 0.2370 0.2449 0.2283 LDC-MEAN-ALL-Macro TinkerBell_KB_SPA_5 1 0.0917 0.0917 0.0889 LDC-MEAN-ALL-Macro TinkerBell_KB_SPA_5 ALL 0.1857 0.1908 0.1791 *ALL-Macro Prec, Recall and F1 refer to mean-precision, mean-recall and mean-F1. --------------------- Composite KB Evaluation (EVENT-SLOTS only): Scores based on confidence values and Average Precision (AP); k=3 justifications allowed: SUMMARY: This section provides summary of AP scores Metric RunID Hop AP SF-ALL-Macro TinkerBell_KB_SPA_5 0 0.0028 SF-ALL-Macro TinkerBell_KB_SPA_5 ALL 0.0028 LDC-MEAN-ALL-Macro TinkerBell_KB_SPA_5 0 0.0051 LDC-MEAN-ALL-Macro TinkerBell_KB_SPA_5 ALL 0.0051 (PRIMARY METRIC) *ALL-Macro AP refer to mean of corresponding AP values. Scores based on confidence values and Average Precision (AP); only k=1 justification allowed: SUMMARY: This section provides summary of AP scores Metric RunID Hop AP SF-ALL-Macro TinkerBell_KB_SPA_5 0 0.0028 SF-ALL-Macro TinkerBell_KB_SPA_5 ALL 0.0028 LDC-MEAN-ALL-Macro TinkerBell_KB_SPA_5 0 0.0051 LDC-MEAN-ALL-Macro TinkerBell_KB_SPA_5 ALL 0.0051 *ALL-Macro AP refer to mean of corresponding AP values. Scores based on P/R/F1; only k=1 justification allowed: Metric RunID Hop Prec Recall F1 SF-ALL-Micro TinkerBell_KB_SPA_5 0 0.2500 0.0086 0.0167 SF-ALL-Micro TinkerBell_KB_SPA_5 ALL 0.2500 0.0086 0.0167 SF-ALL-Macro TinkerBell_KB_SPA_5 0 0.0114 0.0028 0.0045 SF-ALL-Macro TinkerBell_KB_SPA_5 ALL 0.0114 0.0028 0.0045 LDC-MAX-ALL-Micro TinkerBell_KB_SPA_5 0 0.3333 0.0152 0.0290 LDC-MAX-ALL-Micro TinkerBell_KB_SPA_5 ALL 0.3333 0.0152 0.0290 LDC-MAX-ALL-Macro TinkerBell_KB_SPA_5 0 0.0204 0.0051 0.0082 LDC-MAX-ALL-Macro TinkerBell_KB_SPA_5 ALL 0.0204 0.0051 0.0082 LDC-MEAN-ALL-Macro TinkerBell_KB_SPA_5 0 0.0204 0.0051 0.0082 LDC-MEAN-ALL-Macro TinkerBell_KB_SPA_5 ALL 0.0204 0.0051 0.0082 *ALL-Macro Prec, Recall and F1 refer to mean-precision, mean-recall and mean-F1. --------------------- Composite KB Evaluation (SENTIMENT-SLOTS only): Scores based on confidence values and Average Precision (AP); k=3 justifications allowed: Metric RunID Hop AP SF-ALL-Macro TinkerBell_KB_SPA_5 0 0.0358 SF-ALL-Macro TinkerBell_KB_SPA_5 1 0.0000 SF-ALL-Macro TinkerBell_KB_SPA_5 ALL 0.0333 LDC-MEAN-ALL-Macro TinkerBell_KB_SPA_5 0 0.0380 LDC-MEAN-ALL-Macro TinkerBell_KB_SPA_5 1 0.0000 LDC-MEAN-ALL-Macro TinkerBell_KB_SPA_5 ALL 0.0364 (PRIMARY METRIC) *ALL-Macro AP refer to mean of corresponding AP values. Scores based on confidence values and Average Precision (AP); only k=1 justification allowed: Metric RunID Hop AP SF-ALL-Macro TinkerBell_KB_SPA_5 0 0.0375 SF-ALL-Macro TinkerBell_KB_SPA_5 1 0.0000 SF-ALL-Macro TinkerBell_KB_SPA_5 ALL 0.0347 LDC-MEAN-ALL-Macro TinkerBell_KB_SPA_5 0 0.0394 LDC-MEAN-ALL-Macro TinkerBell_KB_SPA_5 1 0.0000 LDC-MEAN-ALL-Macro TinkerBell_KB_SPA_5 ALL 0.0376 *ALL-Macro AP refer to mean of corresponding AP values. Scores based on P/R/F1; only k=1 justification allowed: Metric RunID Hop Prec Recall F1 SF-ALL-Micro TinkerBell_KB_SPA_5 0 0.0699 0.3892 0.1185 SF-ALL-Micro TinkerBell_KB_SPA_5 1 0.0000 0.0000 0.0000 SF-ALL-Micro TinkerBell_KB_SPA_5 ALL 0.0644 0.3351 0.1081 SF-ALL-Macro TinkerBell_KB_SPA_5 0 0.0343 0.2367 0.0473 SF-ALL-Macro TinkerBell_KB_SPA_5 1 0.0000 0.0000 0.0000 SF-ALL-Macro TinkerBell_KB_SPA_5 ALL 0.0245 0.1687 0.0337 LDC-MAX-ALL-Micro TinkerBell_KB_SPA_5 0 0.0678 0.4868 0.1190 LDC-MAX-ALL-Micro TinkerBell_KB_SPA_5 1 0.0000 0.0000 0.0000 LDC-MAX-ALL-Micro TinkerBell_KB_SPA_5 ALL 0.0632 0.4066 0.1095 LDC-MAX-ALL-Macro TinkerBell_KB_SPA_5 0 0.0304 0.2589 0.0519 LDC-MAX-ALL-Macro TinkerBell_KB_SPA_5 1 0.0000 0.0000 0.0000 LDC-MAX-ALL-Macro TinkerBell_KB_SPA_5 ALL 0.0220 0.1873 0.0376 LDC-MEAN-ALL-Macro TinkerBell_KB_SPA_5 0 0.0309 0.2365 0.0458 LDC-MEAN-ALL-Macro TinkerBell_KB_SPA_5 1 0.0000 0.0000 0.0000 LDC-MEAN-ALL-Macro TinkerBell_KB_SPA_5 ALL 0.0224 0.1711 0.0331 *ALL-Macro Prec, Recall and F1 refer to mean-precision, mean-recall and mean-F1.