===================================================================== TAC KBP 2017 ENGLISH KB CONSTRUCTION: COMPOSITE KB EVALUATION RESULTS ===================================================================== Team ID: TinkerBell Organization: RPI, UIUC, Stanford, Columbia, Cornell, JHU, UPenn ************************************************************* Run ID: TinkerBell_KB_ENG_1 Did the run access the live Web during the evaluation window: No Did the run extract relations from the Cold Start source corpus: Yes Did the run generate meaningful confidence values: Yes Did the run attempt the full Entity Discovery and Linking task: Yes Did the run attempt the full Event Nugget Detection and Coreference task: Yes Did the run include SF relations: Yes Did the run attempt the Event Argument Extraction and Linking task: Yes Did the run include Sentiment relations: Yes --------------------- Composite KB Evaluation (All slot types): Scores based on confidence values and Average Precision (AP); k=3 justifications allowed: Metric RunID Hop AP SF-ALL-Macro TinkerBell_KB_ENG_1 0 0.1091 SF-ALL-Macro TinkerBell_KB_ENG_1 1 0.0241 SF-ALL-Macro TinkerBell_KB_ENG_1 ALL 0.0850 LDC-MEAN-ALL-Macro TinkerBell_KB_ENG_1 0 0.1262 LDC-MEAN-ALL-Macro TinkerBell_KB_ENG_1 1 0.0334 LDC-MEAN-ALL-Macro TinkerBell_KB_ENG_1 ALL 0.0997 (PRIMARY METRIC) *ALL-Macro AP refer to mean of corresponding AP values. Scores based on confidence values and Average Precision (AP); only k=1 justification allowed: Metric RunID Hop AP SF-ALL-Macro TinkerBell_KB_ENG_1 0 0.1300 SF-ALL-Macro TinkerBell_KB_ENG_1 1 0.0361 SF-ALL-Macro TinkerBell_KB_ENG_1 ALL 0.1057 LDC-MEAN-ALL-Macro TinkerBell_KB_ENG_1 0 0.1515 LDC-MEAN-ALL-Macro TinkerBell_KB_ENG_1 1 0.0452 LDC-MEAN-ALL-Macro TinkerBell_KB_ENG_1 ALL 0.1251 *ALL-Macro AP refer to mean of corresponding AP values. Scores based on P/R/F1; only k=1 justification allowed: Metric RunID Hop Prec Recall F1 SF-ALL-Micro TinkerBell_KB_ENG_1 0 0.1314 0.1887 0.1549 SF-ALL-Micro TinkerBell_KB_ENG_1 1 0.0076 0.1599 0.0145 SF-ALL-Micro TinkerBell_KB_ENG_1 ALL 0.0309 0.1821 0.0528 SF-ALL-Macro TinkerBell_KB_ENG_1 0 0.1016 0.1599 0.1042 SF-ALL-Macro TinkerBell_KB_ENG_1 1 0.1182 0.1546 0.1225 SF-ALL-Macro TinkerBell_KB_ENG_1 ALL 0.1073 0.1581 0.1105 LDC-MAX-ALL-Micro TinkerBell_KB_ENG_1 0 0.1524 0.2012 0.1734 LDC-MAX-ALL-Micro TinkerBell_KB_ENG_1 1 0.0083 0.1691 0.0159 LDC-MAX-ALL-Micro TinkerBell_KB_ENG_1 ALL 0.0358 0.1943 0.0605 LDC-MAX-ALL-Macro TinkerBell_KB_ENG_1 0 0.1254 0.1955 0.1282 LDC-MAX-ALL-Macro TinkerBell_KB_ENG_1 1 0.1194 0.1698 0.1275 LDC-MAX-ALL-Macro TinkerBell_KB_ENG_1 ALL 0.1233 0.1867 0.1279 LDC-MEAN-ALL-Macro TinkerBell_KB_ENG_1 0 0.1201 0.1845 0.1221 LDC-MEAN-ALL-Macro TinkerBell_KB_ENG_1 1 0.1233 0.1712 0.1300 LDC-MEAN-ALL-Macro TinkerBell_KB_ENG_1 ALL 0.1212 0.1799 0.1248 *ALL-Macro Prec, Recall and F1 refer to mean-precision, mean-recall and mean-F1. --------------------- Composite KB Evaluation (SF-SLOTS only): Scores based on confidence values and Average Precision (AP); k=3 justifications allowed: Metric RunID Hop AP SF-ALL-Macro TinkerBell_KB_ENG_1 0 0.2235 SF-ALL-Macro TinkerBell_KB_ENG_1 1 0.0657 SF-ALL-Macro TinkerBell_KB_ENG_1 ALL 0.1738 LDC-MEAN-ALL-Macro TinkerBell_KB_ENG_1 0 0.2448 LDC-MEAN-ALL-Macro TinkerBell_KB_ENG_1 1 0.0858 LDC-MEAN-ALL-Macro TinkerBell_KB_ENG_1 ALL 0.1942 (PRIMARY METRIC) *ALL-Macro AP refer to mean of corresponding AP values. Scores based on confidence values and Average Precision (AP); only k=1 justification allowed: Metric RunID Hop AP SF-ALL-Macro TinkerBell_KB_ENG_1 0 0.2748 SF-ALL-Macro TinkerBell_KB_ENG_1 1 0.0990 SF-ALL-Macro TinkerBell_KB_ENG_1 ALL 0.2226 LDC-MEAN-ALL-Macro TinkerBell_KB_ENG_1 0 0.3042 LDC-MEAN-ALL-Macro TinkerBell_KB_ENG_1 1 0.1166 LDC-MEAN-ALL-Macro TinkerBell_KB_ENG_1 ALL 0.2511 *ALL-Macro AP refer to mean of corresponding AP values. Scores based on P/R/F1; only k=1 justification allowed: Metric RunID Hop Prec Recall F1 SF-ALL-Micro TinkerBell_KB_ENG_1 0 0.1742 0.2962 0.2194 SF-ALL-Micro TinkerBell_KB_ENG_1 1 0.0325 0.2568 0.0577 SF-ALL-Micro TinkerBell_KB_ENG_1 ALL 0.0899 0.2867 0.1368 SF-ALL-Macro TinkerBell_KB_ENG_1 0 0.1977 0.3138 0.2168 SF-ALL-Macro TinkerBell_KB_ENG_1 1 0.2024 0.2506 0.2068 SF-ALL-Macro TinkerBell_KB_ENG_1 ALL 0.1996 0.2888 0.2129 LDC-MAX-ALL-Micro TinkerBell_KB_ENG_1 0 0.2095 0.3111 0.2504 LDC-MAX-ALL-Micro TinkerBell_KB_ENG_1 1 0.0366 0.2468 0.0637 LDC-MAX-ALL-Micro TinkerBell_KB_ENG_1 ALL 0.1095 0.2962 0.1599 LDC-MAX-ALL-Macro TinkerBell_KB_ENG_1 0 0.2304 0.3663 0.2532 LDC-MAX-ALL-Macro TinkerBell_KB_ENG_1 1 0.1914 0.2571 0.2015 LDC-MAX-ALL-Macro TinkerBell_KB_ENG_1 ALL 0.2153 0.3239 0.2331 LDC-MEAN-ALL-Macro TinkerBell_KB_ENG_1 0 0.2205 0.3473 0.2422 LDC-MEAN-ALL-Macro TinkerBell_KB_ENG_1 1 0.1984 0.2596 0.2060 LDC-MEAN-ALL-Macro TinkerBell_KB_ENG_1 ALL 0.2119 0.3132 0.2281 *ALL-Macro Prec, Recall and F1 refer to mean-precision, mean-recall and mean-F1. --------------------- Composite KB Evaluation (EVENT-SLOTS only): Scores based on confidence values and Average Precision (AP); k=3 justifications allowed: SUMMARY: This section provides summary of AP scores Metric RunID Hop AP SF-ALL-Macro TinkerBell_KB_ENG_1 0 0.0022 SF-ALL-Macro TinkerBell_KB_ENG_1 ALL 0.0022 LDC-MEAN-ALL-Macro TinkerBell_KB_ENG_1 0 0.0037 LDC-MEAN-ALL-Macro TinkerBell_KB_ENG_1 ALL 0.0037 (PRIMARY METRIC) *ALL-Macro AP refer to mean of corresponding AP values. Scores based on confidence values and Average Precision (AP); only k=1 justification allowed: SUMMARY: This section provides summary of AP scores Metric RunID Hop AP SF-ALL-Macro TinkerBell_KB_ENG_1 0 0.0024 SF-ALL-Macro TinkerBell_KB_ENG_1 ALL 0.0024 LDC-MEAN-ALL-Macro TinkerBell_KB_ENG_1 0 0.0040 LDC-MEAN-ALL-Macro TinkerBell_KB_ENG_1 ALL 0.0040 *ALL-Macro AP refer to mean of corresponding AP values. Scores based on P/R/F1; only k=1 justification allowed: Metric RunID Hop Prec Recall F1 SF-ALL-Micro TinkerBell_KB_ENG_1 0 0.0673 0.0081 0.0145 SF-ALL-Micro TinkerBell_KB_ENG_1 ALL 0.0673 0.0081 0.0145 SF-ALL-Macro TinkerBell_KB_ENG_1 0 0.0179 0.0034 0.0044 SF-ALL-Macro TinkerBell_KB_ENG_1 ALL 0.0179 0.0034 0.0044 LDC-MAX-ALL-Micro TinkerBell_KB_ENG_1 0 0.1316 0.0134 0.0243 LDC-MAX-ALL-Micro TinkerBell_KB_ENG_1 ALL 0.1316 0.0134 0.0243 LDC-MAX-ALL-Macro TinkerBell_KB_ENG_1 0 0.0326 0.0074 0.0085 LDC-MAX-ALL-Macro TinkerBell_KB_ENG_1 ALL 0.0326 0.0074 0.0085 LDC-MEAN-ALL-Macro TinkerBell_KB_ENG_1 0 0.0318 0.0055 0.0074 LDC-MEAN-ALL-Macro TinkerBell_KB_ENG_1 ALL 0.0318 0.0055 0.0074 *ALL-Macro Prec, Recall and F1 refer to mean-precision, mean-recall and mean-F1. --------------------- Composite KB Evaluation (SENTIMENT-SLOTS only): Scores based on confidence values and Average Precision (AP); k=3 justifications allowed: Metric RunID Hop AP SF-ALL-Macro TinkerBell_KB_ENG_1 0 0.0601 SF-ALL-Macro TinkerBell_KB_ENG_1 1 0.0033 SF-ALL-Macro TinkerBell_KB_ENG_1 ALL 0.0466 LDC-MEAN-ALL-Macro TinkerBell_KB_ENG_1 0 0.0637 LDC-MEAN-ALL-Macro TinkerBell_KB_ENG_1 1 0.0045 LDC-MEAN-ALL-Macro TinkerBell_KB_ENG_1 ALL 0.0469 (PRIMARY METRIC) *ALL-Macro AP refer to mean of corresponding AP values. Scores based on confidence values and Average Precision (AP); only k=1 justification allowed: Metric RunID Hop AP SF-ALL-Macro TinkerBell_KB_ENG_1 0 0.0529 SF-ALL-Macro TinkerBell_KB_ENG_1 1 0.0038 SF-ALL-Macro TinkerBell_KB_ENG_1 ALL 0.0439 LDC-MEAN-ALL-Macro TinkerBell_KB_ENG_1 0 0.0506 LDC-MEAN-ALL-Macro TinkerBell_KB_ENG_1 1 0.0052 LDC-MEAN-ALL-Macro TinkerBell_KB_ENG_1 ALL 0.0401 *ALL-Macro AP refer to mean of corresponding AP values. Scores based on P/R/F1; only k=1 justification allowed: Metric RunID Hop Prec Recall F1 SF-ALL-Micro TinkerBell_KB_ENG_1 0 0.0709 0.2862 0.1137 SF-ALL-Micro TinkerBell_KB_ENG_1 1 0.0023 0.1500 0.0045 SF-ALL-Micro TinkerBell_KB_ENG_1 ALL 0.0157 0.2588 0.0295 SF-ALL-Macro TinkerBell_KB_ENG_1 0 0.0485 0.1194 0.0453 SF-ALL-Macro TinkerBell_KB_ENG_1 1 0.0555 0.1096 0.0674 SF-ALL-Macro TinkerBell_KB_ENG_1 ALL 0.0506 0.1165 0.0518 LDC-MAX-ALL-Micro TinkerBell_KB_ENG_1 0 0.0677 0.3125 0.1113 LDC-MAX-ALL-Micro TinkerBell_KB_ENG_1 1 0.0033 0.2000 0.0064 LDC-MAX-ALL-Micro TinkerBell_KB_ENG_1 ALL 0.0174 0.2887 0.0327 LDC-MAX-ALL-Macro TinkerBell_KB_ENG_1 0 0.0384 0.1288 0.0430 LDC-MAX-ALL-Macro TinkerBell_KB_ENG_1 1 0.0750 0.1481 0.0911 LDC-MAX-ALL-Macro TinkerBell_KB_ENG_1 ALL 0.0492 0.1345 0.0573 LDC-MEAN-ALL-Macro TinkerBell_KB_ENG_1 0 0.0362 0.1205 0.0397 LDC-MEAN-ALL-Macro TinkerBell_KB_ENG_1 1 0.0750 0.1481 0.0911 LDC-MEAN-ALL-Macro TinkerBell_KB_ENG_1 ALL 0.0477 0.1287 0.0549 *ALL-Macro Prec, Recall and F1 refer to mean-precision, mean-recall and mean-F1. ************************************************************* Run ID: TinkerBell_KB_ENG_2 Did the run access the live Web during the evaluation window: No Did the run extract relations from the Cold Start source corpus: Yes Did the run generate meaningful confidence values: Yes Did the run attempt the full Entity Discovery and Linking task: Yes Did the run attempt the full Event Nugget Detection and Coreference task: Yes Did the run include SF relations: Yes Did the run attempt the Event Argument Extraction and Linking task: Yes Did the run include Sentiment relations: Yes --------------------- Composite KB Evaluation (All slot types): Scores based on confidence values and Average Precision (AP); k=3 justifications allowed: Metric RunID Hop AP SF-ALL-Macro TinkerBell_KB_ENG_2 0 0.1091 SF-ALL-Macro TinkerBell_KB_ENG_2 1 0.0241 SF-ALL-Macro TinkerBell_KB_ENG_2 ALL 0.0850 LDC-MEAN-ALL-Macro TinkerBell_KB_ENG_2 0 0.1262 LDC-MEAN-ALL-Macro TinkerBell_KB_ENG_2 1 0.0334 LDC-MEAN-ALL-Macro TinkerBell_KB_ENG_2 ALL 0.0997 (PRIMARY METRIC) *ALL-Macro AP refer to mean of corresponding AP values. Scores based on confidence values and Average Precision (AP); only k=1 justification allowed: Metric RunID Hop AP SF-ALL-Macro TinkerBell_KB_ENG_2 0 0.1300 SF-ALL-Macro TinkerBell_KB_ENG_2 1 0.0361 SF-ALL-Macro TinkerBell_KB_ENG_2 ALL 0.1057 LDC-MEAN-ALL-Macro TinkerBell_KB_ENG_2 0 0.1515 LDC-MEAN-ALL-Macro TinkerBell_KB_ENG_2 1 0.0452 LDC-MEAN-ALL-Macro TinkerBell_KB_ENG_2 ALL 0.1251 *ALL-Macro AP refer to mean of corresponding AP values. Scores based on P/R/F1; only k=1 justification allowed: Metric RunID Hop Prec Recall F1 SF-ALL-Micro TinkerBell_KB_ENG_2 0 0.1314 0.1887 0.1549 SF-ALL-Micro TinkerBell_KB_ENG_2 1 0.0076 0.1599 0.0145 SF-ALL-Micro TinkerBell_KB_ENG_2 ALL 0.0309 0.1821 0.0528 SF-ALL-Macro TinkerBell_KB_ENG_2 0 0.1016 0.1599 0.1042 SF-ALL-Macro TinkerBell_KB_ENG_2 1 0.1182 0.1546 0.1225 SF-ALL-Macro TinkerBell_KB_ENG_2 ALL 0.1073 0.1581 0.1105 LDC-MAX-ALL-Micro TinkerBell_KB_ENG_2 0 0.1523 0.2012 0.1734 LDC-MAX-ALL-Micro TinkerBell_KB_ENG_2 1 0.0083 0.1691 0.0158 LDC-MAX-ALL-Micro TinkerBell_KB_ENG_2 ALL 0.0358 0.1943 0.0604 LDC-MAX-ALL-Macro TinkerBell_KB_ENG_2 0 0.1254 0.1955 0.1282 LDC-MAX-ALL-Macro TinkerBell_KB_ENG_2 1 0.1194 0.1698 0.1275 LDC-MAX-ALL-Macro TinkerBell_KB_ENG_2 ALL 0.1233 0.1867 0.1279 LDC-MEAN-ALL-Macro TinkerBell_KB_ENG_2 0 0.1201 0.1845 0.1221 LDC-MEAN-ALL-Macro TinkerBell_KB_ENG_2 1 0.1233 0.1712 0.1300 LDC-MEAN-ALL-Macro TinkerBell_KB_ENG_2 ALL 0.1212 0.1799 0.1248 *ALL-Macro Prec, Recall and F1 refer to mean-precision, mean-recall and mean-F1. --------------------- Composite KB Evaluation (SF-SLOTS only): Scores based on confidence values and Average Precision (AP); k=3 justifications allowed: Metric RunID Hop AP SF-ALL-Macro TinkerBell_KB_ENG_2 0 0.2235 SF-ALL-Macro TinkerBell_KB_ENG_2 1 0.0657 SF-ALL-Macro TinkerBell_KB_ENG_2 ALL 0.1738 LDC-MEAN-ALL-Macro TinkerBell_KB_ENG_2 0 0.2448 LDC-MEAN-ALL-Macro TinkerBell_KB_ENG_2 1 0.0858 LDC-MEAN-ALL-Macro TinkerBell_KB_ENG_2 ALL 0.1942 (PRIMARY METRIC) *ALL-Macro AP refer to mean of corresponding AP values. Scores based on confidence values and Average Precision (AP); only k=1 justification allowed: Metric RunID Hop AP SF-ALL-Macro TinkerBell_KB_ENG_2 0 0.2748 SF-ALL-Macro TinkerBell_KB_ENG_2 1 0.0990 SF-ALL-Macro TinkerBell_KB_ENG_2 ALL 0.2226 LDC-MEAN-ALL-Macro TinkerBell_KB_ENG_2 0 0.3042 LDC-MEAN-ALL-Macro TinkerBell_KB_ENG_2 1 0.1166 LDC-MEAN-ALL-Macro TinkerBell_KB_ENG_2 ALL 0.2511 *ALL-Macro AP refer to mean of corresponding AP values. Scores based on P/R/F1; only k=1 justification allowed: Metric RunID Hop Prec Recall F1 SF-ALL-Micro TinkerBell_KB_ENG_2 0 0.1742 0.2962 0.2194 SF-ALL-Micro TinkerBell_KB_ENG_2 1 0.0325 0.2568 0.0577 SF-ALL-Micro TinkerBell_KB_ENG_2 ALL 0.0899 0.2867 0.1368 SF-ALL-Macro TinkerBell_KB_ENG_2 0 0.1977 0.3138 0.2168 SF-ALL-Macro TinkerBell_KB_ENG_2 1 0.2024 0.2506 0.2068 SF-ALL-Macro TinkerBell_KB_ENG_2 ALL 0.1996 0.2888 0.2129 LDC-MAX-ALL-Micro TinkerBell_KB_ENG_2 0 0.2069 0.3111 0.2485 LDC-MAX-ALL-Micro TinkerBell_KB_ENG_2 1 0.0349 0.2468 0.0611 LDC-MAX-ALL-Micro TinkerBell_KB_ENG_2 ALL 0.1059 0.2962 0.1560 LDC-MAX-ALL-Macro TinkerBell_KB_ENG_2 0 0.2304 0.3663 0.2532 LDC-MAX-ALL-Macro TinkerBell_KB_ENG_2 1 0.1914 0.2571 0.2015 LDC-MAX-ALL-Macro TinkerBell_KB_ENG_2 ALL 0.2153 0.3239 0.2331 LDC-MEAN-ALL-Macro TinkerBell_KB_ENG_2 0 0.2205 0.3473 0.2422 LDC-MEAN-ALL-Macro TinkerBell_KB_ENG_2 1 0.1984 0.2596 0.2060 LDC-MEAN-ALL-Macro TinkerBell_KB_ENG_2 ALL 0.2119 0.3132 0.2281 *ALL-Macro Prec, Recall and F1 refer to mean-precision, mean-recall and mean-F1. --------------------- Composite KB Evaluation (EVENT-SLOTS only): Scores based on confidence values and Average Precision (AP); k=3 justifications allowed: SUMMARY: This section provides summary of AP scores Metric RunID Hop AP SF-ALL-Macro TinkerBell_KB_ENG_2 0 0.0022 SF-ALL-Macro TinkerBell_KB_ENG_2 ALL 0.0022 LDC-MEAN-ALL-Macro TinkerBell_KB_ENG_2 0 0.0037 LDC-MEAN-ALL-Macro TinkerBell_KB_ENG_2 ALL 0.0037 (PRIMARY METRIC) *ALL-Macro AP refer to mean of corresponding AP values. Scores based on confidence values and Average Precision (AP); only k=1 justification allowed: SUMMARY: This section provides summary of AP scores Metric RunID Hop AP SF-ALL-Macro TinkerBell_KB_ENG_2 0 0.0024 SF-ALL-Macro TinkerBell_KB_ENG_2 ALL 0.0024 LDC-MEAN-ALL-Macro TinkerBell_KB_ENG_2 0 0.0040 LDC-MEAN-ALL-Macro TinkerBell_KB_ENG_2 ALL 0.0040 *ALL-Macro AP refer to mean of corresponding AP values. Scores based on P/R/F1; only k=1 justification allowed: Metric RunID Hop Prec Recall F1 SF-ALL-Micro TinkerBell_KB_ENG_2 0 0.0673 0.0081 0.0145 SF-ALL-Micro TinkerBell_KB_ENG_2 ALL 0.0673 0.0081 0.0145 SF-ALL-Macro TinkerBell_KB_ENG_2 0 0.0179 0.0034 0.0044 SF-ALL-Macro TinkerBell_KB_ENG_2 ALL 0.0179 0.0034 0.0044 LDC-MAX-ALL-Micro TinkerBell_KB_ENG_2 0 0.0893 0.0134 0.0233 LDC-MAX-ALL-Micro TinkerBell_KB_ENG_2 ALL 0.0893 0.0134 0.0233 LDC-MAX-ALL-Macro TinkerBell_KB_ENG_2 0 0.0326 0.0074 0.0085 LDC-MAX-ALL-Macro TinkerBell_KB_ENG_2 ALL 0.0326 0.0074 0.0085 LDC-MEAN-ALL-Macro TinkerBell_KB_ENG_2 0 0.0318 0.0055 0.0074 LDC-MEAN-ALL-Macro TinkerBell_KB_ENG_2 ALL 0.0318 0.0055 0.0074 *ALL-Macro Prec, Recall and F1 refer to mean-precision, mean-recall and mean-F1. --------------------- Composite KB Evaluation (SENTIMENT-SLOTS only): Scores based on confidence values and Average Precision (AP); k=3 justifications allowed: Metric RunID Hop AP SF-ALL-Macro TinkerBell_KB_ENG_2 0 0.0601 SF-ALL-Macro TinkerBell_KB_ENG_2 1 0.0033 SF-ALL-Macro TinkerBell_KB_ENG_2 ALL 0.0466 LDC-MEAN-ALL-Macro TinkerBell_KB_ENG_2 0 0.0637 LDC-MEAN-ALL-Macro TinkerBell_KB_ENG_2 1 0.0045 LDC-MEAN-ALL-Macro TinkerBell_KB_ENG_2 ALL 0.0469 (PRIMARY METRIC) *ALL-Macro AP refer to mean of corresponding AP values. Scores based on confidence values and Average Precision (AP); only k=1 justification allowed: Metric RunID Hop AP SF-ALL-Macro TinkerBell_KB_ENG_2 0 0.0529 SF-ALL-Macro TinkerBell_KB_ENG_2 1 0.0038 SF-ALL-Macro TinkerBell_KB_ENG_2 ALL 0.0439 LDC-MEAN-ALL-Macro TinkerBell_KB_ENG_2 0 0.0506 LDC-MEAN-ALL-Macro TinkerBell_KB_ENG_2 1 0.0052 LDC-MEAN-ALL-Macro TinkerBell_KB_ENG_2 ALL 0.0401 *ALL-Macro AP refer to mean of corresponding AP values. Scores based on P/R/F1; only k=1 justification allowed: Metric RunID Hop Prec Recall F1 SF-ALL-Micro TinkerBell_KB_ENG_2 0 0.0709 0.2862 0.1137 SF-ALL-Micro TinkerBell_KB_ENG_2 1 0.0023 0.1500 0.0045 SF-ALL-Micro TinkerBell_KB_ENG_2 ALL 0.0157 0.2588 0.0295 SF-ALL-Macro TinkerBell_KB_ENG_2 0 0.0485 0.1194 0.0453 SF-ALL-Macro TinkerBell_KB_ENG_2 1 0.0555 0.1096 0.0674 SF-ALL-Macro TinkerBell_KB_ENG_2 ALL 0.0506 0.1165 0.0518 LDC-MAX-ALL-Micro TinkerBell_KB_ENG_2 0 0.0726 0.3125 0.1178 LDC-MAX-ALL-Micro TinkerBell_KB_ENG_2 1 0.0033 0.2000 0.0064 LDC-MAX-ALL-Micro TinkerBell_KB_ENG_2 ALL 0.0176 0.2887 0.0332 LDC-MAX-ALL-Macro TinkerBell_KB_ENG_2 0 0.0384 0.1288 0.0430 LDC-MAX-ALL-Macro TinkerBell_KB_ENG_2 1 0.0750 0.1481 0.0911 LDC-MAX-ALL-Macro TinkerBell_KB_ENG_2 ALL 0.0492 0.1345 0.0573 LDC-MEAN-ALL-Macro TinkerBell_KB_ENG_2 0 0.0362 0.1205 0.0397 LDC-MEAN-ALL-Macro TinkerBell_KB_ENG_2 1 0.0750 0.1481 0.0911 LDC-MEAN-ALL-Macro TinkerBell_KB_ENG_2 ALL 0.0477 0.1287 0.0549 *ALL-Macro Prec, Recall and F1 refer to mean-precision, mean-recall and mean-F1. ************************************************************* Run ID: TinkerBell_KB_ENG_3 Did the run access the live Web during the evaluation window: No Did the run extract relations from the Cold Start source corpus: Yes Did the run generate meaningful confidence values: Yes Did the run attempt the full Entity Discovery and Linking task: Yes Did the run attempt the full Event Nugget Detection and Coreference task: Yes Did the run include SF relations: Yes Did the run attempt the Event Argument Extraction and Linking task: Yes Did the run include Sentiment relations: Yes --------------------- Composite KB Evaluation (All slot types): Scores based on confidence values and Average Precision (AP); k=3 justifications allowed: Metric RunID Hop AP SF-ALL-Macro TinkerBell_KB_ENG_3 0 0.1097 SF-ALL-Macro TinkerBell_KB_ENG_3 1 0.0243 SF-ALL-Macro TinkerBell_KB_ENG_3 ALL 0.0855 LDC-MEAN-ALL-Macro TinkerBell_KB_ENG_3 0 0.1270 LDC-MEAN-ALL-Macro TinkerBell_KB_ENG_3 1 0.0336 LDC-MEAN-ALL-Macro TinkerBell_KB_ENG_3 ALL 0.1004 (PRIMARY METRIC) *ALL-Macro AP refer to mean of corresponding AP values. Scores based on confidence values and Average Precision (AP); only k=1 justification allowed: Metric RunID Hop AP SF-ALL-Macro TinkerBell_KB_ENG_3 0 0.1296 SF-ALL-Macro TinkerBell_KB_ENG_3 1 0.0361 SF-ALL-Macro TinkerBell_KB_ENG_3 ALL 0.1056 LDC-MEAN-ALL-Macro TinkerBell_KB_ENG_3 0 0.1514 LDC-MEAN-ALL-Macro TinkerBell_KB_ENG_3 1 0.0452 LDC-MEAN-ALL-Macro TinkerBell_KB_ENG_3 ALL 0.1251 *ALL-Macro AP refer to mean of corresponding AP values. Scores based on P/R/F1; only k=1 justification allowed: Metric RunID Hop Prec Recall F1 SF-ALL-Micro TinkerBell_KB_ENG_3 0 0.1317 0.1887 0.1551 SF-ALL-Micro TinkerBell_KB_ENG_3 1 0.0076 0.1599 0.0145 SF-ALL-Micro TinkerBell_KB_ENG_3 ALL 0.0309 0.1821 0.0528 SF-ALL-Macro TinkerBell_KB_ENG_3 0 0.1016 0.1599 0.1042 SF-ALL-Macro TinkerBell_KB_ENG_3 1 0.1182 0.1546 0.1225 SF-ALL-Macro TinkerBell_KB_ENG_3 ALL 0.1073 0.1581 0.1105 LDC-MAX-ALL-Micro TinkerBell_KB_ENG_3 0 0.1524 0.2012 0.1734 LDC-MAX-ALL-Micro TinkerBell_KB_ENG_3 1 0.0083 0.1691 0.0158 LDC-MAX-ALL-Micro TinkerBell_KB_ENG_3 ALL 0.0358 0.1943 0.0604 LDC-MAX-ALL-Macro TinkerBell_KB_ENG_3 0 0.1254 0.1955 0.1282 LDC-MAX-ALL-Macro TinkerBell_KB_ENG_3 1 0.1194 0.1698 0.1275 LDC-MAX-ALL-Macro TinkerBell_KB_ENG_3 ALL 0.1233 0.1867 0.1279 LDC-MEAN-ALL-Macro TinkerBell_KB_ENG_3 0 0.1201 0.1845 0.1221 LDC-MEAN-ALL-Macro TinkerBell_KB_ENG_3 1 0.1233 0.1712 0.1300 LDC-MEAN-ALL-Macro TinkerBell_KB_ENG_3 ALL 0.1212 0.1799 0.1248 *ALL-Macro Prec, Recall and F1 refer to mean-precision, mean-recall and mean-F1. --------------------- Composite KB Evaluation (SF-SLOTS only): Scores based on confidence values and Average Precision (AP); k=3 justifications allowed: Metric RunID Hop AP SF-ALL-Macro TinkerBell_KB_ENG_3 0 0.2235 SF-ALL-Macro TinkerBell_KB_ENG_3 1 0.0657 SF-ALL-Macro TinkerBell_KB_ENG_3 ALL 0.1738 LDC-MEAN-ALL-Macro TinkerBell_KB_ENG_3 0 0.2448 LDC-MEAN-ALL-Macro TinkerBell_KB_ENG_3 1 0.0858 LDC-MEAN-ALL-Macro TinkerBell_KB_ENG_3 ALL 0.1941 (PRIMARY METRIC) *ALL-Macro AP refer to mean of corresponding AP values. Scores based on confidence values and Average Precision (AP); only k=1 justification allowed: Metric RunID Hop AP SF-ALL-Macro TinkerBell_KB_ENG_3 0 0.2748 SF-ALL-Macro TinkerBell_KB_ENG_3 1 0.0990 SF-ALL-Macro TinkerBell_KB_ENG_3 ALL 0.2226 LDC-MEAN-ALL-Macro TinkerBell_KB_ENG_3 0 0.3042 LDC-MEAN-ALL-Macro TinkerBell_KB_ENG_3 1 0.1166 LDC-MEAN-ALL-Macro TinkerBell_KB_ENG_3 ALL 0.2511 *ALL-Macro AP refer to mean of corresponding AP values. Scores based on P/R/F1; only k=1 justification allowed: Metric RunID Hop Prec Recall F1 SF-ALL-Micro TinkerBell_KB_ENG_3 0 0.1742 0.2962 0.2194 SF-ALL-Micro TinkerBell_KB_ENG_3 1 0.0325 0.2568 0.0577 SF-ALL-Micro TinkerBell_KB_ENG_3 ALL 0.0899 0.2867 0.1368 SF-ALL-Macro TinkerBell_KB_ENG_3 0 0.1977 0.3138 0.2168 SF-ALL-Macro TinkerBell_KB_ENG_3 1 0.2024 0.2506 0.2068 SF-ALL-Macro TinkerBell_KB_ENG_3 ALL 0.1996 0.2888 0.2129 LDC-MAX-ALL-Micro TinkerBell_KB_ENG_3 0 0.2098 0.3111 0.2506 LDC-MAX-ALL-Micro TinkerBell_KB_ENG_3 1 0.0366 0.2468 0.0638 LDC-MAX-ALL-Micro TinkerBell_KB_ENG_3 ALL 0.1097 0.2962 0.1601 LDC-MAX-ALL-Macro TinkerBell_KB_ENG_3 0 0.2304 0.3663 0.2532 LDC-MAX-ALL-Macro TinkerBell_KB_ENG_3 1 0.1914 0.2571 0.2015 LDC-MAX-ALL-Macro TinkerBell_KB_ENG_3 ALL 0.2153 0.3239 0.2331 LDC-MEAN-ALL-Macro TinkerBell_KB_ENG_3 0 0.2205 0.3473 0.2422 LDC-MEAN-ALL-Macro TinkerBell_KB_ENG_3 1 0.1984 0.2596 0.2060 LDC-MEAN-ALL-Macro TinkerBell_KB_ENG_3 ALL 0.2119 0.3132 0.2281 *ALL-Macro Prec, Recall and F1 refer to mean-precision, mean-recall and mean-F1. --------------------- Composite KB Evaluation (EVENT-SLOTS only): Scores based on confidence values and Average Precision (AP); k=3 justifications allowed: SUMMARY: This section provides summary of AP scores Metric RunID Hop AP SF-ALL-Macro TinkerBell_KB_ENG_3 0 0.0022 SF-ALL-Macro TinkerBell_KB_ENG_3 ALL 0.0022 LDC-MEAN-ALL-Macro TinkerBell_KB_ENG_3 0 0.0037 LDC-MEAN-ALL-Macro TinkerBell_KB_ENG_3 ALL 0.0037 (PRIMARY METRIC) *ALL-Macro AP refer to mean of corresponding AP values. Scores based on confidence values and Average Precision (AP); only k=1 justification allowed: SUMMARY: This section provides summary of AP scores Metric RunID Hop AP SF-ALL-Macro TinkerBell_KB_ENG_3 0 0.0024 SF-ALL-Macro TinkerBell_KB_ENG_3 ALL 0.0024 LDC-MEAN-ALL-Macro TinkerBell_KB_ENG_3 0 0.0040 LDC-MEAN-ALL-Macro TinkerBell_KB_ENG_3 ALL 0.0040 *ALL-Macro AP refer to mean of corresponding AP values. Scores based on P/R/F1; only k=1 justification allowed: Metric RunID Hop Prec Recall F1 SF-ALL-Micro TinkerBell_KB_ENG_3 0 0.0673 0.0081 0.0145 SF-ALL-Micro TinkerBell_KB_ENG_3 ALL 0.0673 0.0081 0.0145 SF-ALL-Macro TinkerBell_KB_ENG_3 0 0.0179 0.0034 0.0044 SF-ALL-Macro TinkerBell_KB_ENG_3 ALL 0.0179 0.0034 0.0044 LDC-MAX-ALL-Micro TinkerBell_KB_ENG_3 0 0.0943 0.0134 0.0235 LDC-MAX-ALL-Micro TinkerBell_KB_ENG_3 ALL 0.0943 0.0134 0.0235 LDC-MAX-ALL-Macro TinkerBell_KB_ENG_3 0 0.0326 0.0074 0.0085 LDC-MAX-ALL-Macro TinkerBell_KB_ENG_3 ALL 0.0326 0.0074 0.0085 LDC-MEAN-ALL-Macro TinkerBell_KB_ENG_3 0 0.0318 0.0055 0.0074 LDC-MEAN-ALL-Macro TinkerBell_KB_ENG_3 ALL 0.0318 0.0055 0.0074 *ALL-Macro Prec, Recall and F1 refer to mean-precision, mean-recall and mean-F1. --------------------- Composite KB Evaluation (SENTIMENT-SLOTS only): Scores based on confidence values and Average Precision (AP); k=3 justifications allowed: Metric RunID Hop AP SF-ALL-Macro TinkerBell_KB_ENG_3 0 0.0632 SF-ALL-Macro TinkerBell_KB_ENG_3 1 0.0041 SF-ALL-Macro TinkerBell_KB_ENG_3 ALL 0.0494 LDC-MEAN-ALL-Macro TinkerBell_KB_ENG_3 0 0.0685 LDC-MEAN-ALL-Macro TinkerBell_KB_ENG_3 1 0.0057 LDC-MEAN-ALL-Macro TinkerBell_KB_ENG_3 ALL 0.0508 (PRIMARY METRIC) *ALL-Macro AP refer to mean of corresponding AP values. Scores based on confidence values and Average Precision (AP); only k=1 justification allowed: Metric RunID Hop AP SF-ALL-Macro TinkerBell_KB_ENG_3 0 0.0510 SF-ALL-Macro TinkerBell_KB_ENG_3 1 0.0039 SF-ALL-Macro TinkerBell_KB_ENG_3 ALL 0.0433 LDC-MEAN-ALL-Macro TinkerBell_KB_ENG_3 0 0.0497 LDC-MEAN-ALL-Macro TinkerBell_KB_ENG_3 1 0.0054 LDC-MEAN-ALL-Macro TinkerBell_KB_ENG_3 ALL 0.0403 *ALL-Macro AP refer to mean of corresponding AP values. Scores based on P/R/F1; only k=1 justification allowed: Metric RunID Hop Prec Recall F1 SF-ALL-Micro TinkerBell_KB_ENG_3 0 0.0714 0.2862 0.1142 SF-ALL-Micro TinkerBell_KB_ENG_3 1 0.0023 0.1500 0.0045 SF-ALL-Micro TinkerBell_KB_ENG_3 ALL 0.0157 0.2588 0.0296 SF-ALL-Macro TinkerBell_KB_ENG_3 0 0.0485 0.1194 0.0454 SF-ALL-Macro TinkerBell_KB_ENG_3 1 0.0555 0.1096 0.0674 SF-ALL-Macro TinkerBell_KB_ENG_3 ALL 0.0506 0.1165 0.0518 LDC-MAX-ALL-Micro TinkerBell_KB_ENG_3 0 0.0697 0.3125 0.1140 LDC-MAX-ALL-Micro TinkerBell_KB_ENG_3 1 0.0033 0.2000 0.0064 LDC-MAX-ALL-Micro TinkerBell_KB_ENG_3 ALL 0.0175 0.2887 0.0329 LDC-MAX-ALL-Macro TinkerBell_KB_ENG_3 0 0.0384 0.1288 0.0431 LDC-MAX-ALL-Macro TinkerBell_KB_ENG_3 1 0.0750 0.1481 0.0911 LDC-MAX-ALL-Macro TinkerBell_KB_ENG_3 ALL 0.0492 0.1345 0.0573 LDC-MEAN-ALL-Macro TinkerBell_KB_ENG_3 0 0.0362 0.1205 0.0397 LDC-MEAN-ALL-Macro TinkerBell_KB_ENG_3 1 0.0750 0.1481 0.0911 LDC-MEAN-ALL-Macro TinkerBell_KB_ENG_3 ALL 0.0477 0.1287 0.0550 *ALL-Macro Prec, Recall and F1 refer to mean-precision, mean-recall and mean-F1. ************************************************************* Run ID: TinkerBell_KB_ENG_4 Did the run access the live Web during the evaluation window: No Did the run extract relations from the Cold Start source corpus: Yes Did the run generate meaningful confidence values: Yes Did the run attempt the full Entity Discovery and Linking task: Yes Did the run attempt the full Event Nugget Detection and Coreference task: Yes Did the run include SF relations: Yes Did the run attempt the Event Argument Extraction and Linking task: Yes Did the run include Sentiment relations: Yes --------------------- Composite KB Evaluation (All slot types): Scores based on confidence values and Average Precision (AP); k=3 justifications allowed: Metric RunID Hop AP SF-ALL-Macro TinkerBell_KB_ENG_4 0 0.0786 SF-ALL-Macro TinkerBell_KB_ENG_4 1 0.0273 SF-ALL-Macro TinkerBell_KB_ENG_4 ALL 0.0640 LDC-MEAN-ALL-Macro TinkerBell_KB_ENG_4 0 0.0904 LDC-MEAN-ALL-Macro TinkerBell_KB_ENG_4 1 0.0283 LDC-MEAN-ALL-Macro TinkerBell_KB_ENG_4 ALL 0.0746 (PRIMARY METRIC) *ALL-Macro AP refer to mean of corresponding AP values. Scores based on confidence values and Average Precision (AP); only k=1 justification allowed: Metric RunID Hop AP SF-ALL-Macro TinkerBell_KB_ENG_4 0 0.1336 SF-ALL-Macro TinkerBell_KB_ENG_4 1 0.0408 SF-ALL-Macro TinkerBell_KB_ENG_4 ALL 0.1101 LDC-MEAN-ALL-Macro TinkerBell_KB_ENG_4 0 0.1553 LDC-MEAN-ALL-Macro TinkerBell_KB_ENG_4 1 0.0462 LDC-MEAN-ALL-Macro TinkerBell_KB_ENG_4 ALL 0.1298 *ALL-Macro AP refer to mean of corresponding AP values. Scores based on P/R/F1; only k=1 justification allowed: Metric RunID Hop Prec Recall F1 SF-ALL-Micro TinkerBell_KB_ENG_4 0 0.1509 0.1917 0.1689 SF-ALL-Micro TinkerBell_KB_ENG_4 1 0.0084 0.1599 0.0159 SF-ALL-Micro TinkerBell_KB_ENG_4 ALL 0.0346 0.1845 0.0583 SF-ALL-Macro TinkerBell_KB_ENG_4 0 0.1120 0.1632 0.1136 SF-ALL-Macro TinkerBell_KB_ENG_4 1 0.1203 0.1525 0.1249 SF-ALL-Macro TinkerBell_KB_ENG_4 ALL 0.1148 0.1595 0.1175 LDC-MAX-ALL-Micro TinkerBell_KB_ENG_4 0 0.1735 0.2042 0.1876 LDC-MAX-ALL-Micro TinkerBell_KB_ENG_4 1 0.0088 0.1655 0.0168 LDC-MAX-ALL-Micro TinkerBell_KB_ENG_4 ALL 0.0394 0.1958 0.0656 LDC-MAX-ALL-Macro TinkerBell_KB_ENG_4 0 0.1369 0.1995 0.1388 LDC-MAX-ALL-Macro TinkerBell_KB_ENG_4 1 0.1209 0.1644 0.1291 LDC-MAX-ALL-Macro TinkerBell_KB_ENG_4 ALL 0.1314 0.1875 0.1355 LDC-MEAN-ALL-Macro TinkerBell_KB_ENG_4 0 0.1314 0.1887 0.1325 LDC-MEAN-ALL-Macro TinkerBell_KB_ENG_4 1 0.1248 0.1658 0.1315 LDC-MEAN-ALL-Macro TinkerBell_KB_ENG_4 ALL 0.1291 0.1809 0.1322 *ALL-Macro Prec, Recall and F1 refer to mean-precision, mean-recall and mean-F1. --------------------- Composite KB Evaluation (SF-SLOTS only): Scores based on confidence values and Average Precision (AP); k=3 justifications allowed: Metric RunID Hop AP SF-ALL-Macro TinkerBell_KB_ENG_4 0 0.1531 SF-ALL-Macro TinkerBell_KB_ENG_4 1 0.0747 SF-ALL-Macro TinkerBell_KB_ENG_4 ALL 0.1254 LDC-MEAN-ALL-Macro TinkerBell_KB_ENG_4 0 0.1675 LDC-MEAN-ALL-Macro TinkerBell_KB_ENG_4 1 0.0724 LDC-MEAN-ALL-Macro TinkerBell_KB_ENG_4 ALL 0.1400 (PRIMARY METRIC) *ALL-Macro AP refer to mean of corresponding AP values. Scores based on confidence values and Average Precision (AP); only k=1 justification allowed: Metric RunID Hop AP SF-ALL-Macro TinkerBell_KB_ENG_4 0 0.2832 SF-ALL-Macro TinkerBell_KB_ENG_4 1 0.1120 SF-ALL-Macro TinkerBell_KB_ENG_4 ALL 0.2327 LDC-MEAN-ALL-Macro TinkerBell_KB_ENG_4 0 0.3123 LDC-MEAN-ALL-Macro TinkerBell_KB_ENG_4 1 0.1193 LDC-MEAN-ALL-Macro TinkerBell_KB_ENG_4 ALL 0.2613 *ALL-Macro AP refer to mean of corresponding AP values. Scores based on P/R/F1; only k=1 justification allowed: Metric RunID Hop Prec Recall F1 SF-ALL-Micro TinkerBell_KB_ENG_4 0 0.2214 0.3022 0.2556 SF-ALL-Micro TinkerBell_KB_ENG_4 1 0.0610 0.2568 0.0985 SF-ALL-Micro TinkerBell_KB_ENG_4 ALL 0.1422 0.2913 0.1911 SF-ALL-Macro TinkerBell_KB_ENG_4 0 0.2216 0.3214 0.2385 SF-ALL-Macro TinkerBell_KB_ENG_4 1 0.2062 0.2468 0.2114 SF-ALL-Macro TinkerBell_KB_ENG_4 ALL 0.2155 0.2918 0.2277 LDC-MAX-ALL-Micro TinkerBell_KB_ENG_4 0 0.2566 0.3168 0.2835 LDC-MAX-ALL-Micro TinkerBell_KB_ENG_4 1 0.0563 0.2405 0.0912 LDC-MAX-ALL-Micro TinkerBell_KB_ENG_4 ALL 0.1543 0.2991 0.2036 LDC-MAX-ALL-Macro TinkerBell_KB_ENG_4 0 0.2553 0.3750 0.2761 LDC-MAX-ALL-Macro TinkerBell_KB_ENG_4 1 0.1941 0.2476 0.2044 LDC-MAX-ALL-Macro TinkerBell_KB_ENG_4 ALL 0.2315 0.3256 0.2482 LDC-MEAN-ALL-Macro TinkerBell_KB_ENG_4 0 0.2448 0.3564 0.2647 LDC-MEAN-ALL-Macro TinkerBell_KB_ENG_4 1 0.2009 0.2501 0.2087 LDC-MEAN-ALL-Macro TinkerBell_KB_ENG_4 ALL 0.2278 0.3152 0.2429 *ALL-Macro Prec, Recall and F1 refer to mean-precision, mean-recall and mean-F1. --------------------- Composite KB Evaluation (EVENT-SLOTS only): Scores based on confidence values and Average Precision (AP); k=3 justifications allowed: SUMMARY: This section provides summary of AP scores Metric RunID Hop AP SF-ALL-Macro TinkerBell_KB_ENG_4 0 0.0022 SF-ALL-Macro TinkerBell_KB_ENG_4 ALL 0.0022 LDC-MEAN-ALL-Macro TinkerBell_KB_ENG_4 0 0.0037 LDC-MEAN-ALL-Macro TinkerBell_KB_ENG_4 ALL 0.0037 (PRIMARY METRIC) *ALL-Macro AP refer to mean of corresponding AP values. Scores based on confidence values and Average Precision (AP); only k=1 justification allowed: SUMMARY: This section provides summary of AP scores Metric RunID Hop AP SF-ALL-Macro TinkerBell_KB_ENG_4 0 0.0024 SF-ALL-Macro TinkerBell_KB_ENG_4 ALL 0.0024 LDC-MEAN-ALL-Macro TinkerBell_KB_ENG_4 0 0.0040 LDC-MEAN-ALL-Macro TinkerBell_KB_ENG_4 ALL 0.0040 *ALL-Macro AP refer to mean of corresponding AP values. Scores based on P/R/F1; only k=1 justification allowed: Metric RunID Hop Prec Recall F1 SF-ALL-Micro TinkerBell_KB_ENG_4 0 0.0673 0.0081 0.0145 SF-ALL-Micro TinkerBell_KB_ENG_4 ALL 0.0673 0.0081 0.0145 SF-ALL-Macro TinkerBell_KB_ENG_4 0 0.0179 0.0034 0.0044 SF-ALL-Macro TinkerBell_KB_ENG_4 ALL 0.0179 0.0034 0.0044 LDC-MAX-ALL-Micro TinkerBell_KB_ENG_4 0 0.0962 0.0134 0.0235 LDC-MAX-ALL-Micro TinkerBell_KB_ENG_4 ALL 0.0962 0.0134 0.0235 LDC-MAX-ALL-Macro TinkerBell_KB_ENG_4 0 0.0326 0.0074 0.0085 LDC-MAX-ALL-Macro TinkerBell_KB_ENG_4 ALL 0.0326 0.0074 0.0085 LDC-MEAN-ALL-Macro TinkerBell_KB_ENG_4 0 0.0318 0.0055 0.0074 LDC-MEAN-ALL-Macro TinkerBell_KB_ENG_4 ALL 0.0318 0.0055 0.0074 *ALL-Macro Prec, Recall and F1 refer to mean-precision, mean-recall and mean-F1. --------------------- Composite KB Evaluation (SENTIMENT-SLOTS only): Scores based on confidence values and Average Precision (AP); k=3 justifications allowed: Metric RunID Hop AP SF-ALL-Macro TinkerBell_KB_ENG_4 0 0.0601 SF-ALL-Macro TinkerBell_KB_ENG_4 1 0.0033 SF-ALL-Macro TinkerBell_KB_ENG_4 ALL 0.0466 LDC-MEAN-ALL-Macro TinkerBell_KB_ENG_4 0 0.0637 LDC-MEAN-ALL-Macro TinkerBell_KB_ENG_4 1 0.0045 LDC-MEAN-ALL-Macro TinkerBell_KB_ENG_4 ALL 0.0469 (PRIMARY METRIC) *ALL-Macro AP refer to mean of corresponding AP values. Scores based on confidence values and Average Precision (AP); only k=1 justification allowed: Metric RunID Hop AP SF-ALL-Macro TinkerBell_KB_ENG_4 0 0.0529 SF-ALL-Macro TinkerBell_KB_ENG_4 1 0.0038 SF-ALL-Macro TinkerBell_KB_ENG_4 ALL 0.0439 LDC-MEAN-ALL-Macro TinkerBell_KB_ENG_4 0 0.0506 LDC-MEAN-ALL-Macro TinkerBell_KB_ENG_4 1 0.0052 LDC-MEAN-ALL-Macro TinkerBell_KB_ENG_4 ALL 0.0401 *ALL-Macro AP refer to mean of corresponding AP values. Scores based on P/R/F1; only k=1 justification allowed: Metric RunID Hop Prec Recall F1 SF-ALL-Micro TinkerBell_KB_ENG_4 0 0.0709 0.2862 0.1137 SF-ALL-Micro TinkerBell_KB_ENG_4 1 0.0023 0.1500 0.0045 SF-ALL-Micro TinkerBell_KB_ENG_4 ALL 0.0157 0.2588 0.0295 SF-ALL-Macro TinkerBell_KB_ENG_4 0 0.0485 0.1194 0.0453 SF-ALL-Macro TinkerBell_KB_ENG_4 1 0.0555 0.1096 0.0674 SF-ALL-Macro TinkerBell_KB_ENG_4 ALL 0.0506 0.1165 0.0518 LDC-MAX-ALL-Micro TinkerBell_KB_ENG_4 0 0.0688 0.3125 0.1127 LDC-MAX-ALL-Micro TinkerBell_KB_ENG_4 1 0.0033 0.2000 0.0064 LDC-MAX-ALL-Micro TinkerBell_KB_ENG_4 ALL 0.0175 0.2887 0.0329 LDC-MAX-ALL-Macro TinkerBell_KB_ENG_4 0 0.0384 0.1288 0.0430 LDC-MAX-ALL-Macro TinkerBell_KB_ENG_4 1 0.0750 0.1481 0.0911 LDC-MAX-ALL-Macro TinkerBell_KB_ENG_4 ALL 0.0492 0.1345 0.0573 LDC-MEAN-ALL-Macro TinkerBell_KB_ENG_4 0 0.0362 0.1205 0.0397 LDC-MEAN-ALL-Macro TinkerBell_KB_ENG_4 1 0.0750 0.1481 0.0911 LDC-MEAN-ALL-Macro TinkerBell_KB_ENG_4 ALL 0.0477 0.1287 0.0549 *ALL-Macro Prec, Recall and F1 refer to mean-precision, mean-recall and mean-F1. ************************************************************* Run ID: TinkerBell_KB_ENG_5 Did the run access the live Web during the evaluation window: No Did the run extract relations from the Cold Start source corpus: Yes Did the run generate meaningful confidence values: Yes Did the run attempt the full Entity Discovery and Linking task: Yes Did the run attempt the full Event Nugget Detection and Coreference task: Yes Did the run include SF relations: Yes Did the run attempt the Event Argument Extraction and Linking task: Yes Did the run include Sentiment relations: Yes --------------------- Composite KB Evaluation (All slot types): Scores based on confidence values and Average Precision (AP); k=3 justifications allowed: Metric RunID Hop AP SF-ALL-Macro TinkerBell_KB_ENG_5 0 0.0803 SF-ALL-Macro TinkerBell_KB_ENG_5 1 0.0167 SF-ALL-Macro TinkerBell_KB_ENG_5 ALL 0.0638 LDC-MEAN-ALL-Macro TinkerBell_KB_ENG_5 0 0.0895 LDC-MEAN-ALL-Macro TinkerBell_KB_ENG_5 1 0.0153 LDC-MEAN-ALL-Macro TinkerBell_KB_ENG_5 ALL 0.0716 (PRIMARY METRIC) *ALL-Macro AP refer to mean of corresponding AP values. Scores based on confidence values and Average Precision (AP); only k=1 justification allowed: Metric RunID Hop AP SF-ALL-Macro TinkerBell_KB_ENG_5 0 0.1029 SF-ALL-Macro TinkerBell_KB_ENG_5 1 0.0242 SF-ALL-Macro TinkerBell_KB_ENG_5 ALL 0.0845 LDC-MEAN-ALL-Macro TinkerBell_KB_ENG_5 0 0.1181 LDC-MEAN-ALL-Macro TinkerBell_KB_ENG_5 1 0.0213 LDC-MEAN-ALL-Macro TinkerBell_KB_ENG_5 ALL 0.0972 *ALL-Macro AP refer to mean of corresponding AP values. Scores based on P/R/F1; only k=1 justification allowed: Metric RunID Hop Prec Recall F1 SF-ALL-Micro TinkerBell_KB_ENG_5 0 0.1586 0.1297 0.1427 SF-ALL-Micro TinkerBell_KB_ENG_5 1 0.0105 0.0930 0.0188 SF-ALL-Micro TinkerBell_KB_ENG_5 ALL 0.0457 0.1213 0.0664 SF-ALL-Macro TinkerBell_KB_ENG_5 0 0.1202 0.1212 0.1011 SF-ALL-Macro TinkerBell_KB_ENG_5 1 0.0841 0.0961 0.0835 SF-ALL-Macro TinkerBell_KB_ENG_5 ALL 0.1078 0.1125 0.0951 LDC-MAX-ALL-Micro TinkerBell_KB_ENG_5 0 0.1762 0.1338 0.1521 LDC-MAX-ALL-Micro TinkerBell_KB_ENG_5 1 0.0117 0.0935 0.0208 LDC-MAX-ALL-Micro TinkerBell_KB_ENG_5 ALL 0.0538 0.1251 0.0753 LDC-MAX-ALL-Macro TinkerBell_KB_ENG_5 0 0.1445 0.1435 0.1211 LDC-MAX-ALL-Macro TinkerBell_KB_ENG_5 1 0.0815 0.0905 0.0786 LDC-MAX-ALL-Macro TinkerBell_KB_ENG_5 ALL 0.1230 0.1254 0.1066 LDC-MEAN-ALL-Macro TinkerBell_KB_ENG_5 0 0.1393 0.1357 0.1157 LDC-MEAN-ALL-Macro TinkerBell_KB_ENG_5 1 0.0798 0.0905 0.0782 LDC-MEAN-ALL-Macro TinkerBell_KB_ENG_5 ALL 0.1190 0.1203 0.1029 *ALL-Macro Prec, Recall and F1 refer to mean-precision, mean-recall and mean-F1. --------------------- Composite KB Evaluation (SF-SLOTS only): Scores based on confidence values and Average Precision (AP); k=3 justifications allowed: Metric RunID Hop AP SF-ALL-Macro TinkerBell_KB_ENG_5 0 0.1570 SF-ALL-Macro TinkerBell_KB_ENG_5 1 0.0444 SF-ALL-Macro TinkerBell_KB_ENG_5 ALL 0.1249 LDC-MEAN-ALL-Macro TinkerBell_KB_ENG_5 0 0.1656 LDC-MEAN-ALL-Macro TinkerBell_KB_ENG_5 1 0.0375 LDC-MEAN-ALL-Macro TinkerBell_KB_ENG_5 ALL 0.1335 (PRIMARY METRIC) *ALL-Macro AP refer to mean of corresponding AP values. Scores based on confidence values and Average Precision (AP); only k=1 justification allowed: Metric RunID Hop AP SF-ALL-Macro TinkerBell_KB_ENG_5 0 0.2123 SF-ALL-Macro TinkerBell_KB_ENG_5 1 0.0654 SF-ALL-Macro TinkerBell_KB_ENG_5 ALL 0.1737 LDC-MEAN-ALL-Macro TinkerBell_KB_ENG_5 0 0.2322 LDC-MEAN-ALL-Macro TinkerBell_KB_ENG_5 1 0.0533 LDC-MEAN-ALL-Macro TinkerBell_KB_ENG_5 ALL 0.1910 *ALL-Macro AP refer to mean of corresponding AP values. Scores based on P/R/F1; only k=1 justification allowed: Metric RunID Hop Prec Recall F1 SF-ALL-Micro TinkerBell_KB_ENG_5 0 0.3920 0.1770 0.2439 SF-ALL-Micro TinkerBell_KB_ENG_5 1 0.2909 0.1311 0.1808 SF-ALL-Micro TinkerBell_KB_ENG_5 ALL 0.3677 0.1660 0.2288 SF-ALL-Macro TinkerBell_KB_ENG_5 0 0.2407 0.2243 0.2097 SF-ALL-Macro TinkerBell_KB_ENG_5 1 0.1396 0.1428 0.1350 SF-ALL-Macro TinkerBell_KB_ENG_5 ALL 0.2006 0.1920 0.1801 LDC-MAX-ALL-Micro TinkerBell_KB_ENG_5 0 0.4222 0.1813 0.2537 LDC-MAX-ALL-Micro TinkerBell_KB_ENG_5 1 0.2571 0.1139 0.1579 LDC-MAX-ALL-Micro TinkerBell_KB_ENG_5 ALL 0.3831 0.1657 0.2313 LDC-MAX-ALL-Macro TinkerBell_KB_ENG_5 0 0.2717 0.2542 0.2380 LDC-MAX-ALL-Macro TinkerBell_KB_ENG_5 1 0.1245 0.1171 0.1153 LDC-MAX-ALL-Macro TinkerBell_KB_ENG_5 ALL 0.2146 0.2010 0.1903 LDC-MEAN-ALL-Macro TinkerBell_KB_ENG_5 0 0.2620 0.2422 0.2286 LDC-MEAN-ALL-Macro TinkerBell_KB_ENG_5 1 0.1215 0.1173 0.1145 LDC-MEAN-ALL-Macro TinkerBell_KB_ENG_5 ALL 0.2074 0.1937 0.1843 *ALL-Macro Prec, Recall and F1 refer to mean-precision, mean-recall and mean-F1. --------------------- Composite KB Evaluation (EVENT-SLOTS only): Scores based on confidence values and Average Precision (AP); k=3 justifications allowed: SUMMARY: This section provides summary of AP scores Metric RunID Hop AP SF-ALL-Macro TinkerBell_KB_ENG_5 0 0.0022 SF-ALL-Macro TinkerBell_KB_ENG_5 ALL 0.0022 LDC-MEAN-ALL-Macro TinkerBell_KB_ENG_5 0 0.0037 LDC-MEAN-ALL-Macro TinkerBell_KB_ENG_5 ALL 0.0037 (PRIMARY METRIC) *ALL-Macro AP refer to mean of corresponding AP values. Scores based on confidence values and Average Precision (AP); only k=1 justification allowed: SUMMARY: This section provides summary of AP scores Metric RunID Hop AP SF-ALL-Macro TinkerBell_KB_ENG_5 0 0.0024 SF-ALL-Macro TinkerBell_KB_ENG_5 ALL 0.0024 LDC-MEAN-ALL-Macro TinkerBell_KB_ENG_5 0 0.0040 LDC-MEAN-ALL-Macro TinkerBell_KB_ENG_5 ALL 0.0040 *ALL-Macro AP refer to mean of corresponding AP values. Scores based on P/R/F1; only k=1 justification allowed: Metric RunID Hop Prec Recall F1 SF-ALL-Micro TinkerBell_KB_ENG_5 0 0.0673 0.0081 0.0145 SF-ALL-Micro TinkerBell_KB_ENG_5 ALL 0.0673 0.0081 0.0145 SF-ALL-Macro TinkerBell_KB_ENG_5 0 0.0179 0.0034 0.0044 SF-ALL-Macro TinkerBell_KB_ENG_5 ALL 0.0179 0.0034 0.0044 LDC-MAX-ALL-Micro TinkerBell_KB_ENG_5 0 0.1163 0.0134 0.0240 LDC-MAX-ALL-Micro TinkerBell_KB_ENG_5 ALL 0.1163 0.0134 0.0240 LDC-MAX-ALL-Macro TinkerBell_KB_ENG_5 0 0.0326 0.0074 0.0085 LDC-MAX-ALL-Macro TinkerBell_KB_ENG_5 ALL 0.0326 0.0074 0.0085 LDC-MEAN-ALL-Macro TinkerBell_KB_ENG_5 0 0.0318 0.0055 0.0074 LDC-MEAN-ALL-Macro TinkerBell_KB_ENG_5 ALL 0.0318 0.0055 0.0074 *ALL-Macro Prec, Recall and F1 refer to mean-precision, mean-recall and mean-F1. --------------------- Composite KB Evaluation (SENTIMENT-SLOTS only): Scores based on confidence values and Average Precision (AP); k=3 justifications allowed: Metric RunID Hop AP SF-ALL-Macro TinkerBell_KB_ENG_5 0 0.0601 SF-ALL-Macro TinkerBell_KB_ENG_5 1 0.0033 SF-ALL-Macro TinkerBell_KB_ENG_5 ALL 0.0466 LDC-MEAN-ALL-Macro TinkerBell_KB_ENG_5 0 0.0637 LDC-MEAN-ALL-Macro TinkerBell_KB_ENG_5 1 0.0045 LDC-MEAN-ALL-Macro TinkerBell_KB_ENG_5 ALL 0.0469 (PRIMARY METRIC) *ALL-Macro AP refer to mean of corresponding AP values. Scores based on confidence values and Average Precision (AP); only k=1 justification allowed: Metric RunID Hop AP SF-ALL-Macro TinkerBell_KB_ENG_5 0 0.0529 SF-ALL-Macro TinkerBell_KB_ENG_5 1 0.0038 SF-ALL-Macro TinkerBell_KB_ENG_5 ALL 0.0439 LDC-MEAN-ALL-Macro TinkerBell_KB_ENG_5 0 0.0506 LDC-MEAN-ALL-Macro TinkerBell_KB_ENG_5 1 0.0052 LDC-MEAN-ALL-Macro TinkerBell_KB_ENG_5 ALL 0.0401 *ALL-Macro AP refer to mean of corresponding AP values. Scores based on P/R/F1; only k=1 justification allowed: Metric RunID Hop Prec Recall F1 SF-ALL-Micro TinkerBell_KB_ENG_5 0 0.0709 0.2862 0.1137 SF-ALL-Micro TinkerBell_KB_ENG_5 1 0.0023 0.1500 0.0045 SF-ALL-Micro TinkerBell_KB_ENG_5 ALL 0.0157 0.2588 0.0295 SF-ALL-Macro TinkerBell_KB_ENG_5 0 0.0485 0.1194 0.0453 SF-ALL-Macro TinkerBell_KB_ENG_5 1 0.0555 0.1096 0.0674 SF-ALL-Macro TinkerBell_KB_ENG_5 ALL 0.0506 0.1165 0.0518 LDC-MAX-ALL-Micro TinkerBell_KB_ENG_5 0 0.0714 0.3125 0.1163 LDC-MAX-ALL-Micro TinkerBell_KB_ENG_5 1 0.0033 0.2000 0.0064 LDC-MAX-ALL-Micro TinkerBell_KB_ENG_5 ALL 0.0176 0.2887 0.0331 LDC-MAX-ALL-Macro TinkerBell_KB_ENG_5 0 0.0384 0.1288 0.0430 LDC-MAX-ALL-Macro TinkerBell_KB_ENG_5 1 0.0750 0.1481 0.0911 LDC-MAX-ALL-Macro TinkerBell_KB_ENG_5 ALL 0.0492 0.1345 0.0573 LDC-MEAN-ALL-Macro TinkerBell_KB_ENG_5 0 0.0362 0.1205 0.0397 LDC-MEAN-ALL-Macro TinkerBell_KB_ENG_5 1 0.0750 0.1481 0.0911 LDC-MEAN-ALL-Macro TinkerBell_KB_ENG_5 ALL 0.0477 0.1287 0.0549 *ALL-Macro Prec, Recall and F1 refer to mean-precision, mean-recall and mean-F1.