=========================================================================== TAC KBP 2017 CROSS-LINGUAL KB CONSTRUCTION: COMPOSITE KB EVALUATION RESULTS =========================================================================== Team ID: TinkerBell Organization: RPI, UIUC, Stanford, Columbia, Cornell, JHU, UPenn ************************************************************* Run ID: TinkerBell_KB_XLING_1 Did the run access the live Web during the evaluation window: No Did the run extract relations from the Cold Start source corpus: Yes Did the run generate meaningful confidence values: Yes Run number of the English CSKB system that is most closely configured to the English component of this run: 1 Run number of the Spanish CSKB system that is most closely configured to the Spanish component of this run: 1 Run number of the Chinese CSKB system that is most closely configured to the Chinese component of this run: 1 Did the run attempt the full Entity Discovery and Linking task: Yes Did the run attempt the full Event Nugget Detection and Coreference task: Yes Did the run include SF relations: Yes Did the run attempt the Event Argument Extraction and Linking task: Yes Did the run include Sentiment relations: Yes --------------------- Composite KB Evaluation (All slot types): Scores based on confidence values and Average Precision (AP); k=3 justifications allowed: Metric RunID Hop AP SF-ALL-Macro TinkerBell_KB_XLING_1 0 0.1087 SF-ALL-Macro TinkerBell_KB_XLING_1 1 0.0145 SF-ALL-Macro TinkerBell_KB_XLING_1 ALL 0.0753 LDC-MEAN-ALL-Macro TinkerBell_KB_XLING_1 0 0.1071 LDC-MEAN-ALL-Macro TinkerBell_KB_XLING_1 1 0.0142 LDC-MEAN-ALL-Macro TinkerBell_KB_XLING_1 ALL 0.0741 (PRIMARY METRIC) *ALL-Macro AP refer to mean of corresponding AP values. Scores based on confidence values and Average Precision (AP); only k=1 justification allowed: Metric RunID Hop AP SF-ALL-Macro TinkerBell_KB_XLING_1 0 0.1350 SF-ALL-Macro TinkerBell_KB_XLING_1 1 0.0204 SF-ALL-Macro TinkerBell_KB_XLING_1 ALL 0.0989 LDC-MEAN-ALL-Macro TinkerBell_KB_XLING_1 0 0.1333 LDC-MEAN-ALL-Macro TinkerBell_KB_XLING_1 1 0.0200 LDC-MEAN-ALL-Macro TinkerBell_KB_XLING_1 ALL 0.0977 *ALL-Macro AP refer to mean of corresponding AP values. Scores based on P/R/F1; only k=1 justification allowed: Metric RunID Hop Prec Recall F1 SF-ALL-Micro TinkerBell_KB_XLING_1 0 0.1860 0.2385 0.2090 SF-ALL-Micro TinkerBell_KB_XLING_1 1 0.0189 0.3400 0.0359 SF-ALL-Micro TinkerBell_KB_XLING_1 ALL 0.0436 0.2681 0.0750 SF-ALL-Macro TinkerBell_KB_XLING_1 0 0.1556 0.1762 0.1347 SF-ALL-Macro TinkerBell_KB_XLING_1 1 0.1000 0.1522 0.1055 SF-ALL-Macro TinkerBell_KB_XLING_1 ALL 0.1313 0.1657 0.1219 LDC-MAX-ALL-Micro TinkerBell_KB_XLING_1 0 0.2002 0.2754 0.2319 LDC-MAX-ALL-Micro TinkerBell_KB_XLING_1 1 0.0248 0.3429 0.0463 LDC-MAX-ALL-Micro TinkerBell_KB_XLING_1 ALL 0.0589 0.2951 0.0982 LDC-MAX-ALL-Macro TinkerBell_KB_XLING_1 0 0.2007 0.2216 0.1733 LDC-MAX-ALL-Macro TinkerBell_KB_XLING_1 1 0.0998 0.1589 0.1076 LDC-MAX-ALL-Macro TinkerBell_KB_XLING_1 ALL 0.1568 0.1943 0.1447 LDC-MEAN-ALL-Macro TinkerBell_KB_XLING_1 0 0.1534 0.1737 0.1328 LDC-MEAN-ALL-Macro TinkerBell_KB_XLING_1 1 0.0988 0.1505 0.1042 LDC-MEAN-ALL-Macro TinkerBell_KB_XLING_1 ALL 0.1297 0.1636 0.1204 *ALL-Macro Prec, Recall and F1 refer to mean-precision, mean-recall and mean-F1. --------------------- Composite KB Evaluation (SF-SLOTS only): Scores based on confidence values and Average Precision (AP); k=3 justifications allowed: Metric RunID Hop AP SF-ALL-Macro TinkerBell_KB_XLING_1 0 0.1818 SF-ALL-Macro TinkerBell_KB_XLING_1 1 0.0328 SF-ALL-Macro TinkerBell_KB_XLING_1 ALL 0.1138 LDC-MEAN-ALL-Macro TinkerBell_KB_XLING_1 0 0.1810 LDC-MEAN-ALL-Macro TinkerBell_KB_XLING_1 1 0.0328 LDC-MEAN-ALL-Macro TinkerBell_KB_XLING_1 ALL 0.1134 (PRIMARY METRIC) *ALL-Macro AP refer to mean of corresponding AP values. Scores based on confidence values and Average Precision (AP); only k=1 justification allowed: Metric RunID Hop AP SF-ALL-Macro TinkerBell_KB_XLING_1 0 0.2353 SF-ALL-Macro TinkerBell_KB_XLING_1 1 0.0485 SF-ALL-Macro TinkerBell_KB_XLING_1 ALL 0.1587 LDC-MEAN-ALL-Macro TinkerBell_KB_XLING_1 0 0.2348 LDC-MEAN-ALL-Macro TinkerBell_KB_XLING_1 1 0.0485 LDC-MEAN-ALL-Macro TinkerBell_KB_XLING_1 ALL 0.1583 *ALL-Macro AP refer to mean of corresponding AP values. Scores based on P/R/F1; only k=1 justification allowed: Metric RunID Hop Prec Recall F1 SF-ALL-Micro TinkerBell_KB_XLING_1 0 0.2070 0.2182 0.2125 SF-ALL-Micro TinkerBell_KB_XLING_1 1 0.0281 0.1930 0.0491 SF-ALL-Micro TinkerBell_KB_XLING_1 ALL 0.0842 0.2119 0.1205 SF-ALL-Macro TinkerBell_KB_XLING_1 0 0.2051 0.2605 0.1994 SF-ALL-Macro TinkerBell_KB_XLING_1 1 0.1302 0.1584 0.1297 SF-ALL-Macro TinkerBell_KB_XLING_1 ALL 0.1687 0.2108 0.1655 LDC-MAX-ALL-Micro TinkerBell_KB_XLING_1 0 0.2461 0.2505 0.2483 LDC-MAX-ALL-Micro TinkerBell_KB_XLING_1 1 0.0212 0.2000 0.0384 LDC-MAX-ALL-Micro TinkerBell_KB_XLING_1 ALL 0.0760 0.2378 0.1152 LDC-MAX-ALL-Macro TinkerBell_KB_XLING_1 0 0.2754 0.3260 0.2646 LDC-MAX-ALL-Macro TinkerBell_KB_XLING_1 1 0.1297 0.1707 0.1338 LDC-MAX-ALL-Macro TinkerBell_KB_XLING_1 ALL 0.2049 0.2509 0.2013 LDC-MEAN-ALL-Macro TinkerBell_KB_XLING_1 0 0.2048 0.2596 0.1989 LDC-MEAN-ALL-Macro TinkerBell_KB_XLING_1 1 0.1302 0.1584 0.1297 LDC-MEAN-ALL-Macro TinkerBell_KB_XLING_1 ALL 0.1687 0.2106 0.1654 *ALL-Macro Prec, Recall and F1 refer to mean-precision, mean-recall and mean-F1. --------------------- Composite KB Evaluation (EVENT-SLOTS only): Scores based on confidence values and Average Precision (AP); k=3 justifications allowed: SUMMARY: This section provides summary of AP scores Metric RunID Hop AP SF-ALL-Macro TinkerBell_KB_XLING_1 0 0.0413 SF-ALL-Macro TinkerBell_KB_XLING_1 ALL 0.0413 LDC-MEAN-ALL-Macro TinkerBell_KB_XLING_1 0 0.0404 LDC-MEAN-ALL-Macro TinkerBell_KB_XLING_1 ALL 0.0404 (PRIMARY METRIC) *ALL-Macro AP refer to mean of corresponding AP values. Scores based on confidence values and Average Precision (AP); only k=1 justification allowed: SUMMARY: This section provides summary of AP scores Metric RunID Hop AP SF-ALL-Macro TinkerBell_KB_XLING_1 0 0.0568 SF-ALL-Macro TinkerBell_KB_XLING_1 ALL 0.0568 LDC-MEAN-ALL-Macro TinkerBell_KB_XLING_1 0 0.0560 LDC-MEAN-ALL-Macro TinkerBell_KB_XLING_1 ALL 0.0560 *ALL-Macro AP refer to mean of corresponding AP values. Scores based on P/R/F1; only k=1 justification allowed: Metric RunID Hop Prec Recall F1 SF-ALL-Micro TinkerBell_KB_XLING_1 0 0.3064 0.1608 0.2109 SF-ALL-Micro TinkerBell_KB_XLING_1 ALL 0.3064 0.1608 0.2109 SF-ALL-Macro TinkerBell_KB_XLING_1 0 0.1461 0.0811 0.0913 SF-ALL-Macro TinkerBell_KB_XLING_1 ALL 0.1461 0.0811 0.0913 LDC-MAX-ALL-Micro TinkerBell_KB_XLING_1 0 0.4028 0.1908 0.2589 LDC-MAX-ALL-Micro TinkerBell_KB_XLING_1 ALL 0.4028 0.1908 0.2589 LDC-MAX-ALL-Macro TinkerBell_KB_XLING_1 0 0.1916 0.1049 0.1183 LDC-MAX-ALL-Macro TinkerBell_KB_XLING_1 ALL 0.1916 0.1049 0.1183 LDC-MEAN-ALL-Macro TinkerBell_KB_XLING_1 0 0.1430 0.0796 0.0896 LDC-MEAN-ALL-Macro TinkerBell_KB_XLING_1 ALL 0.1430 0.0796 0.0896 *ALL-Macro Prec, Recall and F1 refer to mean-precision, mean-recall and mean-F1. --------------------- Composite KB Evaluation (SENTIMENT-SLOTS only): Scores based on confidence values and Average Precision (AP); k=3 justifications allowed: Metric RunID Hop AP SF-ALL-Macro TinkerBell_KB_XLING_1 0 0.0774 SF-ALL-Macro TinkerBell_KB_XLING_1 1 0.0073 SF-ALL-Macro TinkerBell_KB_XLING_1 ALL 0.0565 LDC-MEAN-ALL-Macro TinkerBell_KB_XLING_1 0 0.0751 LDC-MEAN-ALL-Macro TinkerBell_KB_XLING_1 1 0.0070 LDC-MEAN-ALL-Macro TinkerBell_KB_XLING_1 ALL 0.0548 (PRIMARY METRIC) *ALL-Macro AP refer to mean of corresponding AP values. Scores based on confidence values and Average Precision (AP); only k=1 justification allowed: Metric RunID Hop AP SF-ALL-Macro TinkerBell_KB_XLING_1 0 0.0711 SF-ALL-Macro TinkerBell_KB_XLING_1 1 0.0080 SF-ALL-Macro TinkerBell_KB_XLING_1 ALL 0.0543 LDC-MEAN-ALL-Macro TinkerBell_KB_XLING_1 0 0.0689 LDC-MEAN-ALL-Macro TinkerBell_KB_XLING_1 1 0.0077 LDC-MEAN-ALL-Macro TinkerBell_KB_XLING_1 ALL 0.0526 *ALL-Macro AP refer to mean of corresponding AP values. Scores based on P/R/F1; only k=1 justification allowed: Metric RunID Hop Prec Recall F1 SF-ALL-Micro TinkerBell_KB_XLING_1 0 0.1339 0.4275 0.2040 SF-ALL-Micro TinkerBell_KB_XLING_1 1 0.0119 0.3908 0.0231 SF-ALL-Micro TinkerBell_KB_XLING_1 ALL 0.0379 0.4178 0.0695 SF-ALL-Macro TinkerBell_KB_XLING_1 0 0.0819 0.1652 0.0830 SF-ALL-Macro TinkerBell_KB_XLING_1 1 0.1371 0.3015 0.1647 SF-ALL-Macro TinkerBell_KB_XLING_1 ALL 0.1044 0.2207 0.1163 LDC-MAX-ALL-Micro TinkerBell_KB_XLING_1 0 0.1303 0.4843 0.2053 LDC-MAX-ALL-Micro TinkerBell_KB_XLING_1 1 0.0155 0.3907 0.0298 LDC-MAX-ALL-Micro TinkerBell_KB_XLING_1 ALL 0.0486 0.4594 0.0879 LDC-MAX-ALL-Macro TinkerBell_KB_XLING_1 0 0.0894 0.2047 0.0951 LDC-MAX-ALL-Macro TinkerBell_KB_XLING_1 1 0.1386 0.3015 0.1661 LDC-MAX-ALL-Macro TinkerBell_KB_XLING_1 ALL 0.1094 0.2440 0.1239 LDC-MEAN-ALL-Macro TinkerBell_KB_XLING_1 0 0.0794 0.1602 0.0805 LDC-MEAN-ALL-Macro TinkerBell_KB_XLING_1 1 0.1337 0.2942 0.1607 LDC-MEAN-ALL-Macro TinkerBell_KB_XLING_1 ALL 0.1015 0.2146 0.1131 *ALL-Macro Prec, Recall and F1 refer to mean-precision, mean-recall and mean-F1. ************************************************************* Run ID: TinkerBell_KB_XLING_2 Did the run access the live Web during the evaluation window: No Did the run extract relations from the Cold Start source corpus: Yes Did the run generate meaningful confidence values: Yes Run number of the English CSKB system that is most closely configured to the English component of this run: 2 Run number of the Spanish CSKB system that is most closely configured to the Spanish component of this run: 2 Run number of the Chinese CSKB system that is most closely configured to the Chinese component of this run: 2 Did the run attempt the full Entity Discovery and Linking task: Yes Did the run attempt the full Event Nugget Detection and Coreference task: Yes Did the run include SF relations: Yes Did the run attempt the Event Argument Extraction and Linking task: Yes Did the run include Sentiment relations: Yes --------------------- Composite KB Evaluation (All slot types): Scores based on confidence values and Average Precision (AP); k=3 justifications allowed: Metric RunID Hop AP SF-ALL-Macro TinkerBell_KB_XLING_2 0 0.1067 SF-ALL-Macro TinkerBell_KB_XLING_2 1 0.0136 SF-ALL-Macro TinkerBell_KB_XLING_2 ALL 0.0748 LDC-MEAN-ALL-Macro TinkerBell_KB_XLING_2 0 0.1051 LDC-MEAN-ALL-Macro TinkerBell_KB_XLING_2 1 0.0134 LDC-MEAN-ALL-Macro TinkerBell_KB_XLING_2 ALL 0.0737 (PRIMARY METRIC) *ALL-Macro AP refer to mean of corresponding AP values. Scores based on confidence values and Average Precision (AP); only k=1 justification allowed: Metric RunID Hop AP SF-ALL-Macro TinkerBell_KB_XLING_2 0 0.1348 SF-ALL-Macro TinkerBell_KB_XLING_2 1 0.0196 SF-ALL-Macro TinkerBell_KB_XLING_2 ALL 0.0989 LDC-MEAN-ALL-Macro TinkerBell_KB_XLING_2 0 0.1331 LDC-MEAN-ALL-Macro TinkerBell_KB_XLING_2 1 0.0192 LDC-MEAN-ALL-Macro TinkerBell_KB_XLING_2 ALL 0.0976 *ALL-Macro AP refer to mean of corresponding AP values. Scores based on P/R/F1; only k=1 justification allowed: Metric RunID Hop Prec Recall F1 SF-ALL-Micro TinkerBell_KB_XLING_2 0 0.1870 0.2379 0.2094 SF-ALL-Micro TinkerBell_KB_XLING_2 1 0.0190 0.3400 0.0361 SF-ALL-Micro TinkerBell_KB_XLING_2 ALL 0.0438 0.2677 0.0753 SF-ALL-Macro TinkerBell_KB_XLING_2 0 0.1550 0.1760 0.1343 SF-ALL-Macro TinkerBell_KB_XLING_2 1 0.1000 0.1522 0.1055 SF-ALL-Macro TinkerBell_KB_XLING_2 ALL 0.1310 0.1656 0.1217 LDC-MAX-ALL-Micro TinkerBell_KB_XLING_2 0 0.2037 0.2754 0.2342 LDC-MAX-ALL-Micro TinkerBell_KB_XLING_2 1 0.0251 0.3429 0.0467 LDC-MAX-ALL-Micro TinkerBell_KB_XLING_2 ALL 0.0596 0.2951 0.0992 LDC-MAX-ALL-Macro TinkerBell_KB_XLING_2 0 0.2005 0.2216 0.1731 LDC-MAX-ALL-Macro TinkerBell_KB_XLING_2 1 0.0999 0.1589 0.1077 LDC-MAX-ALL-Macro TinkerBell_KB_XLING_2 ALL 0.1567 0.1943 0.1447 LDC-MEAN-ALL-Macro TinkerBell_KB_XLING_2 0 0.1529 0.1734 0.1325 LDC-MEAN-ALL-Macro TinkerBell_KB_XLING_2 1 0.0989 0.1505 0.1043 LDC-MEAN-ALL-Macro TinkerBell_KB_XLING_2 ALL 0.1294 0.1634 0.1202 *ALL-Macro Prec, Recall and F1 refer to mean-precision, mean-recall and mean-F1. --------------------- Composite KB Evaluation (SF-SLOTS only): Scores based on confidence values and Average Precision (AP); k=3 justifications allowed: Metric RunID Hop AP SF-ALL-Macro TinkerBell_KB_XLING_2 0 0.1773 SF-ALL-Macro TinkerBell_KB_XLING_2 1 0.0322 SF-ALL-Macro TinkerBell_KB_XLING_2 ALL 0.1130 LDC-MEAN-ALL-Macro TinkerBell_KB_XLING_2 0 0.1766 LDC-MEAN-ALL-Macro TinkerBell_KB_XLING_2 1 0.0322 LDC-MEAN-ALL-Macro TinkerBell_KB_XLING_2 ALL 0.1125 (PRIMARY METRIC) *ALL-Macro AP refer to mean of corresponding AP values. Scores based on confidence values and Average Precision (AP); only k=1 justification allowed: Metric RunID Hop AP SF-ALL-Macro TinkerBell_KB_XLING_2 0 0.2353 SF-ALL-Macro TinkerBell_KB_XLING_2 1 0.0486 SF-ALL-Macro TinkerBell_KB_XLING_2 ALL 0.1591 LDC-MEAN-ALL-Macro TinkerBell_KB_XLING_2 0 0.2348 LDC-MEAN-ALL-Macro TinkerBell_KB_XLING_2 1 0.0486 LDC-MEAN-ALL-Macro TinkerBell_KB_XLING_2 ALL 0.1587 *ALL-Macro AP refer to mean of corresponding AP values. Scores based on P/R/F1; only k=1 justification allowed: Metric RunID Hop Prec Recall F1 SF-ALL-Micro TinkerBell_KB_XLING_2 0 0.2112 0.2182 0.2147 SF-ALL-Micro TinkerBell_KB_XLING_2 1 0.0293 0.1930 0.0508 SF-ALL-Micro TinkerBell_KB_XLING_2 ALL 0.0870 0.2119 0.1234 SF-ALL-Macro TinkerBell_KB_XLING_2 0 0.2050 0.2605 0.1993 SF-ALL-Macro TinkerBell_KB_XLING_2 1 0.1303 0.1584 0.1298 SF-ALL-Macro TinkerBell_KB_XLING_2 ALL 0.1687 0.2108 0.1655 LDC-MAX-ALL-Micro TinkerBell_KB_XLING_2 0 0.2440 0.2505 0.2472 LDC-MAX-ALL-Micro TinkerBell_KB_XLING_2 1 0.0456 0.2000 0.0742 LDC-MAX-ALL-Micro TinkerBell_KB_XLING_2 ALL 0.1270 0.2378 0.1656 LDC-MAX-ALL-Macro TinkerBell_KB_XLING_2 0 0.2748 0.3260 0.2642 LDC-MAX-ALL-Macro TinkerBell_KB_XLING_2 1 0.1298 0.1707 0.1340 LDC-MAX-ALL-Macro TinkerBell_KB_XLING_2 ALL 0.2046 0.2509 0.2012 LDC-MEAN-ALL-Macro TinkerBell_KB_XLING_2 0 0.2047 0.2596 0.1988 LDC-MEAN-ALL-Macro TinkerBell_KB_XLING_2 1 0.1303 0.1584 0.1298 LDC-MEAN-ALL-Macro TinkerBell_KB_XLING_2 ALL 0.1687 0.2106 0.1654 *ALL-Macro Prec, Recall and F1 refer to mean-precision, mean-recall and mean-F1. --------------------- Composite KB Evaluation (EVENT-SLOTS only): Scores based on confidence values and Average Precision (AP); k=3 justifications allowed: SUMMARY: This section provides summary of AP scores Metric RunID Hop AP SF-ALL-Macro TinkerBell_KB_XLING_2 0 0.0409 SF-ALL-Macro TinkerBell_KB_XLING_2 ALL 0.0409 LDC-MEAN-ALL-Macro TinkerBell_KB_XLING_2 0 0.0401 LDC-MEAN-ALL-Macro TinkerBell_KB_XLING_2 ALL 0.0401 (PRIMARY METRIC) *ALL-Macro AP refer to mean of corresponding AP values. Scores based on confidence values and Average Precision (AP); only k=1 justification allowed: SUMMARY: This section provides summary of AP scores Metric RunID Hop AP SF-ALL-Macro TinkerBell_KB_XLING_2 0 0.0563 SF-ALL-Macro TinkerBell_KB_XLING_2 ALL 0.0563 LDC-MEAN-ALL-Macro TinkerBell_KB_XLING_2 0 0.0554 LDC-MEAN-ALL-Macro TinkerBell_KB_XLING_2 ALL 0.0554 *ALL-Macro AP refer to mean of corresponding AP values. Scores based on P/R/F1; only k=1 justification allowed: Metric RunID Hop Prec Recall F1 SF-ALL-Micro TinkerBell_KB_XLING_2 0 0.3044 0.1589 0.2088 SF-ALL-Micro TinkerBell_KB_XLING_2 ALL 0.3044 0.1589 0.2088 SF-ALL-Macro TinkerBell_KB_XLING_2 0 0.1446 0.0804 0.0905 SF-ALL-Macro TinkerBell_KB_XLING_2 ALL 0.1446 0.0804 0.0905 LDC-MAX-ALL-Micro TinkerBell_KB_XLING_2 0 0.3598 0.1908 0.2494 LDC-MAX-ALL-Micro TinkerBell_KB_XLING_2 ALL 0.3598 0.1908 0.2494 LDC-MAX-ALL-Macro TinkerBell_KB_XLING_2 0 0.1916 0.1049 0.1183 LDC-MAX-ALL-Macro TinkerBell_KB_XLING_2 ALL 0.1916 0.1049 0.1183 LDC-MEAN-ALL-Macro TinkerBell_KB_XLING_2 0 0.1415 0.0790 0.0887 LDC-MEAN-ALL-Macro TinkerBell_KB_XLING_2 ALL 0.1415 0.0790 0.0887 *ALL-Macro Prec, Recall and F1 refer to mean-precision, mean-recall and mean-F1. --------------------- Composite KB Evaluation (SENTIMENT-SLOTS only): Scores based on confidence values and Average Precision (AP); k=3 justifications allowed: Metric RunID Hop AP SF-ALL-Macro TinkerBell_KB_XLING_2 0 0.0774 SF-ALL-Macro TinkerBell_KB_XLING_2 1 0.0073 SF-ALL-Macro TinkerBell_KB_XLING_2 ALL 0.0564 LDC-MEAN-ALL-Macro TinkerBell_KB_XLING_2 0 0.0751 LDC-MEAN-ALL-Macro TinkerBell_KB_XLING_2 1 0.0070 LDC-MEAN-ALL-Macro TinkerBell_KB_XLING_2 ALL 0.0547 (PRIMARY METRIC) *ALL-Macro AP refer to mean of corresponding AP values. Scores based on confidence values and Average Precision (AP); only k=1 justification allowed: Metric RunID Hop AP SF-ALL-Macro TinkerBell_KB_XLING_2 0 0.0711 SF-ALL-Macro TinkerBell_KB_XLING_2 1 0.0080 SF-ALL-Macro TinkerBell_KB_XLING_2 ALL 0.0542 LDC-MEAN-ALL-Macro TinkerBell_KB_XLING_2 0 0.0689 LDC-MEAN-ALL-Macro TinkerBell_KB_XLING_2 1 0.0077 LDC-MEAN-ALL-Macro TinkerBell_KB_XLING_2 ALL 0.0526 *ALL-Macro AP refer to mean of corresponding AP values. Scores based on P/R/F1; only k=1 justification allowed: Metric RunID Hop Prec Recall F1 SF-ALL-Micro TinkerBell_KB_XLING_2 0 0.1339 0.4275 0.2040 SF-ALL-Micro TinkerBell_KB_XLING_2 1 0.0119 0.3908 0.0231 SF-ALL-Micro TinkerBell_KB_XLING_2 ALL 0.0379 0.4178 0.0695 SF-ALL-Macro TinkerBell_KB_XLING_2 0 0.0819 0.1652 0.0830 SF-ALL-Macro TinkerBell_KB_XLING_2 1 0.1371 0.3015 0.1647 SF-ALL-Macro TinkerBell_KB_XLING_2 ALL 0.1044 0.2207 0.1163 LDC-MAX-ALL-Micro TinkerBell_KB_XLING_2 0 0.1304 0.4843 0.2055 LDC-MAX-ALL-Micro TinkerBell_KB_XLING_2 1 0.0113 0.3907 0.0219 LDC-MAX-ALL-Micro TinkerBell_KB_XLING_2 ALL 0.0384 0.4594 0.0708 LDC-MAX-ALL-Macro TinkerBell_KB_XLING_2 0 0.0894 0.2047 0.0951 LDC-MAX-ALL-Macro TinkerBell_KB_XLING_2 1 0.1386 0.3015 0.1661 LDC-MAX-ALL-Macro TinkerBell_KB_XLING_2 ALL 0.1094 0.2440 0.1239 LDC-MEAN-ALL-Macro TinkerBell_KB_XLING_2 0 0.0794 0.1602 0.0805 LDC-MEAN-ALL-Macro TinkerBell_KB_XLING_2 1 0.1337 0.2942 0.1607 LDC-MEAN-ALL-Macro TinkerBell_KB_XLING_2 ALL 0.1015 0.2146 0.1131 *ALL-Macro Prec, Recall and F1 refer to mean-precision, mean-recall and mean-F1. ************************************************************* Run ID: TinkerBell_KB_XLING_3 Did the run access the live Web during the evaluation window: No Did the run extract relations from the Cold Start source corpus: Yes Did the run generate meaningful confidence values: Yes Run number of the English CSKB system that is most closely configured to the English component of this run: 3 Run number of the Spanish CSKB system that is most closely configured to the Spanish component of this run: 3 Run number of the Chinese CSKB system that is most closely configured to the Chinese component of this run: 3 Did the run attempt the full Entity Discovery and Linking task: Yes Did the run attempt the full Event Nugget Detection and Coreference task: Yes Did the run include SF relations: Yes Did the run attempt the Event Argument Extraction and Linking task: Yes Did the run include Sentiment relations: Yes --------------------- Composite KB Evaluation (All slot types): Scores based on confidence values and Average Precision (AP); k=3 justifications allowed: Metric RunID Hop AP SF-ALL-Macro TinkerBell_KB_XLING_3 0 0.1100 SF-ALL-Macro TinkerBell_KB_XLING_3 1 0.0148 SF-ALL-Macro TinkerBell_KB_XLING_3 ALL 0.0768 LDC-MEAN-ALL-Macro TinkerBell_KB_XLING_3 0 0.1083 LDC-MEAN-ALL-Macro TinkerBell_KB_XLING_3 1 0.0145 LDC-MEAN-ALL-Macro TinkerBell_KB_XLING_3 ALL 0.0756 (PRIMARY METRIC) *ALL-Macro AP refer to mean of corresponding AP values. Scores based on confidence values and Average Precision (AP); only k=1 justification allowed: Metric RunID Hop AP SF-ALL-Macro TinkerBell_KB_XLING_3 0 0.1352 SF-ALL-Macro TinkerBell_KB_XLING_3 1 0.0199 SF-ALL-Macro TinkerBell_KB_XLING_3 ALL 0.0992 LDC-MEAN-ALL-Macro TinkerBell_KB_XLING_3 0 0.1335 LDC-MEAN-ALL-Macro TinkerBell_KB_XLING_3 1 0.0195 LDC-MEAN-ALL-Macro TinkerBell_KB_XLING_3 ALL 0.0979 *ALL-Macro AP refer to mean of corresponding AP values. Scores based on P/R/F1; only k=1 justification allowed: Metric RunID Hop Prec Recall F1 SF-ALL-Micro TinkerBell_KB_XLING_3 0 0.1910 0.2452 0.2147 SF-ALL-Micro TinkerBell_KB_XLING_3 1 0.0190 0.3376 0.0359 SF-ALL-Micro TinkerBell_KB_XLING_3 ALL 0.0446 0.2722 0.0766 SF-ALL-Macro TinkerBell_KB_XLING_3 0 0.1560 0.1779 0.1353 SF-ALL-Macro TinkerBell_KB_XLING_3 1 0.1000 0.1522 0.1055 SF-ALL-Macro TinkerBell_KB_XLING_3 ALL 0.1315 0.1667 0.1223 LDC-MAX-ALL-Micro TinkerBell_KB_XLING_3 0 0.2067 0.2816 0.2384 LDC-MAX-ALL-Micro TinkerBell_KB_XLING_3 1 0.0223 0.3406 0.0419 LDC-MAX-ALL-Micro TinkerBell_KB_XLING_3 ALL 0.0552 0.2988 0.0931 LDC-MAX-ALL-Macro TinkerBell_KB_XLING_3 0 0.2023 0.2223 0.1741 LDC-MAX-ALL-Macro TinkerBell_KB_XLING_3 1 0.0999 0.1588 0.1077 LDC-MAX-ALL-Macro TinkerBell_KB_XLING_3 ALL 0.1577 0.1947 0.1452 LDC-MEAN-ALL-Macro TinkerBell_KB_XLING_3 0 0.1538 0.1753 0.1334 LDC-MEAN-ALL-Macro TinkerBell_KB_XLING_3 1 0.0989 0.1504 0.1043 LDC-MEAN-ALL-Macro TinkerBell_KB_XLING_3 ALL 0.1299 0.1645 0.1208 *ALL-Macro Prec, Recall and F1 refer to mean-precision, mean-recall and mean-F1. --------------------- Composite KB Evaluation (SF-SLOTS only): Scores based on confidence values and Average Precision (AP); k=3 justifications allowed: Metric RunID Hop AP SF-ALL-Macro TinkerBell_KB_XLING_3 0 0.1818 SF-ALL-Macro TinkerBell_KB_XLING_3 1 0.0328 SF-ALL-Macro TinkerBell_KB_XLING_3 ALL 0.1144 LDC-MEAN-ALL-Macro TinkerBell_KB_XLING_3 0 0.1810 LDC-MEAN-ALL-Macro TinkerBell_KB_XLING_3 1 0.0328 LDC-MEAN-ALL-Macro TinkerBell_KB_XLING_3 ALL 0.1140 (PRIMARY METRIC) *ALL-Macro AP refer to mean of corresponding AP values. Scores based on confidence values and Average Precision (AP); only k=1 justification allowed: Metric RunID Hop AP SF-ALL-Macro TinkerBell_KB_XLING_3 0 0.2353 SF-ALL-Macro TinkerBell_KB_XLING_3 1 0.0485 SF-ALL-Macro TinkerBell_KB_XLING_3 ALL 0.1587 LDC-MEAN-ALL-Macro TinkerBell_KB_XLING_3 0 0.2348 LDC-MEAN-ALL-Macro TinkerBell_KB_XLING_3 1 0.0485 LDC-MEAN-ALL-Macro TinkerBell_KB_XLING_3 ALL 0.1583 *ALL-Macro AP refer to mean of corresponding AP values. Scores based on P/R/F1; only k=1 justification allowed: Metric RunID Hop Prec Recall F1 SF-ALL-Micro TinkerBell_KB_XLING_3 0 0.2070 0.2182 0.2125 SF-ALL-Micro TinkerBell_KB_XLING_3 1 0.0281 0.1930 0.0491 SF-ALL-Micro TinkerBell_KB_XLING_3 ALL 0.0842 0.2119 0.1205 SF-ALL-Macro TinkerBell_KB_XLING_3 0 0.2051 0.2605 0.1994 SF-ALL-Macro TinkerBell_KB_XLING_3 1 0.1302 0.1584 0.1297 SF-ALL-Macro TinkerBell_KB_XLING_3 ALL 0.1687 0.2108 0.1655 LDC-MAX-ALL-Micro TinkerBell_KB_XLING_3 0 0.2568 0.2505 0.2536 LDC-MAX-ALL-Micro TinkerBell_KB_XLING_3 1 0.0475 0.2000 0.0767 LDC-MAX-ALL-Micro TinkerBell_KB_XLING_3 ALL 0.1329 0.2378 0.1705 LDC-MAX-ALL-Macro TinkerBell_KB_XLING_3 0 0.2754 0.3260 0.2646 LDC-MAX-ALL-Macro TinkerBell_KB_XLING_3 1 0.1297 0.1707 0.1338 LDC-MAX-ALL-Macro TinkerBell_KB_XLING_3 ALL 0.2049 0.2509 0.2013 LDC-MEAN-ALL-Macro TinkerBell_KB_XLING_3 0 0.2048 0.2596 0.1989 LDC-MEAN-ALL-Macro TinkerBell_KB_XLING_3 1 0.1302 0.1584 0.1297 LDC-MEAN-ALL-Macro TinkerBell_KB_XLING_3 ALL 0.1687 0.2106 0.1654 *ALL-Macro Prec, Recall and F1 refer to mean-precision, mean-recall and mean-F1. --------------------- Composite KB Evaluation (EVENT-SLOTS only): Scores based on confidence values and Average Precision (AP); k=3 justifications allowed: SUMMARY: This section provides summary of AP scores Metric RunID Hop AP SF-ALL-Macro TinkerBell_KB_XLING_3 0 0.0413 SF-ALL-Macro TinkerBell_KB_XLING_3 ALL 0.0413 LDC-MEAN-ALL-Macro TinkerBell_KB_XLING_3 0 0.0404 LDC-MEAN-ALL-Macro TinkerBell_KB_XLING_3 ALL 0.0404 (PRIMARY METRIC) *ALL-Macro AP refer to mean of corresponding AP values. Scores based on confidence values and Average Precision (AP); only k=1 justification allowed: SUMMARY: This section provides summary of AP scores Metric RunID Hop AP SF-ALL-Macro TinkerBell_KB_XLING_3 0 0.0568 SF-ALL-Macro TinkerBell_KB_XLING_3 ALL 0.0568 LDC-MEAN-ALL-Macro TinkerBell_KB_XLING_3 0 0.0560 LDC-MEAN-ALL-Macro TinkerBell_KB_XLING_3 ALL 0.0560 *ALL-Macro AP refer to mean of corresponding AP values. Scores based on P/R/F1; only k=1 justification allowed: Metric RunID Hop Prec Recall F1 SF-ALL-Micro TinkerBell_KB_XLING_3 0 0.3064 0.1608 0.2109 SF-ALL-Micro TinkerBell_KB_XLING_3 ALL 0.3064 0.1608 0.2109 SF-ALL-Macro TinkerBell_KB_XLING_3 0 0.1461 0.0811 0.0913 SF-ALL-Macro TinkerBell_KB_XLING_3 ALL 0.1461 0.0811 0.0913 LDC-MAX-ALL-Micro TinkerBell_KB_XLING_3 0 0.4085 0.1908 0.2601 LDC-MAX-ALL-Micro TinkerBell_KB_XLING_3 ALL 0.4085 0.1908 0.2601 LDC-MAX-ALL-Macro TinkerBell_KB_XLING_3 0 0.1916 0.1049 0.1183 LDC-MAX-ALL-Macro TinkerBell_KB_XLING_3 ALL 0.1916 0.1049 0.1183 LDC-MEAN-ALL-Macro TinkerBell_KB_XLING_3 0 0.1430 0.0796 0.0896 LDC-MEAN-ALL-Macro TinkerBell_KB_XLING_3 ALL 0.1430 0.0796 0.0896 *ALL-Macro Prec, Recall and F1 refer to mean-precision, mean-recall and mean-F1. --------------------- Composite KB Evaluation (SENTIMENT-SLOTS only): Scores based on confidence values and Average Precision (AP); k=3 justifications allowed: Metric RunID Hop AP SF-ALL-Macro TinkerBell_KB_XLING_3 0 0.0825 SF-ALL-Macro TinkerBell_KB_XLING_3 1 0.0055 SF-ALL-Macro TinkerBell_KB_XLING_3 ALL 0.0618 LDC-MEAN-ALL-Macro TinkerBell_KB_XLING_3 0 0.0800 LDC-MEAN-ALL-Macro TinkerBell_KB_XLING_3 1 0.0053 LDC-MEAN-ALL-Macro TinkerBell_KB_XLING_3 ALL 0.0600 (PRIMARY METRIC) *ALL-Macro AP refer to mean of corresponding AP values. Scores based on confidence values and Average Precision (AP); only k=1 justification allowed: Metric RunID Hop AP SF-ALL-Macro TinkerBell_KB_XLING_3 0 0.0718 SF-ALL-Macro TinkerBell_KB_XLING_3 1 0.0060 SF-ALL-Macro TinkerBell_KB_XLING_3 ALL 0.0555 LDC-MEAN-ALL-Macro TinkerBell_KB_XLING_3 0 0.0696 LDC-MEAN-ALL-Macro TinkerBell_KB_XLING_3 1 0.0058 LDC-MEAN-ALL-Macro TinkerBell_KB_XLING_3 ALL 0.0539 *ALL-Macro AP refer to mean of corresponding AP values. Scores based on P/R/F1; only k=1 justification allowed: Metric RunID Hop Prec Recall F1 SF-ALL-Micro TinkerBell_KB_XLING_3 0 0.1443 0.4616 0.2198 SF-ALL-Micro TinkerBell_KB_XLING_3 1 0.0119 0.3908 0.0231 SF-ALL-Micro TinkerBell_KB_XLING_3 ALL 0.0402 0.4429 0.0737 SF-ALL-Macro TinkerBell_KB_XLING_3 0 0.0837 0.1724 0.0857 SF-ALL-Macro TinkerBell_KB_XLING_3 1 0.1371 0.3015 0.1647 SF-ALL-Macro TinkerBell_KB_XLING_3 ALL 0.1055 0.2250 0.1179 LDC-MAX-ALL-Micro TinkerBell_KB_XLING_3 0 0.1382 0.5181 0.2182 LDC-MAX-ALL-Micro TinkerBell_KB_XLING_3 1 0.0114 0.3907 0.0221 LDC-MAX-ALL-Micro TinkerBell_KB_XLING_3 ALL 0.0406 0.4841 0.0749 LDC-MAX-ALL-Macro TinkerBell_KB_XLING_3 0 0.0912 0.2117 0.0978 LDC-MAX-ALL-Macro TinkerBell_KB_XLING_3 1 0.1386 0.3015 0.1661 LDC-MAX-ALL-Macro TinkerBell_KB_XLING_3 ALL 0.1104 0.2482 0.1255 LDC-MEAN-ALL-Macro TinkerBell_KB_XLING_3 0 0.0812 0.1672 0.0832 LDC-MEAN-ALL-Macro TinkerBell_KB_XLING_3 1 0.1337 0.2942 0.1607 LDC-MEAN-ALL-Macro TinkerBell_KB_XLING_3 ALL 0.1025 0.2187 0.1146 *ALL-Macro Prec, Recall and F1 refer to mean-precision, mean-recall and mean-F1. ************************************************************* Run ID: TinkerBell_KB_XLING_4 Did the run access the live Web during the evaluation window: No Did the run extract relations from the Cold Start source corpus: Yes Did the run generate meaningful confidence values: Yes Run number of the English CSKB system that is most closely configured to the English component of this run: 4 Run number of the Spanish CSKB system that is most closely configured to the Spanish component of this run: 4 Run number of the Chinese CSKB system that is most closely configured to the Chinese component of this run: 4 Did the run attempt the full Entity Discovery and Linking task: Yes Did the run attempt the full Event Nugget Detection and Coreference task: Yes Did the run include SF relations: Yes Did the run attempt the Event Argument Extraction and Linking task: Yes Did the run include Sentiment relations: Yes --------------------- Composite KB Evaluation (All slot types): Scores based on confidence values and Average Precision (AP); k=3 justifications allowed: Metric RunID Hop AP SF-ALL-Macro TinkerBell_KB_XLING_4 0 0.0926 SF-ALL-Macro TinkerBell_KB_XLING_4 1 0.0135 SF-ALL-Macro TinkerBell_KB_XLING_4 ALL 0.0662 LDC-MEAN-ALL-Macro TinkerBell_KB_XLING_4 0 0.0913 LDC-MEAN-ALL-Macro TinkerBell_KB_XLING_4 1 0.0132 LDC-MEAN-ALL-Macro TinkerBell_KB_XLING_4 ALL 0.0653 (PRIMARY METRIC) *ALL-Macro AP refer to mean of corresponding AP values. Scores based on confidence values and Average Precision (AP); only k=1 justification allowed: Metric RunID Hop AP SF-ALL-Macro TinkerBell_KB_XLING_4 0 0.1387 SF-ALL-Macro TinkerBell_KB_XLING_4 1 0.0213 SF-ALL-Macro TinkerBell_KB_XLING_4 ALL 0.1034 LDC-MEAN-ALL-Macro TinkerBell_KB_XLING_4 0 0.1369 LDC-MEAN-ALL-Macro TinkerBell_KB_XLING_4 1 0.0209 LDC-MEAN-ALL-Macro TinkerBell_KB_XLING_4 ALL 0.1021 *ALL-Macro AP refer to mean of corresponding AP values. Scores based on P/R/F1; only k=1 justification allowed: Metric RunID Hop Prec Recall F1 SF-ALL-Micro TinkerBell_KB_XLING_4 0 0.1962 0.2397 0.2157 SF-ALL-Micro TinkerBell_KB_XLING_4 1 0.0199 0.3409 0.0377 SF-ALL-Micro TinkerBell_KB_XLING_4 ALL 0.0460 0.2692 0.0785 SF-ALL-Macro TinkerBell_KB_XLING_4 0 0.1606 0.1793 0.1402 SF-ALL-Macro TinkerBell_KB_XLING_4 1 0.1017 0.1506 0.1071 SF-ALL-Macro TinkerBell_KB_XLING_4 ALL 0.1348 0.1667 0.1257 LDC-MAX-ALL-Micro TinkerBell_KB_XLING_4 0 0.2084 0.2830 0.2400 LDC-MAX-ALL-Micro TinkerBell_KB_XLING_4 1 0.0258 0.3532 0.0481 LDC-MAX-ALL-Micro TinkerBell_KB_XLING_4 ALL 0.0612 0.3035 0.1019 LDC-MAX-ALL-Macro TinkerBell_KB_XLING_4 0 0.2090 0.2259 0.1813 LDC-MAX-ALL-Macro TinkerBell_KB_XLING_4 1 0.1124 0.1647 0.1181 LDC-MAX-ALL-Macro TinkerBell_KB_XLING_4 ALL 0.1670 0.1993 0.1538 LDC-MEAN-ALL-Macro TinkerBell_KB_XLING_4 0 0.1583 0.1766 0.1382 LDC-MEAN-ALL-Macro TinkerBell_KB_XLING_4 1 0.1006 0.1489 0.1059 LDC-MEAN-ALL-Macro TinkerBell_KB_XLING_4 ALL 0.1332 0.1646 0.1242 *ALL-Macro Prec, Recall and F1 refer to mean-precision, mean-recall and mean-F1. --------------------- Composite KB Evaluation (SF-SLOTS only): Scores based on confidence values and Average Precision (AP); k=3 justifications allowed: Metric RunID Hop AP SF-ALL-Macro TinkerBell_KB_XLING_4 0 0.1436 SF-ALL-Macro TinkerBell_KB_XLING_4 1 0.0284 SF-ALL-Macro TinkerBell_KB_XLING_4 ALL 0.0924 LDC-MEAN-ALL-Macro TinkerBell_KB_XLING_4 0 0.1433 LDC-MEAN-ALL-Macro TinkerBell_KB_XLING_4 1 0.0284 LDC-MEAN-ALL-Macro TinkerBell_KB_XLING_4 ALL 0.0922 (PRIMARY METRIC) *ALL-Macro AP refer to mean of corresponding AP values. Scores based on confidence values and Average Precision (AP); only k=1 justification allowed: Metric RunID Hop AP SF-ALL-Macro TinkerBell_KB_XLING_4 0 0.2443 SF-ALL-Macro TinkerBell_KB_XLING_4 1 0.0502 SF-ALL-Macro TinkerBell_KB_XLING_4 ALL 0.1699 LDC-MEAN-ALL-Macro TinkerBell_KB_XLING_4 0 0.2437 LDC-MEAN-ALL-Macro TinkerBell_KB_XLING_4 1 0.0502 LDC-MEAN-ALL-Macro TinkerBell_KB_XLING_4 ALL 0.1694 *ALL-Macro AP refer to mean of corresponding AP values. Scores based on P/R/F1; only k=1 justification allowed: Metric RunID Hop Prec Recall F1 SF-ALL-Micro TinkerBell_KB_XLING_4 0 0.2432 0.2212 0.2317 SF-ALL-Micro TinkerBell_KB_XLING_4 1 0.0437 0.1956 0.0715 SF-ALL-Micro TinkerBell_KB_XLING_4 ALL 0.1188 0.2148 0.1530 SF-ALL-Macro TinkerBell_KB_XLING_4 0 0.2183 0.2684 0.2134 SF-ALL-Macro TinkerBell_KB_XLING_4 1 0.1336 0.1553 0.1329 SF-ALL-Macro TinkerBell_KB_XLING_4 ALL 0.1771 0.2133 0.1742 LDC-MAX-ALL-Micro TinkerBell_KB_XLING_4 0 0.2988 0.2676 0.2823 LDC-MAX-ALL-Micro TinkerBell_KB_XLING_4 1 0.0573 0.2286 0.0916 LDC-MAX-ALL-Micro TinkerBell_KB_XLING_4 ALL 0.1540 0.2578 0.1928 LDC-MAX-ALL-Macro TinkerBell_KB_XLING_4 0 0.2926 0.3389 0.2835 LDC-MAX-ALL-Macro TinkerBell_KB_XLING_4 1 0.1547 0.1823 0.1546 LDC-MAX-ALL-Macro TinkerBell_KB_XLING_4 ALL 0.2259 0.2631 0.2211 LDC-MEAN-ALL-Macro TinkerBell_KB_XLING_4 0 0.2178 0.2674 0.2127 LDC-MEAN-ALL-Macro TinkerBell_KB_XLING_4 1 0.1336 0.1553 0.1329 LDC-MEAN-ALL-Macro TinkerBell_KB_XLING_4 ALL 0.1771 0.2132 0.1741 *ALL-Macro Prec, Recall and F1 refer to mean-precision, mean-recall and mean-F1. --------------------- Composite KB Evaluation (EVENT-SLOTS only): Scores based on confidence values and Average Precision (AP); k=3 justifications allowed: SUMMARY: This section provides summary of AP scores Metric RunID Hop AP SF-ALL-Macro TinkerBell_KB_XLING_4 0 0.0409 SF-ALL-Macro TinkerBell_KB_XLING_4 ALL 0.0409 LDC-MEAN-ALL-Macro TinkerBell_KB_XLING_4 0 0.0401 LDC-MEAN-ALL-Macro TinkerBell_KB_XLING_4 ALL 0.0401 (PRIMARY METRIC) *ALL-Macro AP refer to mean of corresponding AP values. Scores based on confidence values and Average Precision (AP); only k=1 justification allowed: SUMMARY: This section provides summary of AP scores Metric RunID Hop AP SF-ALL-Macro TinkerBell_KB_XLING_4 0 0.0563 SF-ALL-Macro TinkerBell_KB_XLING_4 ALL 0.0563 LDC-MEAN-ALL-Macro TinkerBell_KB_XLING_4 0 0.0554 LDC-MEAN-ALL-Macro TinkerBell_KB_XLING_4 ALL 0.0554 *ALL-Macro AP refer to mean of corresponding AP values. Scores based on P/R/F1; only k=1 justification allowed: Metric RunID Hop Prec Recall F1 SF-ALL-Micro TinkerBell_KB_XLING_4 0 0.3050 0.1589 0.2090 SF-ALL-Micro TinkerBell_KB_XLING_4 ALL 0.3050 0.1589 0.2090 SF-ALL-Macro TinkerBell_KB_XLING_4 0 0.1446 0.0804 0.0905 SF-ALL-Macro TinkerBell_KB_XLING_4 ALL 0.1446 0.0804 0.0905 LDC-MAX-ALL-Micro TinkerBell_KB_XLING_4 0 0.3145 0.1908 0.2375 LDC-MAX-ALL-Micro TinkerBell_KB_XLING_4 ALL 0.3145 0.1908 0.2375 LDC-MAX-ALL-Macro TinkerBell_KB_XLING_4 0 0.1916 0.1049 0.1183 LDC-MAX-ALL-Macro TinkerBell_KB_XLING_4 ALL 0.1916 0.1049 0.1183 LDC-MEAN-ALL-Macro TinkerBell_KB_XLING_4 0 0.1415 0.0790 0.0887 LDC-MEAN-ALL-Macro TinkerBell_KB_XLING_4 ALL 0.1415 0.0790 0.0887 *ALL-Macro Prec, Recall and F1 refer to mean-precision, mean-recall and mean-F1. --------------------- Composite KB Evaluation (SENTIMENT-SLOTS only): Scores based on confidence values and Average Precision (AP); k=3 justifications allowed: Metric RunID Hop AP SF-ALL-Macro TinkerBell_KB_XLING_4 0 0.0774 SF-ALL-Macro TinkerBell_KB_XLING_4 1 0.0066 SF-ALL-Macro TinkerBell_KB_XLING_4 ALL 0.0565 LDC-MEAN-ALL-Macro TinkerBell_KB_XLING_4 0 0.0751 LDC-MEAN-ALL-Macro TinkerBell_KB_XLING_4 1 0.0064 LDC-MEAN-ALL-Macro TinkerBell_KB_XLING_4 ALL 0.0548 (PRIMARY METRIC) *ALL-Macro AP refer to mean of corresponding AP values. Scores based on confidence values and Average Precision (AP); only k=1 justification allowed: Metric RunID Hop AP SF-ALL-Macro TinkerBell_KB_XLING_4 0 0.0713 SF-ALL-Macro TinkerBell_KB_XLING_4 1 0.0074 SF-ALL-Macro TinkerBell_KB_XLING_4 ALL 0.0543 LDC-MEAN-ALL-Macro TinkerBell_KB_XLING_4 0 0.0692 LDC-MEAN-ALL-Macro TinkerBell_KB_XLING_4 1 0.0071 LDC-MEAN-ALL-Macro TinkerBell_KB_XLING_4 ALL 0.0527 *ALL-Macro AP refer to mean of corresponding AP values. Scores based on P/R/F1; only k=1 justification allowed: Metric RunID Hop Prec Recall F1 SF-ALL-Micro TinkerBell_KB_XLING_4 0 0.1336 0.4300 0.2038 SF-ALL-Micro TinkerBell_KB_XLING_4 1 0.0119 0.3908 0.0231 SF-ALL-Micro TinkerBell_KB_XLING_4 ALL 0.0380 0.4196 0.0697 SF-ALL-Macro TinkerBell_KB_XLING_4 0 0.0818 0.1651 0.0829 SF-ALL-Macro TinkerBell_KB_XLING_4 1 0.1371 0.3015 0.1647 SF-ALL-Macro TinkerBell_KB_XLING_4 ALL 0.1043 0.2206 0.1163 LDC-MAX-ALL-Micro TinkerBell_KB_XLING_4 0 0.1287 0.4867 0.2036 LDC-MAX-ALL-Micro TinkerBell_KB_XLING_4 1 0.0113 0.3907 0.0219 LDC-MAX-ALL-Micro TinkerBell_KB_XLING_4 ALL 0.0383 0.4611 0.0708 LDC-MAX-ALL-Macro TinkerBell_KB_XLING_4 0 0.0894 0.2046 0.0950 LDC-MAX-ALL-Macro TinkerBell_KB_XLING_4 1 0.1386 0.3015 0.1661 LDC-MAX-ALL-Macro TinkerBell_KB_XLING_4 ALL 0.1093 0.2439 0.1239 LDC-MEAN-ALL-Macro TinkerBell_KB_XLING_4 0 0.0794 0.1601 0.0804 LDC-MEAN-ALL-Macro TinkerBell_KB_XLING_4 1 0.1337 0.2942 0.1607 LDC-MEAN-ALL-Macro TinkerBell_KB_XLING_4 ALL 0.1014 0.2145 0.1130 *ALL-Macro Prec, Recall and F1 refer to mean-precision, mean-recall and mean-F1. ************************************************************* Run ID: TinkerBell_KB_XLING_5 Did the run access the live Web during the evaluation window: No Did the run extract relations from the Cold Start source corpus: Yes Did the run generate meaningful confidence values: Yes Run number of the English CSKB system that is most closely configured to the English component of this run: NA Run number of the Spanish CSKB system that is most closely configured to the Spanish component of this run: NA Run number of the Chinese CSKB system that is most closely configured to the Chinese component of this run: NA Did the run attempt the full Entity Discovery and Linking task: Yes Did the run attempt the full Event Nugget Detection and Coreference task: Yes Did the run include SF relations: Yes Did the run attempt the Event Argument Extraction and Linking task: Yes Did the run include Sentiment relations: Yes --------------------- Composite KB Evaluation (All slot types): Scores based on confidence values and Average Precision (AP); k=3 justifications allowed: Metric RunID Hop AP SF-ALL-Macro TinkerBell_KB_XLING_5 0 0.0582 SF-ALL-Macro TinkerBell_KB_XLING_5 1 0.0071 SF-ALL-Macro TinkerBell_KB_XLING_5 ALL 0.0390 LDC-MEAN-ALL-Macro TinkerBell_KB_XLING_5 0 0.0576 LDC-MEAN-ALL-Macro TinkerBell_KB_XLING_5 1 0.0069 LDC-MEAN-ALL-Macro TinkerBell_KB_XLING_5 ALL 0.0386 (PRIMARY METRIC) *ALL-Macro AP refer to mean of corresponding AP values. Scores based on confidence values and Average Precision (AP); only k=1 justification allowed: Metric RunID Hop AP SF-ALL-Macro TinkerBell_KB_XLING_5 0 0.0799 SF-ALL-Macro TinkerBell_KB_XLING_5 1 0.0112 SF-ALL-Macro TinkerBell_KB_XLING_5 ALL 0.0560 LDC-MEAN-ALL-Macro TinkerBell_KB_XLING_5 0 0.0793 LDC-MEAN-ALL-Macro TinkerBell_KB_XLING_5 1 0.0110 LDC-MEAN-ALL-Macro TinkerBell_KB_XLING_5 ALL 0.0556 *ALL-Macro AP refer to mean of corresponding AP values. Scores based on P/R/F1; only k=1 justification allowed: Metric RunID Hop Prec Recall F1 SF-ALL-Micro TinkerBell_KB_XLING_5 0 0.1767 0.1094 0.1351 SF-ALL-Micro TinkerBell_KB_XLING_5 1 0.0173 0.1060 0.0298 SF-ALL-Micro TinkerBell_KB_XLING_5 ALL 0.0488 0.1084 0.0673 SF-ALL-Macro TinkerBell_KB_XLING_5 0 0.0957 0.0980 0.0800 SF-ALL-Macro TinkerBell_KB_XLING_5 1 0.0531 0.0730 0.0536 SF-ALL-Macro TinkerBell_KB_XLING_5 ALL 0.0771 0.0871 0.0685 LDC-MAX-ALL-Micro TinkerBell_KB_XLING_5 0 0.2426 0.2646 0.2531 LDC-MAX-ALL-Micro TinkerBell_KB_XLING_5 1 0.0501 0.3154 0.0864 LDC-MAX-ALL-Micro TinkerBell_KB_XLING_5 ALL 0.1070 0.2794 0.1547 LDC-MAX-ALL-Macro TinkerBell_KB_XLING_5 0 0.2305 0.2168 0.1908 LDC-MAX-ALL-Macro TinkerBell_KB_XLING_5 1 0.0967 0.1308 0.0970 LDC-MAX-ALL-Macro TinkerBell_KB_XLING_5 ALL 0.1722 0.1793 0.1500 LDC-MEAN-ALL-Macro TinkerBell_KB_XLING_5 0 0.0949 0.0971 0.0794 LDC-MEAN-ALL-Macro TinkerBell_KB_XLING_5 1 0.0525 0.0721 0.0530 LDC-MEAN-ALL-Macro TinkerBell_KB_XLING_5 ALL 0.0765 0.0862 0.0679 *ALL-Macro Prec, Recall and F1 refer to mean-precision, mean-recall and mean-F1. --------------------- Composite KB Evaluation (SF-SLOTS only): Scores based on confidence values and Average Precision (AP); k=3 justifications allowed: Metric RunID Hop AP SF-ALL-Macro TinkerBell_KB_XLING_5 0 0.1063 SF-ALL-Macro TinkerBell_KB_XLING_5 1 0.0169 SF-ALL-Macro TinkerBell_KB_XLING_5 ALL 0.0657 LDC-MEAN-ALL-Macro TinkerBell_KB_XLING_5 0 0.1064 LDC-MEAN-ALL-Macro TinkerBell_KB_XLING_5 1 0.0169 LDC-MEAN-ALL-Macro TinkerBell_KB_XLING_5 ALL 0.0658 (PRIMARY METRIC) *ALL-Macro AP refer to mean of corresponding AP values. Scores based on confidence values and Average Precision (AP); only k=1 justification allowed: Metric RunID Hop AP SF-ALL-Macro TinkerBell_KB_XLING_5 0 0.1555 SF-ALL-Macro TinkerBell_KB_XLING_5 1 0.0293 SF-ALL-Macro TinkerBell_KB_XLING_5 ALL 0.1023 LDC-MEAN-ALL-Macro TinkerBell_KB_XLING_5 0 0.1558 LDC-MEAN-ALL-Macro TinkerBell_KB_XLING_5 1 0.0293 LDC-MEAN-ALL-Macro TinkerBell_KB_XLING_5 ALL 0.1026 *ALL-Macro AP refer to mean of corresponding AP values. Scores based on P/R/F1; only k=1 justification allowed: Metric RunID Hop Prec Recall F1 SF-ALL-Micro TinkerBell_KB_XLING_5 0 0.2070 0.1110 0.1445 SF-ALL-Micro TinkerBell_KB_XLING_5 1 0.0431 0.0825 0.0566 SF-ALL-Micro TinkerBell_KB_XLING_5 ALL 0.1175 0.1039 0.1102 SF-ALL-Macro TinkerBell_KB_XLING_5 0 0.1629 0.1687 0.1444 SF-ALL-Macro TinkerBell_KB_XLING_5 1 0.0740 0.0768 0.0690 SF-ALL-Macro TinkerBell_KB_XLING_5 ALL 0.1196 0.1240 0.1077 LDC-MAX-ALL-Micro TinkerBell_KB_XLING_5 0 0.3511 0.2313 0.2789 LDC-MAX-ALL-Micro TinkerBell_KB_XLING_5 1 0.1888 0.1492 0.1667 LDC-MAX-ALL-Micro TinkerBell_KB_XLING_5 ALL 0.3045 0.2107 0.2491 LDC-MAX-ALL-Macro TinkerBell_KB_XLING_5 0 0.3463 0.3270 0.3112 LDC-MAX-ALL-Macro TinkerBell_KB_XLING_5 1 0.1264 0.1263 0.1156 LDC-MAX-ALL-Macro TinkerBell_KB_XLING_5 ALL 0.2399 0.2299 0.2165 LDC-MEAN-ALL-Macro TinkerBell_KB_XLING_5 0 0.1634 0.1689 0.1449 LDC-MEAN-ALL-Macro TinkerBell_KB_XLING_5 1 0.0740 0.0768 0.0690 LDC-MEAN-ALL-Macro TinkerBell_KB_XLING_5 ALL 0.1201 0.1243 0.1082 *ALL-Macro Prec, Recall and F1 refer to mean-precision, mean-recall and mean-F1. --------------------- Composite KB Evaluation (EVENT-SLOTS only): Scores based on confidence values and Average Precision (AP); k=3 justifications allowed: SUMMARY: This section provides summary of AP scores Metric RunID Hop AP SF-ALL-Macro TinkerBell_KB_XLING_5 0 0.0127 SF-ALL-Macro TinkerBell_KB_XLING_5 ALL 0.0127 LDC-MEAN-ALL-Macro TinkerBell_KB_XLING_5 0 0.0126 LDC-MEAN-ALL-Macro TinkerBell_KB_XLING_5 ALL 0.0126 (PRIMARY METRIC) *ALL-Macro AP refer to mean of corresponding AP values. Scores based on confidence values and Average Precision (AP); only k=1 justification allowed: SUMMARY: This section provides summary of AP scores Metric RunID Hop AP SF-ALL-Macro TinkerBell_KB_XLING_5 0 0.0191 SF-ALL-Macro TinkerBell_KB_XLING_5 ALL 0.0191 LDC-MEAN-ALL-Macro TinkerBell_KB_XLING_5 0 0.0192 LDC-MEAN-ALL-Macro TinkerBell_KB_XLING_5 ALL 0.0192 *ALL-Macro AP refer to mean of corresponding AP values. Scores based on P/R/F1; only k=1 justification allowed: Metric RunID Hop Prec Recall F1 SF-ALL-Micro TinkerBell_KB_XLING_5 0 0.3096 0.0540 0.0920 SF-ALL-Micro TinkerBell_KB_XLING_5 ALL 0.3096 0.0540 0.0920 SF-ALL-Macro TinkerBell_KB_XLING_5 0 0.0530 0.0270 0.0300 SF-ALL-Macro TinkerBell_KB_XLING_5 ALL 0.0530 0.0270 0.0300 LDC-MAX-ALL-Micro TinkerBell_KB_XLING_5 0 0.4317 0.1789 0.2530 LDC-MAX-ALL-Micro TinkerBell_KB_XLING_5 ALL 0.4317 0.1789 0.2530 LDC-MAX-ALL-Macro TinkerBell_KB_XLING_5 0 0.1876 0.0951 0.1096 LDC-MAX-ALL-Macro TinkerBell_KB_XLING_5 ALL 0.1876 0.0951 0.1096 LDC-MEAN-ALL-Macro TinkerBell_KB_XLING_5 0 0.0522 0.0269 0.0298 LDC-MEAN-ALL-Macro TinkerBell_KB_XLING_5 ALL 0.0522 0.0269 0.0298 *ALL-Macro Prec, Recall and F1 refer to mean-precision, mean-recall and mean-F1. --------------------- Composite KB Evaluation (SENTIMENT-SLOTS only): Scores based on confidence values and Average Precision (AP); k=3 justifications allowed: Metric RunID Hop AP SF-ALL-Macro TinkerBell_KB_XLING_5 0 0.0389 SF-ALL-Macro TinkerBell_KB_XLING_5 1 0.0026 SF-ALL-Macro TinkerBell_KB_XLING_5 ALL 0.0300 LDC-MEAN-ALL-Macro TinkerBell_KB_XLING_5 0 0.0378 LDC-MEAN-ALL-Macro TinkerBell_KB_XLING_5 1 0.0025 LDC-MEAN-ALL-Macro TinkerBell_KB_XLING_5 ALL 0.0291 (PRIMARY METRIC) *ALL-Macro AP refer to mean of corresponding AP values. Scores based on confidence values and Average Precision (AP); only k=1 justification allowed: Metric RunID Hop AP SF-ALL-Macro TinkerBell_KB_XLING_5 0 0.0344 SF-ALL-Macro TinkerBell_KB_XLING_5 1 0.0027 SF-ALL-Macro TinkerBell_KB_XLING_5 ALL 0.0277 LDC-MEAN-ALL-Macro TinkerBell_KB_XLING_5 0 0.0334 LDC-MEAN-ALL-Macro TinkerBell_KB_XLING_5 1 0.0026 LDC-MEAN-ALL-Macro TinkerBell_KB_XLING_5 ALL 0.0268 *ALL-Macro AP refer to mean of corresponding AP values. Scores based on P/R/F1; only k=1 justification allowed: Metric RunID Hop Prec Recall F1 SF-ALL-Micro TinkerBell_KB_XLING_5 0 0.1277 0.2072 0.1580 SF-ALL-Micro TinkerBell_KB_XLING_5 1 0.0131 0.2103 0.0247 SF-ALL-Micro TinkerBell_KB_XLING_5 ALL 0.0383 0.2080 0.0648 SF-ALL-Macro TinkerBell_KB_XLING_5 0 0.0390 0.0761 0.0386 SF-ALL-Macro TinkerBell_KB_XLING_5 1 0.0692 0.1554 0.0833 SF-ALL-Macro TinkerBell_KB_XLING_5 ALL 0.0513 0.1084 0.0568 LDC-MAX-ALL-Micro TinkerBell_KB_XLING_5 0 0.1536 0.4964 0.2346 LDC-MAX-ALL-Micro TinkerBell_KB_XLING_5 1 0.0132 0.3841 0.0255 LDC-MAX-ALL-Micro TinkerBell_KB_XLING_5 ALL 0.0459 0.4664 0.0837 LDC-MAX-ALL-Macro TinkerBell_KB_XLING_5 0 0.0943 0.2011 0.0997 LDC-MAX-ALL-Macro TinkerBell_KB_XLING_5 1 0.1361 0.2893 0.1620 LDC-MAX-ALL-Macro TinkerBell_KB_XLING_5 ALL 0.1113 0.2369 0.1250 LDC-MEAN-ALL-Macro TinkerBell_KB_XLING_5 0 0.0378 0.0738 0.0375 LDC-MEAN-ALL-Macro TinkerBell_KB_XLING_5 1 0.0676 0.1516 0.0813 LDC-MEAN-ALL-Macro TinkerBell_KB_XLING_5 ALL 0.0499 0.1054 0.0553 *ALL-Macro Prec, Recall and F1 refer to mean-precision, mean-recall and mean-F1.