===================================================================== TAC KBP 2017 CHINESE KB CONSTRUCTION: COMPOSITE KB EVALUATION RESULTS ===================================================================== Team ID: TinkerBell Organization: RPI, UIUC, Stanford, Columbia, Cornell, JHU, UPenn ************************************************************* Run ID: TinkerBell_KB_CMN_1 Did the run access the live Web during the evaluation window: No Did the run extract relations from the Cold Start source corpus: Yes Did the run generate meaningful confidence values: Yes Did the run attempt the full Entity Discovery and Linking task: Yes Did the run attempt the full Event Nugget Detection and Coreference task: Yes Did the run include SF relations: Yes Did the run attempt the Event Argument Extraction and Linking task: Yes Did the run include Sentiment relations: Yes --------------------- Composite KB Evaluation (All slot types): Scores based on confidence values and Average Precision (AP); k=3 justifications allowed: Metric RunID Hop AP SF-ALL-Macro TinkerBell_KB_CMN_1 0 0.1390 SF-ALL-Macro TinkerBell_KB_CMN_1 1 0.0192 SF-ALL-Macro TinkerBell_KB_CMN_1 ALL 0.1160 LDC-MEAN-ALL-Macro TinkerBell_KB_CMN_1 0 0.1633 LDC-MEAN-ALL-Macro TinkerBell_KB_CMN_1 1 0.0199 LDC-MEAN-ALL-Macro TinkerBell_KB_CMN_1 ALL 0.1403 (PRIMARY METRIC) *ALL-Macro AP refer to mean of corresponding AP values. Scores based on confidence values and Average Precision (AP); only k=1 justification allowed: Metric RunID Hop AP SF-ALL-Macro TinkerBell_KB_CMN_1 0 0.1838 SF-ALL-Macro TinkerBell_KB_CMN_1 1 0.0278 SF-ALL-Macro TinkerBell_KB_CMN_1 ALL 0.1538 LDC-MEAN-ALL-Macro TinkerBell_KB_CMN_1 0 0.2057 LDC-MEAN-ALL-Macro TinkerBell_KB_CMN_1 1 0.0244 LDC-MEAN-ALL-Macro TinkerBell_KB_CMN_1 ALL 0.1784 *ALL-Macro AP refer to mean of corresponding AP values. Scores based on P/R/F1; only k=1 justification allowed: Metric RunID Hop Prec Recall F1 SF-ALL-Micro TinkerBell_KB_CMN_1 0 0.3020 0.3516 0.3249 SF-ALL-Micro TinkerBell_KB_CMN_1 1 0.0321 0.4558 0.0600 SF-ALL-Micro TinkerBell_KB_CMN_1 ALL 0.0736 0.3840 0.1235 SF-ALL-Macro TinkerBell_KB_CMN_1 0 0.2181 0.2266 0.2044 SF-ALL-Macro TinkerBell_KB_CMN_1 1 0.1256 0.2442 0.1415 SF-ALL-Macro TinkerBell_KB_CMN_1 ALL 0.1836 0.2331 0.1810 LDC-MAX-ALL-Micro TinkerBell_KB_CMN_1 0 0.2946 0.3459 0.3182 LDC-MAX-ALL-Micro TinkerBell_KB_CMN_1 1 0.0325 0.4958 0.0610 LDC-MAX-ALL-Micro TinkerBell_KB_CMN_1 ALL 0.0697 0.3935 0.1184 LDC-MAX-ALL-Macro TinkerBell_KB_CMN_1 0 0.2616 0.2672 0.2415 LDC-MAX-ALL-Macro TinkerBell_KB_CMN_1 1 0.1093 0.1944 0.1221 LDC-MAX-ALL-Macro TinkerBell_KB_CMN_1 ALL 0.2084 0.2418 0.1997 LDC-MEAN-ALL-Macro TinkerBell_KB_CMN_1 0 0.2482 0.2565 0.2299 LDC-MEAN-ALL-Macro TinkerBell_KB_CMN_1 1 0.1055 0.1909 0.1185 LDC-MEAN-ALL-Macro TinkerBell_KB_CMN_1 ALL 0.1983 0.2336 0.1909 *ALL-Macro Prec, Recall and F1 refer to mean-precision, mean-recall and mean-F1. --------------------- Composite KB Evaluation (SF-SLOTS only): Scores based on confidence values and Average Precision (AP); k=3 justifications allowed: Metric RunID Hop AP SF-ALL-Macro TinkerBell_KB_CMN_1 0 0.1261 SF-ALL-Macro TinkerBell_KB_CMN_1 1 0.0249 SF-ALL-Macro TinkerBell_KB_CMN_1 ALL 0.0885 LDC-MEAN-ALL-Macro TinkerBell_KB_CMN_1 0 0.1401 LDC-MEAN-ALL-Macro TinkerBell_KB_CMN_1 1 0.0193 LDC-MEAN-ALL-Macro TinkerBell_KB_CMN_1 ALL 0.1035 (PRIMARY METRIC) *ALL-Macro AP refer to mean of corresponding AP values. Scores based on confidence values and Average Precision (AP); only k=1 justification allowed: Metric RunID Hop AP SF-ALL-Macro TinkerBell_KB_CMN_1 0 0.1929 SF-ALL-Macro TinkerBell_KB_CMN_1 1 0.0450 SF-ALL-Macro TinkerBell_KB_CMN_1 ALL 0.1393 LDC-MEAN-ALL-Macro TinkerBell_KB_CMN_1 0 0.2042 LDC-MEAN-ALL-Macro TinkerBell_KB_CMN_1 1 0.0331 LDC-MEAN-ALL-Macro TinkerBell_KB_CMN_1 ALL 0.1580 *ALL-Macro AP refer to mean of corresponding AP values. Scores based on P/R/F1; only k=1 justification allowed: Metric RunID Hop Prec Recall F1 SF-ALL-Micro TinkerBell_KB_CMN_1 0 0.4775 0.1707 0.2515 SF-ALL-Micro TinkerBell_KB_CMN_1 1 0.2558 0.1215 0.1648 SF-ALL-Micro TinkerBell_KB_CMN_1 ALL 0.4156 0.1596 0.2306 SF-ALL-Macro TinkerBell_KB_CMN_1 0 0.2228 0.1985 0.2024 SF-ALL-Macro TinkerBell_KB_CMN_1 1 0.0732 0.0748 0.0669 SF-ALL-Macro TinkerBell_KB_CMN_1 ALL 0.1719 0.1564 0.1563 LDC-MAX-ALL-Micro TinkerBell_KB_CMN_1 0 0.4852 0.1814 0.2641 LDC-MAX-ALL-Micro TinkerBell_KB_CMN_1 1 0.2500 0.1316 0.1724 LDC-MAX-ALL-Micro TinkerBell_KB_CMN_1 ALL 0.4236 0.1714 0.2440 LDC-MAX-ALL-Macro TinkerBell_KB_CMN_1 0 0.2544 0.2245 0.2299 LDC-MAX-ALL-Macro TinkerBell_KB_CMN_1 1 0.0737 0.0831 0.0731 LDC-MAX-ALL-Macro TinkerBell_KB_CMN_1 ALL 0.1897 0.1739 0.1738 LDC-MEAN-ALL-Macro TinkerBell_KB_CMN_1 0 0.2425 0.2121 0.2172 LDC-MEAN-ALL-Macro TinkerBell_KB_CMN_1 1 0.0661 0.0762 0.0660 LDC-MEAN-ALL-Macro TinkerBell_KB_CMN_1 ALL 0.1794 0.1634 0.1631 *ALL-Macro Prec, Recall and F1 refer to mean-precision, mean-recall and mean-F1. --------------------- Composite KB Evaluation (EVENT-SLOTS only): Scores based on confidence values and Average Precision (AP); k=3 justifications allowed: SUMMARY: This section provides summary of AP scores Metric RunID Hop AP SF-ALL-Macro TinkerBell_KB_CMN_1 0 0.1486 SF-ALL-Macro TinkerBell_KB_CMN_1 ALL 0.1486 LDC-MEAN-ALL-Macro TinkerBell_KB_CMN_1 0 0.1908 LDC-MEAN-ALL-Macro TinkerBell_KB_CMN_1 ALL 0.1908 (PRIMARY METRIC) *ALL-Macro AP refer to mean of corresponding AP values. Scores based on confidence values and Average Precision (AP); only k=1 justification allowed: SUMMARY: This section provides summary of AP scores Metric RunID Hop AP SF-ALL-Macro TinkerBell_KB_CMN_1 0 0.1969 SF-ALL-Macro TinkerBell_KB_CMN_1 ALL 0.1969 LDC-MEAN-ALL-Macro TinkerBell_KB_CMN_1 0 0.2382 LDC-MEAN-ALL-Macro TinkerBell_KB_CMN_1 ALL 0.2382 *ALL-Macro AP refer to mean of corresponding AP values. Scores based on P/R/F1; only k=1 justification allowed: Metric RunID Hop Prec Recall F1 SF-ALL-Micro TinkerBell_KB_CMN_1 0 0.3557 0.3506 0.3532 SF-ALL-Micro TinkerBell_KB_CMN_1 ALL 0.3557 0.3506 0.3532 SF-ALL-Macro TinkerBell_KB_CMN_1 0 0.2499 0.2667 0.2358 SF-ALL-Macro TinkerBell_KB_CMN_1 ALL 0.2499 0.2667 0.2358 LDC-MAX-ALL-Micro TinkerBell_KB_CMN_1 0 0.3768 0.3822 0.3795 LDC-MAX-ALL-Micro TinkerBell_KB_CMN_1 ALL 0.3768 0.3822 0.3795 LDC-MAX-ALL-Macro TinkerBell_KB_CMN_1 0 0.3253 0.3230 0.2968 LDC-MAX-ALL-Macro TinkerBell_KB_CMN_1 ALL 0.3253 0.3230 0.2968 LDC-MEAN-ALL-Macro TinkerBell_KB_CMN_1 0 0.3023 0.3091 0.2807 LDC-MEAN-ALL-Macro TinkerBell_KB_CMN_1 ALL 0.3023 0.3091 0.2807 *ALL-Macro Prec, Recall and F1 refer to mean-precision, mean-recall and mean-F1. --------------------- Composite KB Evaluation (SENTIMENT-SLOTS only): Scores based on confidence values and Average Precision (AP); k=3 justifications allowed: Metric RunID Hop AP SF-ALL-Macro TinkerBell_KB_CMN_1 0 0.1552 SF-ALL-Macro TinkerBell_KB_CMN_1 1 0.0161 SF-ALL-Macro TinkerBell_KB_CMN_1 ALL 0.1287 LDC-MEAN-ALL-Macro TinkerBell_KB_CMN_1 0 0.1763 LDC-MEAN-ALL-Macro TinkerBell_KB_CMN_1 1 0.0199 LDC-MEAN-ALL-Macro TinkerBell_KB_CMN_1 ALL 0.1478 (PRIMARY METRIC) *ALL-Macro AP refer to mean of corresponding AP values. Scores based on confidence values and Average Precision (AP); only k=1 justification allowed: Metric RunID Hop AP SF-ALL-Macro TinkerBell_KB_CMN_1 0 0.1358 SF-ALL-Macro TinkerBell_KB_CMN_1 1 0.0165 SF-ALL-Macro TinkerBell_KB_CMN_1 ALL 0.1130 LDC-MEAN-ALL-Macro TinkerBell_KB_CMN_1 0 0.1492 LDC-MEAN-ALL-Macro TinkerBell_KB_CMN_1 1 0.0212 LDC-MEAN-ALL-Macro TinkerBell_KB_CMN_1 ALL 0.1236 *ALL-Macro AP refer to mean of corresponding AP values. Scores based on P/R/F1; only k=1 justification allowed: Metric RunID Hop Prec Recall F1 SF-ALL-Micro TinkerBell_KB_CMN_1 0 0.2409 0.6040 0.3444 SF-ALL-Micro TinkerBell_KB_CMN_1 1 0.0222 0.5316 0.0426 SF-ALL-Micro TinkerBell_KB_CMN_1 ALL 0.0546 0.5768 0.0998 SF-ALL-Macro TinkerBell_KB_CMN_1 0 0.1474 0.2268 0.1520 SF-ALL-Macro TinkerBell_KB_CMN_1 1 0.2389 0.5471 0.2870 SF-ALL-Macro TinkerBell_KB_CMN_1 ALL 0.1974 0.4020 0.2259 LDC-MAX-ALL-Micro TinkerBell_KB_CMN_1 0 0.2058 0.6121 0.3080 LDC-MAX-ALL-Micro TinkerBell_KB_CMN_1 1 0.0158 0.5000 0.0306 LDC-MAX-ALL-Micro TinkerBell_KB_CMN_1 ALL 0.0481 0.5769 0.0889 LDC-MAX-ALL-Macro TinkerBell_KB_CMN_1 0 0.1631 0.2812 0.1702 LDC-MAX-ALL-Macro TinkerBell_KB_CMN_1 1 0.2223 0.4934 0.2657 LDC-MAX-ALL-Macro TinkerBell_KB_CMN_1 ALL 0.1901 0.3781 0.2138 LDC-MEAN-ALL-Macro TinkerBell_KB_CMN_1 0 0.1631 0.2812 0.1702 LDC-MEAN-ALL-Macro TinkerBell_KB_CMN_1 1 0.2223 0.4934 0.2657 LDC-MEAN-ALL-Macro TinkerBell_KB_CMN_1 ALL 0.1901 0.3781 0.2138 *ALL-Macro Prec, Recall and F1 refer to mean-precision, mean-recall and mean-F1. ************************************************************* Run ID: TinkerBell_KB_CMN_2 Did the run access the live Web during the evaluation window: No Did the run extract relations from the Cold Start source corpus: Yes Did the run generate meaningful confidence values: Yes Did the run attempt the full Entity Discovery and Linking task: Yes Did the run attempt the full Event Nugget Detection and Coreference task: Yes Did the run include SF relations: Yes Did the run attempt the Event Argument Extraction and Linking task: Yes Did the run include Sentiment relations: Yes --------------------- Composite KB Evaluation (All slot types): Scores based on confidence values and Average Precision (AP); k=3 justifications allowed: Metric RunID Hop AP SF-ALL-Macro TinkerBell_KB_CMN_2 0 0.1276 SF-ALL-Macro TinkerBell_KB_CMN_2 1 0.0174 SF-ALL-Macro TinkerBell_KB_CMN_2 ALL 0.1094 LDC-MEAN-ALL-Macro TinkerBell_KB_CMN_2 0 0.1477 LDC-MEAN-ALL-Macro TinkerBell_KB_CMN_2 1 0.0179 LDC-MEAN-ALL-Macro TinkerBell_KB_CMN_2 ALL 0.1311 (PRIMARY METRIC) *ALL-Macro AP refer to mean of corresponding AP values. Scores based on confidence values and Average Precision (AP); only k=1 justification allowed: Metric RunID Hop AP SF-ALL-Macro TinkerBell_KB_CMN_2 0 0.1842 SF-ALL-Macro TinkerBell_KB_CMN_2 1 0.0296 SF-ALL-Macro TinkerBell_KB_CMN_2 ALL 0.1549 LDC-MEAN-ALL-Macro TinkerBell_KB_CMN_2 0 0.2064 LDC-MEAN-ALL-Macro TinkerBell_KB_CMN_2 1 0.0274 LDC-MEAN-ALL-Macro TinkerBell_KB_CMN_2 ALL 0.1801 *ALL-Macro AP refer to mean of corresponding AP values. Scores based on P/R/F1; only k=1 justification allowed: Metric RunID Hop Prec Recall F1 SF-ALL-Micro TinkerBell_KB_CMN_2 0 0.3064 0.3516 0.3274 SF-ALL-Micro TinkerBell_KB_CMN_2 1 0.0455 0.4558 0.0827 SF-ALL-Micro TinkerBell_KB_CMN_2 ALL 0.0983 0.3840 0.1565 SF-ALL-Macro TinkerBell_KB_CMN_2 0 0.2190 0.2266 0.2047 SF-ALL-Macro TinkerBell_KB_CMN_2 1 0.1268 0.2442 0.1423 SF-ALL-Macro TinkerBell_KB_CMN_2 ALL 0.1847 0.2331 0.1815 LDC-MAX-ALL-Micro TinkerBell_KB_CMN_2 0 0.3015 0.3459 0.3222 LDC-MAX-ALL-Micro TinkerBell_KB_CMN_2 1 0.0559 0.4958 0.1005 LDC-MAX-ALL-Micro TinkerBell_KB_CMN_2 ALL 0.1094 0.3935 0.1712 LDC-MAX-ALL-Macro TinkerBell_KB_CMN_2 0 0.2629 0.2672 0.2419 LDC-MAX-ALL-Macro TinkerBell_KB_CMN_2 1 0.1113 0.1944 0.1234 LDC-MAX-ALL-Macro TinkerBell_KB_CMN_2 ALL 0.2099 0.2418 0.2005 LDC-MEAN-ALL-Macro TinkerBell_KB_CMN_2 0 0.2495 0.2565 0.2303 LDC-MEAN-ALL-Macro TinkerBell_KB_CMN_2 1 0.1075 0.1909 0.1198 LDC-MEAN-ALL-Macro TinkerBell_KB_CMN_2 ALL 0.1998 0.2336 0.1917 *ALL-Macro Prec, Recall and F1 refer to mean-precision, mean-recall and mean-F1. --------------------- Composite KB Evaluation (SF-SLOTS only): Scores based on confidence values and Average Precision (AP); k=3 justifications allowed: Metric RunID Hop AP SF-ALL-Macro TinkerBell_KB_CMN_2 0 0.1023 SF-ALL-Macro TinkerBell_KB_CMN_2 1 0.0213 SF-ALL-Macro TinkerBell_KB_CMN_2 ALL 0.0746 LDC-MEAN-ALL-Macro TinkerBell_KB_CMN_2 0 0.1083 LDC-MEAN-ALL-Macro TinkerBell_KB_CMN_2 1 0.0154 LDC-MEAN-ALL-Macro TinkerBell_KB_CMN_2 ALL 0.0847 (PRIMARY METRIC) *ALL-Macro AP refer to mean of corresponding AP values. Scores based on confidence values and Average Precision (AP); only k=1 justification allowed: Metric RunID Hop AP SF-ALL-Macro TinkerBell_KB_CMN_2 0 0.1939 SF-ALL-Macro TinkerBell_KB_CMN_2 1 0.0450 SF-ALL-Macro TinkerBell_KB_CMN_2 ALL 0.1415 LDC-MEAN-ALL-Macro TinkerBell_KB_CMN_2 0 0.2056 LDC-MEAN-ALL-Macro TinkerBell_KB_CMN_2 1 0.0331 LDC-MEAN-ALL-Macro TinkerBell_KB_CMN_2 ALL 0.1614 *ALL-Macro AP refer to mean of corresponding AP values. Scores based on P/R/F1; only k=1 justification allowed: Metric RunID Hop Prec Recall F1 SF-ALL-Micro TinkerBell_KB_CMN_2 0 0.5436 0.1707 0.2598 SF-ALL-Micro TinkerBell_KB_CMN_2 1 0.3056 0.1215 0.1739 SF-ALL-Micro TinkerBell_KB_CMN_2 ALL 0.4794 0.1596 0.2395 SF-ALL-Macro TinkerBell_KB_CMN_2 0 0.2247 0.1985 0.2030 SF-ALL-Macro TinkerBell_KB_CMN_2 1 0.0761 0.0748 0.0689 SF-ALL-Macro TinkerBell_KB_CMN_2 ALL 0.1741 0.1564 0.1573 LDC-MAX-ALL-Micro TinkerBell_KB_CMN_2 0 0.5775 0.1814 0.2761 LDC-MAX-ALL-Micro TinkerBell_KB_CMN_2 1 0.3125 0.1316 0.1852 LDC-MAX-ALL-Micro TinkerBell_KB_CMN_2 ALL 0.5105 0.1714 0.2566 LDC-MAX-ALL-Macro TinkerBell_KB_CMN_2 0 0.2570 0.2245 0.2308 LDC-MAX-ALL-Macro TinkerBell_KB_CMN_2 1 0.0776 0.0831 0.0756 LDC-MAX-ALL-Macro TinkerBell_KB_CMN_2 ALL 0.1928 0.1739 0.1752 LDC-MEAN-ALL-Macro TinkerBell_KB_CMN_2 0 0.2452 0.2121 0.2181 LDC-MEAN-ALL-Macro TinkerBell_KB_CMN_2 1 0.0700 0.0762 0.0685 LDC-MEAN-ALL-Macro TinkerBell_KB_CMN_2 ALL 0.1824 0.1634 0.1645 *ALL-Macro Prec, Recall and F1 refer to mean-precision, mean-recall and mean-F1. --------------------- Composite KB Evaluation (EVENT-SLOTS only): Scores based on confidence values and Average Precision (AP); k=3 justifications allowed: SUMMARY: This section provides summary of AP scores Metric RunID Hop AP SF-ALL-Macro TinkerBell_KB_CMN_2 0 0.1486 SF-ALL-Macro TinkerBell_KB_CMN_2 ALL 0.1486 LDC-MEAN-ALL-Macro TinkerBell_KB_CMN_2 0 0.1908 LDC-MEAN-ALL-Macro TinkerBell_KB_CMN_2 ALL 0.1908 (PRIMARY METRIC) *ALL-Macro AP refer to mean of corresponding AP values. Scores based on confidence values and Average Precision (AP); only k=1 justification allowed: SUMMARY: This section provides summary of AP scores Metric RunID Hop AP SF-ALL-Macro TinkerBell_KB_CMN_2 0 0.1969 SF-ALL-Macro TinkerBell_KB_CMN_2 ALL 0.1969 LDC-MEAN-ALL-Macro TinkerBell_KB_CMN_2 0 0.2382 LDC-MEAN-ALL-Macro TinkerBell_KB_CMN_2 ALL 0.2382 *ALL-Macro AP refer to mean of corresponding AP values. Scores based on P/R/F1; only k=1 justification allowed: Metric RunID Hop Prec Recall F1 SF-ALL-Micro TinkerBell_KB_CMN_2 0 0.3557 0.3506 0.3532 SF-ALL-Micro TinkerBell_KB_CMN_2 ALL 0.3557 0.3506 0.3532 SF-ALL-Macro TinkerBell_KB_CMN_2 0 0.2499 0.2667 0.2358 SF-ALL-Macro TinkerBell_KB_CMN_2 ALL 0.2499 0.2667 0.2358 LDC-MAX-ALL-Micro TinkerBell_KB_CMN_2 0 0.3778 0.3822 0.3800 LDC-MAX-ALL-Micro TinkerBell_KB_CMN_2 ALL 0.3778 0.3822 0.3800 LDC-MAX-ALL-Macro TinkerBell_KB_CMN_2 0 0.3253 0.3230 0.2968 LDC-MAX-ALL-Macro TinkerBell_KB_CMN_2 ALL 0.3253 0.3230 0.2968 LDC-MEAN-ALL-Macro TinkerBell_KB_CMN_2 0 0.3023 0.3091 0.2807 LDC-MEAN-ALL-Macro TinkerBell_KB_CMN_2 ALL 0.3023 0.3091 0.2807 *ALL-Macro Prec, Recall and F1 refer to mean-precision, mean-recall and mean-F1. --------------------- Composite KB Evaluation (SENTIMENT-SLOTS only): Scores based on confidence values and Average Precision (AP); k=3 justifications allowed: Metric RunID Hop AP SF-ALL-Macro TinkerBell_KB_CMN_2 0 0.1552 SF-ALL-Macro TinkerBell_KB_CMN_2 1 0.0161 SF-ALL-Macro TinkerBell_KB_CMN_2 ALL 0.1287 LDC-MEAN-ALL-Macro TinkerBell_KB_CMN_2 0 0.1763 LDC-MEAN-ALL-Macro TinkerBell_KB_CMN_2 1 0.0199 LDC-MEAN-ALL-Macro TinkerBell_KB_CMN_2 ALL 0.1478 (PRIMARY METRIC) *ALL-Macro AP refer to mean of corresponding AP values. Scores based on confidence values and Average Precision (AP); only k=1 justification allowed: Metric RunID Hop AP SF-ALL-Macro TinkerBell_KB_CMN_2 0 0.1358 SF-ALL-Macro TinkerBell_KB_CMN_2 1 0.0165 SF-ALL-Macro TinkerBell_KB_CMN_2 ALL 0.1130 LDC-MEAN-ALL-Macro TinkerBell_KB_CMN_2 0 0.1492 LDC-MEAN-ALL-Macro TinkerBell_KB_CMN_2 1 0.0212 LDC-MEAN-ALL-Macro TinkerBell_KB_CMN_2 ALL 0.1237 *ALL-Macro AP refer to mean of corresponding AP values. Scores based on P/R/F1; only k=1 justification allowed: Metric RunID Hop Prec Recall F1 SF-ALL-Micro TinkerBell_KB_CMN_2 0 0.2409 0.6040 0.3444 SF-ALL-Micro TinkerBell_KB_CMN_2 1 0.0222 0.5316 0.0426 SF-ALL-Micro TinkerBell_KB_CMN_2 ALL 0.0546 0.5768 0.0998 SF-ALL-Macro TinkerBell_KB_CMN_2 0 0.1474 0.2268 0.1520 SF-ALL-Macro TinkerBell_KB_CMN_2 1 0.2389 0.5471 0.2870 SF-ALL-Macro TinkerBell_KB_CMN_2 ALL 0.1974 0.4020 0.2259 LDC-MAX-ALL-Micro TinkerBell_KB_CMN_2 0 0.2058 0.6121 0.3080 LDC-MAX-ALL-Micro TinkerBell_KB_CMN_2 1 0.0158 0.5000 0.0306 LDC-MAX-ALL-Micro TinkerBell_KB_CMN_2 ALL 0.0481 0.5769 0.0889 LDC-MAX-ALL-Macro TinkerBell_KB_CMN_2 0 0.1631 0.2812 0.1702 LDC-MAX-ALL-Macro TinkerBell_KB_CMN_2 1 0.2223 0.4934 0.2657 LDC-MAX-ALL-Macro TinkerBell_KB_CMN_2 ALL 0.1901 0.3781 0.2138 LDC-MEAN-ALL-Macro TinkerBell_KB_CMN_2 0 0.1631 0.2812 0.1702 LDC-MEAN-ALL-Macro TinkerBell_KB_CMN_2 1 0.2223 0.4934 0.2657 LDC-MEAN-ALL-Macro TinkerBell_KB_CMN_2 ALL 0.1901 0.3781 0.2138 *ALL-Macro Prec, Recall and F1 refer to mean-precision, mean-recall and mean-F1. ************************************************************* Run ID: TinkerBell_KB_CMN_3 Did the run access the live Web during the evaluation window: No Did the run extract relations from the Cold Start source corpus: Yes Did the run generate meaningful confidence values: Yes Did the run attempt the full Entity Discovery and Linking task: Yes Did the run attempt the full Event Nugget Detection and Coreference task: Yes Did the run include SF relations: Yes Did the run attempt the Event Argument Extraction and Linking task: Yes Did the run include Sentiment relations: Yes --------------------- Composite KB Evaluation (All slot types): Scores based on confidence values and Average Precision (AP); k=3 justifications allowed: Metric RunID Hop AP SF-ALL-Macro TinkerBell_KB_CMN_3 0 0.1386 SF-ALL-Macro TinkerBell_KB_CMN_3 1 0.0189 SF-ALL-Macro TinkerBell_KB_CMN_3 ALL 0.1158 LDC-MEAN-ALL-Macro TinkerBell_KB_CMN_3 0 0.1629 LDC-MEAN-ALL-Macro TinkerBell_KB_CMN_3 1 0.0195 LDC-MEAN-ALL-Macro TinkerBell_KB_CMN_3 ALL 0.1399 (PRIMARY METRIC) *ALL-Macro AP refer to mean of corresponding AP values. Scores based on confidence values and Average Precision (AP); only k=1 justification allowed: Metric RunID Hop AP SF-ALL-Macro TinkerBell_KB_CMN_3 0 0.1837 SF-ALL-Macro TinkerBell_KB_CMN_3 1 0.0277 SF-ALL-Macro TinkerBell_KB_CMN_3 ALL 0.1536 LDC-MEAN-ALL-Macro TinkerBell_KB_CMN_3 0 0.2055 LDC-MEAN-ALL-Macro TinkerBell_KB_CMN_3 1 0.0240 LDC-MEAN-ALL-Macro TinkerBell_KB_CMN_3 ALL 0.1781 *ALL-Macro AP refer to mean of corresponding AP values. Scores based on P/R/F1; only k=1 justification allowed: Metric RunID Hop Prec Recall F1 SF-ALL-Micro TinkerBell_KB_CMN_3 0 0.2992 0.3485 0.3220 SF-ALL-Micro TinkerBell_KB_CMN_3 1 0.0321 0.4558 0.0600 SF-ALL-Micro TinkerBell_KB_CMN_3 ALL 0.0732 0.3819 0.1228 SF-ALL-Macro TinkerBell_KB_CMN_3 0 0.2180 0.2263 0.2042 SF-ALL-Macro TinkerBell_KB_CMN_3 1 0.1256 0.2442 0.1415 SF-ALL-Macro TinkerBell_KB_CMN_3 ALL 0.1836 0.2329 0.1809 LDC-MAX-ALL-Micro TinkerBell_KB_CMN_3 0 0.2926 0.3430 0.3158 LDC-MAX-ALL-Micro TinkerBell_KB_CMN_3 1 0.0325 0.4958 0.0610 LDC-MAX-ALL-Micro TinkerBell_KB_CMN_3 ALL 0.0694 0.3915 0.1179 LDC-MAX-ALL-Macro TinkerBell_KB_CMN_3 0 0.2615 0.2669 0.2413 LDC-MAX-ALL-Macro TinkerBell_KB_CMN_3 1 0.1093 0.1944 0.1221 LDC-MAX-ALL-Macro TinkerBell_KB_CMN_3 ALL 0.2083 0.2416 0.1996 LDC-MEAN-ALL-Macro TinkerBell_KB_CMN_3 0 0.2480 0.2562 0.2297 LDC-MEAN-ALL-Macro TinkerBell_KB_CMN_3 1 0.1055 0.1909 0.1185 LDC-MEAN-ALL-Macro TinkerBell_KB_CMN_3 ALL 0.1982 0.2334 0.1908 *ALL-Macro Prec, Recall and F1 refer to mean-precision, mean-recall and mean-F1. --------------------- Composite KB Evaluation (SF-SLOTS only): Scores based on confidence values and Average Precision (AP); k=3 justifications allowed: Metric RunID Hop AP SF-ALL-Macro TinkerBell_KB_CMN_3 0 0.1261 SF-ALL-Macro TinkerBell_KB_CMN_3 1 0.0249 SF-ALL-Macro TinkerBell_KB_CMN_3 ALL 0.0887 LDC-MEAN-ALL-Macro TinkerBell_KB_CMN_3 0 0.1401 LDC-MEAN-ALL-Macro TinkerBell_KB_CMN_3 1 0.0193 LDC-MEAN-ALL-Macro TinkerBell_KB_CMN_3 ALL 0.1038 (PRIMARY METRIC) *ALL-Macro AP refer to mean of corresponding AP values. Scores based on confidence values and Average Precision (AP); only k=1 justification allowed: Metric RunID Hop AP SF-ALL-Macro TinkerBell_KB_CMN_3 0 0.1929 SF-ALL-Macro TinkerBell_KB_CMN_3 1 0.0450 SF-ALL-Macro TinkerBell_KB_CMN_3 ALL 0.1393 LDC-MEAN-ALL-Macro TinkerBell_KB_CMN_3 0 0.2042 LDC-MEAN-ALL-Macro TinkerBell_KB_CMN_3 1 0.0331 LDC-MEAN-ALL-Macro TinkerBell_KB_CMN_3 ALL 0.1580 *ALL-Macro AP refer to mean of corresponding AP values. Scores based on P/R/F1; only k=1 justification allowed: Metric RunID Hop Prec Recall F1 SF-ALL-Micro TinkerBell_KB_CMN_3 0 0.4775 0.1707 0.2515 SF-ALL-Micro TinkerBell_KB_CMN_3 1 0.2558 0.1215 0.1648 SF-ALL-Micro TinkerBell_KB_CMN_3 ALL 0.4156 0.1596 0.2306 SF-ALL-Macro TinkerBell_KB_CMN_3 0 0.2228 0.1985 0.2024 SF-ALL-Macro TinkerBell_KB_CMN_3 1 0.0732 0.0748 0.0669 SF-ALL-Macro TinkerBell_KB_CMN_3 ALL 0.1719 0.1564 0.1563 LDC-MAX-ALL-Micro TinkerBell_KB_CMN_3 0 0.4940 0.1814 0.2654 LDC-MAX-ALL-Micro TinkerBell_KB_CMN_3 1 0.2500 0.1316 0.1724 LDC-MAX-ALL-Micro TinkerBell_KB_CMN_3 ALL 0.4292 0.1714 0.2449 LDC-MAX-ALL-Macro TinkerBell_KB_CMN_3 0 0.2544 0.2245 0.2299 LDC-MAX-ALL-Macro TinkerBell_KB_CMN_3 1 0.0737 0.0831 0.0731 LDC-MAX-ALL-Macro TinkerBell_KB_CMN_3 ALL 0.1897 0.1739 0.1738 LDC-MEAN-ALL-Macro TinkerBell_KB_CMN_3 0 0.2425 0.2121 0.2172 LDC-MEAN-ALL-Macro TinkerBell_KB_CMN_3 1 0.0661 0.0762 0.0660 LDC-MEAN-ALL-Macro TinkerBell_KB_CMN_3 ALL 0.1794 0.1634 0.1631 *ALL-Macro Prec, Recall and F1 refer to mean-precision, mean-recall and mean-F1. --------------------- Composite KB Evaluation (EVENT-SLOTS only): Scores based on confidence values and Average Precision (AP); k=3 justifications allowed: SUMMARY: This section provides summary of AP scores Metric RunID Hop AP SF-ALL-Macro TinkerBell_KB_CMN_3 0 0.1486 SF-ALL-Macro TinkerBell_KB_CMN_3 ALL 0.1486 LDC-MEAN-ALL-Macro TinkerBell_KB_CMN_3 0 0.1908 LDC-MEAN-ALL-Macro TinkerBell_KB_CMN_3 ALL 0.1908 (PRIMARY METRIC) *ALL-Macro AP refer to mean of corresponding AP values. Scores based on confidence values and Average Precision (AP); only k=1 justification allowed: SUMMARY: This section provides summary of AP scores Metric RunID Hop AP SF-ALL-Macro TinkerBell_KB_CMN_3 0 0.1969 SF-ALL-Macro TinkerBell_KB_CMN_3 ALL 0.1969 LDC-MEAN-ALL-Macro TinkerBell_KB_CMN_3 0 0.2382 LDC-MEAN-ALL-Macro TinkerBell_KB_CMN_3 ALL 0.2382 *ALL-Macro AP refer to mean of corresponding AP values. Scores based on P/R/F1; only k=1 justification allowed: Metric RunID Hop Prec Recall F1 SF-ALL-Micro TinkerBell_KB_CMN_3 0 0.3557 0.3506 0.3532 SF-ALL-Micro TinkerBell_KB_CMN_3 ALL 0.3557 0.3506 0.3532 SF-ALL-Macro TinkerBell_KB_CMN_3 0 0.2499 0.2667 0.2358 SF-ALL-Macro TinkerBell_KB_CMN_3 ALL 0.2499 0.2667 0.2358 LDC-MAX-ALL-Micro TinkerBell_KB_CMN_3 0 0.3811 0.3822 0.3816 LDC-MAX-ALL-Micro TinkerBell_KB_CMN_3 ALL 0.3811 0.3822 0.3816 LDC-MAX-ALL-Macro TinkerBell_KB_CMN_3 0 0.3253 0.3230 0.2968 LDC-MAX-ALL-Macro TinkerBell_KB_CMN_3 ALL 0.3253 0.3230 0.2968 LDC-MEAN-ALL-Macro TinkerBell_KB_CMN_3 0 0.3023 0.3091 0.2807 LDC-MEAN-ALL-Macro TinkerBell_KB_CMN_3 ALL 0.3023 0.3091 0.2807 *ALL-Macro Prec, Recall and F1 refer to mean-precision, mean-recall and mean-F1. --------------------- Composite KB Evaluation (SENTIMENT-SLOTS only): Scores based on confidence values and Average Precision (AP); k=3 justifications allowed: Metric RunID Hop AP SF-ALL-Macro TinkerBell_KB_CMN_3 0 0.1529 SF-ALL-Macro TinkerBell_KB_CMN_3 1 0.0143 SF-ALL-Macro TinkerBell_KB_CMN_3 ALL 0.1268 LDC-MEAN-ALL-Macro TinkerBell_KB_CMN_3 0 0.1739 LDC-MEAN-ALL-Macro TinkerBell_KB_CMN_3 1 0.0171 LDC-MEAN-ALL-Macro TinkerBell_KB_CMN_3 ALL 0.1450 (PRIMARY METRIC) *ALL-Macro AP refer to mean of corresponding AP values. Scores based on confidence values and Average Precision (AP); only k=1 justification allowed: Metric RunID Hop AP SF-ALL-Macro TinkerBell_KB_CMN_3 0 0.1351 SF-ALL-Macro TinkerBell_KB_CMN_3 1 0.0156 SF-ALL-Macro TinkerBell_KB_CMN_3 ALL 0.1118 LDC-MEAN-ALL-Macro TinkerBell_KB_CMN_3 0 0.1484 LDC-MEAN-ALL-Macro TinkerBell_KB_CMN_3 1 0.0188 LDC-MEAN-ALL-Macro TinkerBell_KB_CMN_3 ALL 0.1219 *ALL-Macro AP refer to mean of corresponding AP values. Scores based on P/R/F1; only k=1 justification allowed: Metric RunID Hop Prec Recall F1 SF-ALL-Micro TinkerBell_KB_CMN_3 0 0.2362 0.5928 0.3378 SF-ALL-Micro TinkerBell_KB_CMN_3 1 0.0222 0.5316 0.0426 SF-ALL-Micro TinkerBell_KB_CMN_3 ALL 0.0539 0.5698 0.0986 SF-ALL-Macro TinkerBell_KB_CMN_3 0 0.1468 0.2254 0.1512 SF-ALL-Macro TinkerBell_KB_CMN_3 1 0.2389 0.5471 0.2870 SF-ALL-Macro TinkerBell_KB_CMN_3 ALL 0.1972 0.4014 0.2255 LDC-MAX-ALL-Micro TinkerBell_KB_CMN_3 0 0.2000 0.5991 0.2999 LDC-MAX-ALL-Micro TinkerBell_KB_CMN_3 1 0.0158 0.5000 0.0306 LDC-MAX-ALL-Micro TinkerBell_KB_CMN_3 ALL 0.0473 0.5680 0.0874 LDC-MAX-ALL-Macro TinkerBell_KB_CMN_3 0 0.1623 0.2796 0.1690 LDC-MAX-ALL-Macro TinkerBell_KB_CMN_3 1 0.2223 0.4934 0.2657 LDC-MAX-ALL-Macro TinkerBell_KB_CMN_3 ALL 0.1897 0.3772 0.2132 LDC-MEAN-ALL-Macro TinkerBell_KB_CMN_3 0 0.1623 0.2796 0.1690 LDC-MEAN-ALL-Macro TinkerBell_KB_CMN_3 1 0.2223 0.4934 0.2657 LDC-MEAN-ALL-Macro TinkerBell_KB_CMN_3 ALL 0.1897 0.3772 0.2132 *ALL-Macro Prec, Recall and F1 refer to mean-precision, mean-recall and mean-F1. ************************************************************* Run ID: TinkerBell_KB_CMN_4 Did the run access the live Web during the evaluation window: No Did the run extract relations from the Cold Start source corpus: Yes Did the run generate meaningful confidence values: Yes Did the run attempt the full Entity Discovery and Linking task: Yes Did the run attempt the full Event Nugget Detection and Coreference task: Yes Did the run include SF relations: Yes Did the run attempt the Event Argument Extraction and Linking task: Yes Did the run include Sentiment relations: Yes --------------------- Composite KB Evaluation (All slot types): Scores based on confidence values and Average Precision (AP); k=3 justifications allowed: Metric RunID Hop AP SF-ALL-Macro TinkerBell_KB_CMN_4 0 0.1271 SF-ALL-Macro TinkerBell_KB_CMN_4 1 0.0171 SF-ALL-Macro TinkerBell_KB_CMN_4 ALL 0.1090 LDC-MEAN-ALL-Macro TinkerBell_KB_CMN_4 0 0.1473 LDC-MEAN-ALL-Macro TinkerBell_KB_CMN_4 1 0.0175 LDC-MEAN-ALL-Macro TinkerBell_KB_CMN_4 ALL 0.1306 (PRIMARY METRIC) *ALL-Macro AP refer to mean of corresponding AP values. Scores based on confidence values and Average Precision (AP); only k=1 justification allowed: Metric RunID Hop AP SF-ALL-Macro TinkerBell_KB_CMN_4 0 0.1841 SF-ALL-Macro TinkerBell_KB_CMN_4 1 0.0294 SF-ALL-Macro TinkerBell_KB_CMN_4 ALL 0.1547 LDC-MEAN-ALL-Macro TinkerBell_KB_CMN_4 0 0.2062 LDC-MEAN-ALL-Macro TinkerBell_KB_CMN_4 1 0.0269 LDC-MEAN-ALL-Macro TinkerBell_KB_CMN_4 ALL 0.1798 *ALL-Macro AP refer to mean of corresponding AP values. Scores based on P/R/F1; only k=1 justification allowed: Metric RunID Hop Prec Recall F1 SF-ALL-Micro TinkerBell_KB_CMN_4 0 0.3035 0.3485 0.3245 SF-ALL-Micro TinkerBell_KB_CMN_4 1 0.0455 0.4558 0.0827 SF-ALL-Micro TinkerBell_KB_CMN_4 ALL 0.0977 0.3819 0.1556 SF-ALL-Macro TinkerBell_KB_CMN_4 0 0.2189 0.2263 0.2045 SF-ALL-Macro TinkerBell_KB_CMN_4 1 0.1268 0.2442 0.1423 SF-ALL-Macro TinkerBell_KB_CMN_4 ALL 0.1846 0.2329 0.1814 LDC-MAX-ALL-Micro TinkerBell_KB_CMN_4 0 0.2982 0.3430 0.3191 LDC-MAX-ALL-Micro TinkerBell_KB_CMN_4 1 0.0559 0.4958 0.1005 LDC-MAX-ALL-Micro TinkerBell_KB_CMN_4 ALL 0.1088 0.3915 0.1703 LDC-MAX-ALL-Macro TinkerBell_KB_CMN_4 0 0.2628 0.2669 0.2417 LDC-MAX-ALL-Macro TinkerBell_KB_CMN_4 1 0.1113 0.1944 0.1234 LDC-MAX-ALL-Macro TinkerBell_KB_CMN_4 ALL 0.2098 0.2416 0.2003 LDC-MEAN-ALL-Macro TinkerBell_KB_CMN_4 0 0.2493 0.2562 0.2301 LDC-MEAN-ALL-Macro TinkerBell_KB_CMN_4 1 0.1075 0.1909 0.1198 LDC-MEAN-ALL-Macro TinkerBell_KB_CMN_4 ALL 0.1997 0.2334 0.1915 *ALL-Macro Prec, Recall and F1 refer to mean-precision, mean-recall and mean-F1. --------------------- Composite KB Evaluation (SF-SLOTS only): Scores based on confidence values and Average Precision (AP); k=3 justifications allowed: Metric RunID Hop AP SF-ALL-Macro TinkerBell_KB_CMN_4 0 0.1023 SF-ALL-Macro TinkerBell_KB_CMN_4 1 0.0213 SF-ALL-Macro TinkerBell_KB_CMN_4 ALL 0.0746 LDC-MEAN-ALL-Macro TinkerBell_KB_CMN_4 0 0.1083 LDC-MEAN-ALL-Macro TinkerBell_KB_CMN_4 1 0.0154 LDC-MEAN-ALL-Macro TinkerBell_KB_CMN_4 ALL 0.0848 (PRIMARY METRIC) *ALL-Macro AP refer to mean of corresponding AP values. Scores based on confidence values and Average Precision (AP); only k=1 justification allowed: Metric RunID Hop AP SF-ALL-Macro TinkerBell_KB_CMN_4 0 0.1939 SF-ALL-Macro TinkerBell_KB_CMN_4 1 0.0450 SF-ALL-Macro TinkerBell_KB_CMN_4 ALL 0.1415 LDC-MEAN-ALL-Macro TinkerBell_KB_CMN_4 0 0.2056 LDC-MEAN-ALL-Macro TinkerBell_KB_CMN_4 1 0.0331 LDC-MEAN-ALL-Macro TinkerBell_KB_CMN_4 ALL 0.1614 *ALL-Macro AP refer to mean of corresponding AP values. Scores based on P/R/F1; only k=1 justification allowed: Metric RunID Hop Prec Recall F1 SF-ALL-Micro TinkerBell_KB_CMN_4 0 0.5436 0.1707 0.2598 SF-ALL-Micro TinkerBell_KB_CMN_4 1 0.3056 0.1215 0.1739 SF-ALL-Micro TinkerBell_KB_CMN_4 ALL 0.4794 0.1596 0.2395 SF-ALL-Macro TinkerBell_KB_CMN_4 0 0.2247 0.1985 0.2030 SF-ALL-Macro TinkerBell_KB_CMN_4 1 0.0761 0.0748 0.0689 SF-ALL-Macro TinkerBell_KB_CMN_4 ALL 0.1741 0.1564 0.1573 LDC-MAX-ALL-Micro TinkerBell_KB_CMN_4 0 0.5775 0.1814 0.2761 LDC-MAX-ALL-Micro TinkerBell_KB_CMN_4 1 0.3125 0.1316 0.1852 LDC-MAX-ALL-Micro TinkerBell_KB_CMN_4 ALL 0.5105 0.1714 0.2566 LDC-MAX-ALL-Macro TinkerBell_KB_CMN_4 0 0.2570 0.2245 0.2308 LDC-MAX-ALL-Macro TinkerBell_KB_CMN_4 1 0.0776 0.0831 0.0756 LDC-MAX-ALL-Macro TinkerBell_KB_CMN_4 ALL 0.1928 0.1739 0.1752 LDC-MEAN-ALL-Macro TinkerBell_KB_CMN_4 0 0.2452 0.2121 0.2181 LDC-MEAN-ALL-Macro TinkerBell_KB_CMN_4 1 0.0700 0.0762 0.0685 LDC-MEAN-ALL-Macro TinkerBell_KB_CMN_4 ALL 0.1824 0.1634 0.1645 *ALL-Macro Prec, Recall and F1 refer to mean-precision, mean-recall and mean-F1. --------------------- Composite KB Evaluation (EVENT-SLOTS only): Scores based on confidence values and Average Precision (AP); k=3 justifications allowed: SUMMARY: This section provides summary of AP scores Metric RunID Hop AP SF-ALL-Macro TinkerBell_KB_CMN_4 0 0.1486 SF-ALL-Macro TinkerBell_KB_CMN_4 ALL 0.1486 LDC-MEAN-ALL-Macro TinkerBell_KB_CMN_4 0 0.1908 LDC-MEAN-ALL-Macro TinkerBell_KB_CMN_4 ALL 0.1908 (PRIMARY METRIC) *ALL-Macro AP refer to mean of corresponding AP values. Scores based on confidence values and Average Precision (AP); only k=1 justification allowed: SUMMARY: This section provides summary of AP scores Metric RunID Hop AP SF-ALL-Macro TinkerBell_KB_CMN_4 0 0.1969 SF-ALL-Macro TinkerBell_KB_CMN_4 ALL 0.1969 LDC-MEAN-ALL-Macro TinkerBell_KB_CMN_4 0 0.2382 LDC-MEAN-ALL-Macro TinkerBell_KB_CMN_4 ALL 0.2382 *ALL-Macro AP refer to mean of corresponding AP values. Scores based on P/R/F1; only k=1 justification allowed: Metric RunID Hop Prec Recall F1 SF-ALL-Micro TinkerBell_KB_CMN_4 0 0.3557 0.3506 0.3532 SF-ALL-Micro TinkerBell_KB_CMN_4 ALL 0.3557 0.3506 0.3532 SF-ALL-Macro TinkerBell_KB_CMN_4 0 0.2499 0.2667 0.2358 SF-ALL-Macro TinkerBell_KB_CMN_4 ALL 0.2499 0.2667 0.2358 LDC-MAX-ALL-Micro TinkerBell_KB_CMN_4 0 0.3811 0.3822 0.3816 LDC-MAX-ALL-Micro TinkerBell_KB_CMN_4 ALL 0.3811 0.3822 0.3816 LDC-MAX-ALL-Macro TinkerBell_KB_CMN_4 0 0.3253 0.3230 0.2968 LDC-MAX-ALL-Macro TinkerBell_KB_CMN_4 ALL 0.3253 0.3230 0.2968 LDC-MEAN-ALL-Macro TinkerBell_KB_CMN_4 0 0.3023 0.3091 0.2807 LDC-MEAN-ALL-Macro TinkerBell_KB_CMN_4 ALL 0.3023 0.3091 0.2807 *ALL-Macro Prec, Recall and F1 refer to mean-precision, mean-recall and mean-F1. --------------------- Composite KB Evaluation (SENTIMENT-SLOTS only): Scores based on confidence values and Average Precision (AP); k=3 justifications allowed: Metric RunID Hop AP SF-ALL-Macro TinkerBell_KB_CMN_4 0 0.1529 SF-ALL-Macro TinkerBell_KB_CMN_4 1 0.0143 SF-ALL-Macro TinkerBell_KB_CMN_4 ALL 0.1268 LDC-MEAN-ALL-Macro TinkerBell_KB_CMN_4 0 0.1739 LDC-MEAN-ALL-Macro TinkerBell_KB_CMN_4 1 0.0171 LDC-MEAN-ALL-Macro TinkerBell_KB_CMN_4 ALL 0.1450 (PRIMARY METRIC) *ALL-Macro AP refer to mean of corresponding AP values. Scores based on confidence values and Average Precision (AP); only k=1 justification allowed: Metric RunID Hop AP SF-ALL-Macro TinkerBell_KB_CMN_4 0 0.1351 SF-ALL-Macro TinkerBell_KB_CMN_4 1 0.0156 SF-ALL-Macro TinkerBell_KB_CMN_4 ALL 0.1119 LDC-MEAN-ALL-Macro TinkerBell_KB_CMN_4 0 0.1484 LDC-MEAN-ALL-Macro TinkerBell_KB_CMN_4 1 0.0188 LDC-MEAN-ALL-Macro TinkerBell_KB_CMN_4 ALL 0.1219 *ALL-Macro AP refer to mean of corresponding AP values. Scores based on P/R/F1; only k=1 justification allowed: Metric RunID Hop Prec Recall F1 SF-ALL-Micro TinkerBell_KB_CMN_4 0 0.2362 0.5928 0.3378 SF-ALL-Micro TinkerBell_KB_CMN_4 1 0.0222 0.5316 0.0426 SF-ALL-Micro TinkerBell_KB_CMN_4 ALL 0.0539 0.5698 0.0986 SF-ALL-Macro TinkerBell_KB_CMN_4 0 0.1468 0.2254 0.1512 SF-ALL-Macro TinkerBell_KB_CMN_4 1 0.2389 0.5471 0.2870 SF-ALL-Macro TinkerBell_KB_CMN_4 ALL 0.1972 0.4014 0.2255 LDC-MAX-ALL-Micro TinkerBell_KB_CMN_4 0 0.2000 0.5991 0.2999 LDC-MAX-ALL-Micro TinkerBell_KB_CMN_4 1 0.0158 0.5000 0.0306 LDC-MAX-ALL-Micro TinkerBell_KB_CMN_4 ALL 0.0473 0.5680 0.0874 LDC-MAX-ALL-Macro TinkerBell_KB_CMN_4 0 0.1623 0.2796 0.1690 LDC-MAX-ALL-Macro TinkerBell_KB_CMN_4 1 0.2223 0.4934 0.2657 LDC-MAX-ALL-Macro TinkerBell_KB_CMN_4 ALL 0.1897 0.3772 0.2132 LDC-MEAN-ALL-Macro TinkerBell_KB_CMN_4 0 0.1623 0.2796 0.1690 LDC-MEAN-ALL-Macro TinkerBell_KB_CMN_4 1 0.2223 0.4934 0.2657 LDC-MEAN-ALL-Macro TinkerBell_KB_CMN_4 ALL 0.1897 0.3772 0.2132 *ALL-Macro Prec, Recall and F1 refer to mean-precision, mean-recall and mean-F1. ************************************************************* Run ID: TinkerBell_KB_CMN_5 Did the run access the live Web during the evaluation window: No Did the run extract relations from the Cold Start source corpus: Yes Did the run generate meaningful confidence values: Yes Did the run attempt the full Entity Discovery and Linking task: Yes Did the run attempt the full Event Nugget Detection and Coreference task: Yes Did the run include SF relations: Yes Did the run attempt the Event Argument Extraction and Linking task: Yes Did the run include Sentiment relations: Yes --------------------- Composite KB Evaluation (All slot types): Scores based on confidence values and Average Precision (AP); k=3 justifications allowed: Metric RunID Hop AP SF-ALL-Macro TinkerBell_KB_CMN_5 0 0.1411 SF-ALL-Macro TinkerBell_KB_CMN_5 1 0.0193 SF-ALL-Macro TinkerBell_KB_CMN_5 ALL 0.1181 LDC-MEAN-ALL-Macro TinkerBell_KB_CMN_5 0 0.1660 LDC-MEAN-ALL-Macro TinkerBell_KB_CMN_5 1 0.0201 LDC-MEAN-ALL-Macro TinkerBell_KB_CMN_5 ALL 0.1430 (PRIMARY METRIC) *ALL-Macro AP refer to mean of corresponding AP values. Scores based on confidence values and Average Precision (AP); only k=1 justification allowed: Metric RunID Hop AP SF-ALL-Macro TinkerBell_KB_CMN_5 0 0.1830 SF-ALL-Macro TinkerBell_KB_CMN_5 1 0.0279 SF-ALL-Macro TinkerBell_KB_CMN_5 ALL 0.1531 LDC-MEAN-ALL-Macro TinkerBell_KB_CMN_5 0 0.2046 LDC-MEAN-ALL-Macro TinkerBell_KB_CMN_5 1 0.0246 LDC-MEAN-ALL-Macro TinkerBell_KB_CMN_5 ALL 0.1774 *ALL-Macro AP refer to mean of corresponding AP values. Scores based on P/R/F1; only k=1 justification allowed: Metric RunID Hop Prec Recall F1 SF-ALL-Micro TinkerBell_KB_CMN_5 0 0.3037 0.3485 0.3246 SF-ALL-Micro TinkerBell_KB_CMN_5 1 0.0321 0.4558 0.0600 SF-ALL-Micro TinkerBell_KB_CMN_5 ALL 0.0734 0.3819 0.1231 SF-ALL-Macro TinkerBell_KB_CMN_5 0 0.2179 0.2248 0.2036 SF-ALL-Macro TinkerBell_KB_CMN_5 1 0.1258 0.2442 0.1416 SF-ALL-Macro TinkerBell_KB_CMN_5 ALL 0.1836 0.2320 0.1805 LDC-MAX-ALL-Micro TinkerBell_KB_CMN_5 0 0.2961 0.3421 0.3174 LDC-MAX-ALL-Micro TinkerBell_KB_CMN_5 1 0.0325 0.4958 0.0611 LDC-MAX-ALL-Micro TinkerBell_KB_CMN_5 ALL 0.0695 0.3909 0.1180 LDC-MAX-ALL-Macro TinkerBell_KB_CMN_5 0 0.2611 0.2648 0.2401 LDC-MAX-ALL-Macro TinkerBell_KB_CMN_5 1 0.1098 0.1944 0.1224 LDC-MAX-ALL-Macro TinkerBell_KB_CMN_5 ALL 0.2082 0.2402 0.1989 LDC-MEAN-ALL-Macro TinkerBell_KB_CMN_5 0 0.2476 0.2541 0.2285 LDC-MEAN-ALL-Macro TinkerBell_KB_CMN_5 1 0.1060 0.1909 0.1187 LDC-MEAN-ALL-Macro TinkerBell_KB_CMN_5 ALL 0.1981 0.2320 0.1901 *ALL-Macro Prec, Recall and F1 refer to mean-precision, mean-recall and mean-F1. --------------------- Composite KB Evaluation (SF-SLOTS only): Scores based on confidence values and Average Precision (AP); k=3 justifications allowed: Metric RunID Hop AP SF-ALL-Macro TinkerBell_KB_CMN_5 0 0.1261 SF-ALL-Macro TinkerBell_KB_CMN_5 1 0.0249 SF-ALL-Macro TinkerBell_KB_CMN_5 ALL 0.0885 LDC-MEAN-ALL-Macro TinkerBell_KB_CMN_5 0 0.1401 LDC-MEAN-ALL-Macro TinkerBell_KB_CMN_5 1 0.0193 LDC-MEAN-ALL-Macro TinkerBell_KB_CMN_5 ALL 0.1036 (PRIMARY METRIC) *ALL-Macro AP refer to mean of corresponding AP values. Scores based on confidence values and Average Precision (AP); only k=1 justification allowed: Metric RunID Hop AP SF-ALL-Macro TinkerBell_KB_CMN_5 0 0.1929 SF-ALL-Macro TinkerBell_KB_CMN_5 1 0.0450 SF-ALL-Macro TinkerBell_KB_CMN_5 ALL 0.1394 LDC-MEAN-ALL-Macro TinkerBell_KB_CMN_5 0 0.2042 LDC-MEAN-ALL-Macro TinkerBell_KB_CMN_5 1 0.0331 LDC-MEAN-ALL-Macro TinkerBell_KB_CMN_5 ALL 0.1581 *ALL-Macro AP refer to mean of corresponding AP values. Scores based on P/R/F1; only k=1 justification allowed: Metric RunID Hop Prec Recall F1 SF-ALL-Micro TinkerBell_KB_CMN_5 0 0.4775 0.1707 0.2515 SF-ALL-Micro TinkerBell_KB_CMN_5 1 0.2558 0.1215 0.1648 SF-ALL-Micro TinkerBell_KB_CMN_5 ALL 0.4156 0.1596 0.2306 SF-ALL-Macro TinkerBell_KB_CMN_5 0 0.2228 0.1985 0.2024 SF-ALL-Macro TinkerBell_KB_CMN_5 1 0.0732 0.0748 0.0669 SF-ALL-Macro TinkerBell_KB_CMN_5 ALL 0.1719 0.1564 0.1563 LDC-MAX-ALL-Micro TinkerBell_KB_CMN_5 0 0.4970 0.1814 0.2658 LDC-MAX-ALL-Micro TinkerBell_KB_CMN_5 1 0.2500 0.1316 0.1724 LDC-MAX-ALL-Micro TinkerBell_KB_CMN_5 ALL 0.4311 0.1714 0.2453 LDC-MAX-ALL-Macro TinkerBell_KB_CMN_5 0 0.2544 0.2245 0.2299 LDC-MAX-ALL-Macro TinkerBell_KB_CMN_5 1 0.0737 0.0831 0.0731 LDC-MAX-ALL-Macro TinkerBell_KB_CMN_5 ALL 0.1897 0.1739 0.1738 LDC-MEAN-ALL-Macro TinkerBell_KB_CMN_5 0 0.2425 0.2121 0.2172 LDC-MEAN-ALL-Macro TinkerBell_KB_CMN_5 1 0.0661 0.0762 0.0660 LDC-MEAN-ALL-Macro TinkerBell_KB_CMN_5 ALL 0.1794 0.1634 0.1631 *ALL-Macro Prec, Recall and F1 refer to mean-precision, mean-recall and mean-F1. --------------------- Composite KB Evaluation (EVENT-SLOTS only): Scores based on confidence values and Average Precision (AP); k=3 justifications allowed: SUMMARY: This section provides summary of AP scores Metric RunID Hop AP SF-ALL-Macro TinkerBell_KB_CMN_5 0 0.1548 SF-ALL-Macro TinkerBell_KB_CMN_5 ALL 0.1548 LDC-MEAN-ALL-Macro TinkerBell_KB_CMN_5 0 0.1988 LDC-MEAN-ALL-Macro TinkerBell_KB_CMN_5 ALL 0.1988 (PRIMARY METRIC) *ALL-Macro AP refer to mean of corresponding AP values. Scores based on confidence values and Average Precision (AP); only k=1 justification allowed: SUMMARY: This section provides summary of AP scores Metric RunID Hop AP SF-ALL-Macro TinkerBell_KB_CMN_5 0 0.1946 SF-ALL-Macro TinkerBell_KB_CMN_5 ALL 0.1946 LDC-MEAN-ALL-Macro TinkerBell_KB_CMN_5 0 0.2350 LDC-MEAN-ALL-Macro TinkerBell_KB_CMN_5 ALL 0.2350 *ALL-Macro AP refer to mean of corresponding AP values. Scores based on P/R/F1; only k=1 justification allowed: Metric RunID Hop Prec Recall F1 SF-ALL-Micro TinkerBell_KB_CMN_5 0 0.3645 0.3417 0.3527 SF-ALL-Micro TinkerBell_KB_CMN_5 ALL 0.3645 0.3417 0.3527 SF-ALL-Macro TinkerBell_KB_CMN_5 0 0.2493 0.2613 0.2334 SF-ALL-Macro TinkerBell_KB_CMN_5 ALL 0.2493 0.2613 0.2334 LDC-MAX-ALL-Micro TinkerBell_KB_CMN_5 0 0.3839 0.3707 0.3772 LDC-MAX-ALL-Micro TinkerBell_KB_CMN_5 ALL 0.3839 0.3707 0.3772 LDC-MAX-ALL-Macro TinkerBell_KB_CMN_5 0 0.3236 0.3158 0.2926 LDC-MAX-ALL-Macro TinkerBell_KB_CMN_5 ALL 0.3236 0.3158 0.2926 LDC-MEAN-ALL-Macro TinkerBell_KB_CMN_5 0 0.3006 0.3019 0.2765 LDC-MEAN-ALL-Macro TinkerBell_KB_CMN_5 ALL 0.3006 0.3019 0.2765 *ALL-Macro Prec, Recall and F1 refer to mean-precision, mean-recall and mean-F1. --------------------- Composite KB Evaluation (SENTIMENT-SLOTS only): Scores based on confidence values and Average Precision (AP); k=3 justifications allowed: Metric RunID Hop AP SF-ALL-Macro TinkerBell_KB_CMN_5 0 0.1552 SF-ALL-Macro TinkerBell_KB_CMN_5 1 0.0161 SF-ALL-Macro TinkerBell_KB_CMN_5 ALL 0.1287 LDC-MEAN-ALL-Macro TinkerBell_KB_CMN_5 0 0.1763 LDC-MEAN-ALL-Macro TinkerBell_KB_CMN_5 1 0.0199 LDC-MEAN-ALL-Macro TinkerBell_KB_CMN_5 ALL 0.1478 (PRIMARY METRIC) *ALL-Macro AP refer to mean of corresponding AP values. Scores based on confidence values and Average Precision (AP); only k=1 justification allowed: Metric RunID Hop AP SF-ALL-Macro TinkerBell_KB_CMN_5 0 0.1358 SF-ALL-Macro TinkerBell_KB_CMN_5 1 0.0165 SF-ALL-Macro TinkerBell_KB_CMN_5 ALL 0.1130 LDC-MEAN-ALL-Macro TinkerBell_KB_CMN_5 0 0.1492 LDC-MEAN-ALL-Macro TinkerBell_KB_CMN_5 1 0.0212 LDC-MEAN-ALL-Macro TinkerBell_KB_CMN_5 ALL 0.1236 *ALL-Macro AP refer to mean of corresponding AP values. Scores based on P/R/F1; only k=1 justification allowed: Metric RunID Hop Prec Recall F1 SF-ALL-Micro TinkerBell_KB_CMN_5 0 0.2409 0.6040 0.3444 SF-ALL-Micro TinkerBell_KB_CMN_5 1 0.0222 0.5316 0.0426 SF-ALL-Micro TinkerBell_KB_CMN_5 ALL 0.0546 0.5768 0.0998 SF-ALL-Macro TinkerBell_KB_CMN_5 0 0.1474 0.2268 0.1520 SF-ALL-Macro TinkerBell_KB_CMN_5 1 0.2389 0.5471 0.2870 SF-ALL-Macro TinkerBell_KB_CMN_5 ALL 0.1974 0.4020 0.2259 LDC-MAX-ALL-Micro TinkerBell_KB_CMN_5 0 0.2058 0.6121 0.3080 LDC-MAX-ALL-Micro TinkerBell_KB_CMN_5 1 0.0158 0.5000 0.0306 LDC-MAX-ALL-Micro TinkerBell_KB_CMN_5 ALL 0.0481 0.5769 0.0889 LDC-MAX-ALL-Macro TinkerBell_KB_CMN_5 0 0.1631 0.2812 0.1702 LDC-MAX-ALL-Macro TinkerBell_KB_CMN_5 1 0.2223 0.4934 0.2657 LDC-MAX-ALL-Macro TinkerBell_KB_CMN_5 ALL 0.1901 0.3781 0.2138 LDC-MEAN-ALL-Macro TinkerBell_KB_CMN_5 0 0.1631 0.2812 0.1702 LDC-MEAN-ALL-Macro TinkerBell_KB_CMN_5 1 0.2223 0.4934 0.2657 LDC-MEAN-ALL-Macro TinkerBell_KB_CMN_5 ALL 0.1901 0.3781 0.2138 *ALL-Macro Prec, Recall and F1 refer to mean-precision, mean-recall and mean-F1.