===================================================================== TAC KBP 2017 ENGLISH KB CONSTRUCTION: COMPOSITE KB EVALUATION RESULTS ===================================================================== Team ID: STANFORD Organization: Stanford University ************************************************************* Run ID: STANFORD_KB_ENG_1 Did the run access the live Web during the evaluation window: No Did the run extract relations from the Cold Start source corpus: Yes Did the run generate meaningful confidence values: Yes Did the run attempt the full Entity Discovery and Linking task: No Did the run attempt the full Event Nugget Detection and Coreference task: No Did the run include SF relations: Yes Did the run attempt the Event Argument Extraction and Linking task: No Did the run include Sentiment relations: No --------------------- Composite KB Evaluation (All slot types): Scores based on confidence values and Average Precision (AP); k=3 justifications allowed: Metric RunID Hop AP SF-ALL-Macro STANFORD_KB_ENG_1 0 0.1005 SF-ALL-Macro STANFORD_KB_ENG_1 1 0.0235 SF-ALL-Macro STANFORD_KB_ENG_1 ALL 0.0786 LDC-MEAN-ALL-Macro STANFORD_KB_ENG_1 0 0.1175 LDC-MEAN-ALL-Macro STANFORD_KB_ENG_1 1 0.0326 LDC-MEAN-ALL-Macro STANFORD_KB_ENG_1 ALL 0.0936 (PRIMARY METRIC) *ALL-Macro AP refer to mean of corresponding AP values. Scores based on confidence values and Average Precision (AP); only k=1 justification allowed: Metric RunID Hop AP SF-ALL-Macro STANFORD_KB_ENG_1 0 0.1270 SF-ALL-Macro STANFORD_KB_ENG_1 1 0.0354 SF-ALL-Macro STANFORD_KB_ENG_1 ALL 0.1034 LDC-MEAN-ALL-Macro STANFORD_KB_ENG_1 0 0.1500 LDC-MEAN-ALL-Macro STANFORD_KB_ENG_1 1 0.0443 LDC-MEAN-ALL-Macro STANFORD_KB_ENG_1 ALL 0.1240 *ALL-Macro AP refer to mean of corresponding AP values. Scores based on P/R/F1; only k=1 justification allowed: Metric RunID Hop Prec Recall F1 SF-ALL-Micro STANFORD_KB_ENG_1 0 0.1798 0.1485 0.1626 SF-ALL-Micro STANFORD_KB_ENG_1 1 0.0304 0.1366 0.0497 SF-ALL-Micro STANFORD_KB_ENG_1 ALL 0.0877 0.1458 0.1095 SF-ALL-Macro STANFORD_KB_ENG_1 0 0.0934 0.1435 0.1017 SF-ALL-Macro STANFORD_KB_ENG_1 1 0.1098 0.1360 0.1122 SF-ALL-Macro STANFORD_KB_ENG_1 ALL 0.0991 0.1409 0.1053 LDC-MAX-ALL-Micro STANFORD_KB_ENG_1 0 0.2126 0.1635 0.1849 LDC-MAX-ALL-Micro STANFORD_KB_ENG_1 1 0.0345 0.1403 0.0554 LDC-MAX-ALL-Micro STANFORD_KB_ENG_1 ALL 0.1070 0.1585 0.1278 LDC-MAX-ALL-Macro STANFORD_KB_ENG_1 0 0.1181 0.1806 0.1286 LDC-MAX-ALL-Macro STANFORD_KB_ENG_1 1 0.1085 0.1457 0.1142 LDC-MAX-ALL-Macro STANFORD_KB_ENG_1 ALL 0.1148 0.1687 0.1237 LDC-MEAN-ALL-Macro STANFORD_KB_ENG_1 0 0.1108 0.1693 0.1209 LDC-MEAN-ALL-Macro STANFORD_KB_ENG_1 1 0.1124 0.1471 0.1168 LDC-MEAN-ALL-Macro STANFORD_KB_ENG_1 ALL 0.1114 0.1618 0.1195 *ALL-Macro Prec, Recall and F1 refer to mean-precision, mean-recall and mean-F1. --------------------- Composite KB Evaluation (SF-SLOTS only): Scores based on confidence values and Average Precision (AP); k=3 justifications allowed: Metric RunID Hop AP SF-ALL-Macro STANFORD_KB_ENG_1 0 0.2321 SF-ALL-Macro STANFORD_KB_ENG_1 1 0.0658 SF-ALL-Macro STANFORD_KB_ENG_1 ALL 0.1815 LDC-MEAN-ALL-Macro STANFORD_KB_ENG_1 0 0.2534 LDC-MEAN-ALL-Macro STANFORD_KB_ENG_1 1 0.0859 LDC-MEAN-ALL-Macro STANFORD_KB_ENG_1 ALL 0.2017 (PRIMARY METRIC) *ALL-Macro AP refer to mean of corresponding AP values. Scores based on confidence values and Average Precision (AP); only k=1 justification allowed: Metric RunID Hop AP SF-ALL-Macro STANFORD_KB_ENG_1 0 0.2932 SF-ALL-Macro STANFORD_KB_ENG_1 1 0.0991 SF-ALL-Macro STANFORD_KB_ENG_1 ALL 0.2386 LDC-MEAN-ALL-Macro STANFORD_KB_ENG_1 0 0.3233 LDC-MEAN-ALL-Macro STANFORD_KB_ENG_1 1 0.1168 LDC-MEAN-ALL-Macro STANFORD_KB_ENG_1 ALL 0.2672 *ALL-Macro AP refer to mean of corresponding AP values. Scores based on P/R/F1; only k=1 justification allowed: Metric RunID Hop Prec Recall F1 SF-ALL-Micro STANFORD_KB_ENG_1 0 0.1798 0.2997 0.2247 SF-ALL-Micro STANFORD_KB_ENG_1 1 0.0304 0.2568 0.0543 SF-ALL-Micro STANFORD_KB_ENG_1 ALL 0.0877 0.2894 0.1346 SF-ALL-Macro STANFORD_KB_ENG_1 0 0.2157 0.3312 0.2348 SF-ALL-Macro STANFORD_KB_ENG_1 1 0.2024 0.2506 0.2068 SF-ALL-Macro STANFORD_KB_ENG_1 ALL 0.2104 0.2993 0.2237 LDC-MAX-ALL-Micro STANFORD_KB_ENG_1 0 0.2118 0.3149 0.2533 LDC-MAX-ALL-Micro STANFORD_KB_ENG_1 1 0.0327 0.2468 0.0578 LDC-MAX-ALL-Micro STANFORD_KB_ENG_1 ALL 0.1036 0.2991 0.1538 LDC-MAX-ALL-Macro STANFORD_KB_ENG_1 0 0.2546 0.3893 0.2772 LDC-MAX-ALL-Macro STANFORD_KB_ENG_1 1 0.1914 0.2571 0.2015 LDC-MAX-ALL-Macro STANFORD_KB_ENG_1 ALL 0.2300 0.3380 0.2479 LDC-MEAN-ALL-Macro STANFORD_KB_ENG_1 0 0.2389 0.3651 0.2606 LDC-MEAN-ALL-Macro STANFORD_KB_ENG_1 1 0.1984 0.2596 0.2060 LDC-MEAN-ALL-Macro STANFORD_KB_ENG_1 ALL 0.2231 0.3241 0.2394 *ALL-Macro Prec, Recall and F1 refer to mean-precision, mean-recall and mean-F1. --------------------- Composite KB Evaluation (EVENT-SLOTS only): Scores based on confidence values and Average Precision (AP); k=3 justifications allowed: SUMMARY: This section provides summary of AP scores Metric RunID Hop AP SF-ALL-Macro STANFORD_KB_ENG_1 0 0.0000 SF-ALL-Macro STANFORD_KB_ENG_1 ALL 0.0000 LDC-MEAN-ALL-Macro STANFORD_KB_ENG_1 0 0.0000 LDC-MEAN-ALL-Macro STANFORD_KB_ENG_1 ALL 0.0000 (PRIMARY METRIC) *ALL-Macro AP refer to mean of corresponding AP values. Scores based on confidence values and Average Precision (AP); only k=1 justification allowed: SUMMARY: This section provides summary of AP scores Metric RunID Hop AP SF-ALL-Macro STANFORD_KB_ENG_1 0 0.0000 SF-ALL-Macro STANFORD_KB_ENG_1 ALL 0.0000 LDC-MEAN-ALL-Macro STANFORD_KB_ENG_1 0 0.0000 LDC-MEAN-ALL-Macro STANFORD_KB_ENG_1 ALL 0.0000 *ALL-Macro AP refer to mean of corresponding AP values. Scores based on P/R/F1; only k=1 justification allowed: Metric RunID Hop Prec Recall F1 SF-ALL-Micro STANFORD_KB_ENG_1 0 0.0000 0.0000 0.0000 SF-ALL-Micro STANFORD_KB_ENG_1 ALL 0.0000 0.0000 0.0000 SF-ALL-Macro STANFORD_KB_ENG_1 0 0.0000 0.0000 0.0000 SF-ALL-Macro STANFORD_KB_ENG_1 ALL 0.0000 0.0000 0.0000 LDC-MAX-ALL-Micro STANFORD_KB_ENG_1 0 0.0000 0.0000 0.0000 LDC-MAX-ALL-Micro STANFORD_KB_ENG_1 ALL 0.0000 0.0000 0.0000 LDC-MAX-ALL-Macro STANFORD_KB_ENG_1 0 0.0000 0.0000 0.0000 LDC-MAX-ALL-Macro STANFORD_KB_ENG_1 ALL 0.0000 0.0000 0.0000 LDC-MEAN-ALL-Macro STANFORD_KB_ENG_1 0 0.0000 0.0000 0.0000 LDC-MEAN-ALL-Macro STANFORD_KB_ENG_1 ALL 0.0000 0.0000 0.0000 *ALL-Macro Prec, Recall and F1 refer to mean-precision, mean-recall and mean-F1. --------------------- Composite KB Evaluation (SENTIMENT-SLOTS only): Scores based on confidence values and Average Precision (AP); k=3 justifications allowed: Metric RunID Hop AP SF-ALL-Macro STANFORD_KB_ENG_1 0 0.0000 SF-ALL-Macro STANFORD_KB_ENG_1 1 0.0000 SF-ALL-Macro STANFORD_KB_ENG_1 ALL 0.0000 LDC-MEAN-ALL-Macro STANFORD_KB_ENG_1 0 0.0000 LDC-MEAN-ALL-Macro STANFORD_KB_ENG_1 1 0.0000 LDC-MEAN-ALL-Macro STANFORD_KB_ENG_1 ALL 0.0000 (PRIMARY METRIC) *ALL-Macro AP refer to mean of corresponding AP values. Scores based on confidence values and Average Precision (AP); only k=1 justification allowed: Metric RunID Hop AP SF-ALL-Macro STANFORD_KB_ENG_1 0 0.0000 SF-ALL-Macro STANFORD_KB_ENG_1 1 0.0000 SF-ALL-Macro STANFORD_KB_ENG_1 ALL 0.0000 LDC-MEAN-ALL-Macro STANFORD_KB_ENG_1 0 0.0000 LDC-MEAN-ALL-Macro STANFORD_KB_ENG_1 1 0.0000 LDC-MEAN-ALL-Macro STANFORD_KB_ENG_1 ALL 0.0000 *ALL-Macro AP refer to mean of corresponding AP values. Scores based on P/R/F1; only k=1 justification allowed: Metric RunID Hop Prec Recall F1 SF-ALL-Micro STANFORD_KB_ENG_1 0 0.0000 0.0000 0.0000 SF-ALL-Micro STANFORD_KB_ENG_1 1 0.0000 0.0000 0.0000 SF-ALL-Micro STANFORD_KB_ENG_1 ALL 0.0000 0.0000 0.0000 SF-ALL-Macro STANFORD_KB_ENG_1 0 0.0000 0.0000 0.0000 SF-ALL-Macro STANFORD_KB_ENG_1 1 0.0000 0.0000 0.0000 SF-ALL-Macro STANFORD_KB_ENG_1 ALL 0.0000 0.0000 0.0000 LDC-MAX-ALL-Micro STANFORD_KB_ENG_1 0 0.0000 0.0000 0.0000 LDC-MAX-ALL-Micro STANFORD_KB_ENG_1 1 0.0000 0.0000 0.0000 LDC-MAX-ALL-Micro STANFORD_KB_ENG_1 ALL 0.0000 0.0000 0.0000 LDC-MAX-ALL-Macro STANFORD_KB_ENG_1 0 0.0000 0.0000 0.0000 LDC-MAX-ALL-Macro STANFORD_KB_ENG_1 1 0.0000 0.0000 0.0000 LDC-MAX-ALL-Macro STANFORD_KB_ENG_1 ALL 0.0000 0.0000 0.0000 LDC-MEAN-ALL-Macro STANFORD_KB_ENG_1 0 0.0000 0.0000 0.0000 LDC-MEAN-ALL-Macro STANFORD_KB_ENG_1 1 0.0000 0.0000 0.0000 LDC-MEAN-ALL-Macro STANFORD_KB_ENG_1 ALL 0.0000 0.0000 0.0000 *ALL-Macro Prec, Recall and F1 refer to mean-precision, mean-recall and mean-F1. ************************************************************* Run ID: STANFORD_KB_ENG_2 Did the run access the live Web during the evaluation window: No Did the run extract relations from the Cold Start source corpus: Yes Did the run generate meaningful confidence values: Yes Did the run attempt the full Entity Discovery and Linking task: No Did the run attempt the full Event Nugget Detection and Coreference task: No Did the run include SF relations: Yes Did the run attempt the Event Argument Extraction and Linking task: No Did the run include Sentiment relations: No --------------------- Composite KB Evaluation (All slot types): Scores based on confidence values and Average Precision (AP); k=3 justifications allowed: Metric RunID Hop AP SF-ALL-Macro STANFORD_KB_ENG_2 0 0.0919 SF-ALL-Macro STANFORD_KB_ENG_2 1 0.0154 SF-ALL-Macro STANFORD_KB_ENG_2 ALL 0.0665 LDC-MEAN-ALL-Macro STANFORD_KB_ENG_2 0 0.1064 LDC-MEAN-ALL-Macro STANFORD_KB_ENG_2 1 0.0193 LDC-MEAN-ALL-Macro STANFORD_KB_ENG_2 ALL 0.0803 (PRIMARY METRIC) *ALL-Macro AP refer to mean of corresponding AP values. Scores based on confidence values and Average Precision (AP); only k=1 justification allowed: Metric RunID Hop AP SF-ALL-Macro STANFORD_KB_ENG_2 0 0.1233 SF-ALL-Macro STANFORD_KB_ENG_2 1 0.0253 SF-ALL-Macro STANFORD_KB_ENG_2 ALL 0.0966 LDC-MEAN-ALL-Macro STANFORD_KB_ENG_2 0 0.1429 LDC-MEAN-ALL-Macro STANFORD_KB_ENG_2 1 0.0292 LDC-MEAN-ALL-Macro STANFORD_KB_ENG_2 ALL 0.1156 *ALL-Macro AP refer to mean of corresponding AP values. Scores based on P/R/F1; only k=1 justification allowed: Metric RunID Hop Prec Recall F1 SF-ALL-Micro STANFORD_KB_ENG_2 0 0.2086 0.1352 0.1641 SF-ALL-Micro STANFORD_KB_ENG_2 1 0.0318 0.1177 0.0500 SF-ALL-Micro STANFORD_KB_ENG_2 ALL 0.0977 0.1312 0.1120 SF-ALL-Macro STANFORD_KB_ENG_2 0 0.0900 0.1358 0.0938 SF-ALL-Macro STANFORD_KB_ENG_2 1 0.0911 0.1142 0.0921 SF-ALL-Macro STANFORD_KB_ENG_2 ALL 0.0903 0.1284 0.0932 LDC-MAX-ALL-Micro STANFORD_KB_ENG_2 0 0.2159 0.1477 0.1754 LDC-MAX-ALL-Micro STANFORD_KB_ENG_2 1 0.0302 0.1331 0.0493 LDC-MAX-ALL-Micro STANFORD_KB_ENG_2 ALL 0.0972 0.1445 0.1162 LDC-MAX-ALL-Macro STANFORD_KB_ENG_2 0 0.1134 0.1641 0.1173 LDC-MAX-ALL-Macro STANFORD_KB_ENG_2 1 0.1054 0.1350 0.1087 LDC-MAX-ALL-Macro STANFORD_KB_ENG_2 ALL 0.1106 0.1542 0.1144 LDC-MEAN-ALL-Macro STANFORD_KB_ENG_2 0 0.1057 0.1553 0.1100 LDC-MEAN-ALL-Macro STANFORD_KB_ENG_2 1 0.1054 0.1350 0.1087 LDC-MEAN-ALL-Macro STANFORD_KB_ENG_2 ALL 0.1056 0.1484 0.1096 *ALL-Macro Prec, Recall and F1 refer to mean-precision, mean-recall and mean-F1. --------------------- Composite KB Evaluation (SF-SLOTS only): Scores based on confidence values and Average Precision (AP); k=3 justifications allowed: Metric RunID Hop AP SF-ALL-Macro STANFORD_KB_ENG_2 0 0.2121 SF-ALL-Macro STANFORD_KB_ENG_2 1 0.0430 SF-ALL-Macro STANFORD_KB_ENG_2 ALL 0.1536 LDC-MEAN-ALL-Macro STANFORD_KB_ENG_2 0 0.2294 LDC-MEAN-ALL-Macro STANFORD_KB_ENG_2 1 0.0510 LDC-MEAN-ALL-Macro STANFORD_KB_ENG_2 ALL 0.1731 (PRIMARY METRIC) *ALL-Macro AP refer to mean of corresponding AP values. Scores based on confidence values and Average Precision (AP); only k=1 justification allowed: Metric RunID Hop AP SF-ALL-Macro STANFORD_KB_ENG_2 0 0.2846 SF-ALL-Macro STANFORD_KB_ENG_2 1 0.0708 SF-ALL-Macro STANFORD_KB_ENG_2 ALL 0.2231 LDC-MEAN-ALL-Macro STANFORD_KB_ENG_2 0 0.3081 LDC-MEAN-ALL-Macro STANFORD_KB_ENG_2 1 0.0770 LDC-MEAN-ALL-Macro STANFORD_KB_ENG_2 ALL 0.2492 *ALL-Macro AP refer to mean of corresponding AP values. Scores based on P/R/F1; only k=1 justification allowed: Metric RunID Hop Prec Recall F1 SF-ALL-Micro STANFORD_KB_ENG_2 0 0.2086 0.2729 0.2364 SF-ALL-Micro STANFORD_KB_ENG_2 1 0.0318 0.2213 0.0556 SF-ALL-Micro STANFORD_KB_ENG_2 ALL 0.0977 0.2605 0.1421 SF-ALL-Macro STANFORD_KB_ENG_2 0 0.2076 0.3134 0.2166 SF-ALL-Macro STANFORD_KB_ENG_2 1 0.1679 0.2106 0.1698 SF-ALL-Macro STANFORD_KB_ENG_2 ALL 0.1919 0.2726 0.1980 LDC-MAX-ALL-Micro STANFORD_KB_ENG_2 0 0.2214 0.2844 0.2490 LDC-MAX-ALL-Micro STANFORD_KB_ENG_2 1 0.0305 0.2342 0.0539 LDC-MAX-ALL-Micro STANFORD_KB_ENG_2 ALL 0.0986 0.2727 0.1448 LDC-MAX-ALL-Macro STANFORD_KB_ENG_2 0 0.2444 0.3538 0.2530 LDC-MAX-ALL-Macro STANFORD_KB_ENG_2 1 0.1860 0.2382 0.1918 LDC-MAX-ALL-Macro STANFORD_KB_ENG_2 ALL 0.2217 0.3089 0.2292 LDC-MEAN-ALL-Macro STANFORD_KB_ENG_2 0 0.2279 0.3348 0.2371 LDC-MEAN-ALL-Macro STANFORD_KB_ENG_2 1 0.1860 0.2382 0.1918 LDC-MEAN-ALL-Macro STANFORD_KB_ENG_2 ALL 0.2116 0.2973 0.2195 *ALL-Macro Prec, Recall and F1 refer to mean-precision, mean-recall and mean-F1. --------------------- Composite KB Evaluation (EVENT-SLOTS only): Scores based on confidence values and Average Precision (AP); k=3 justifications allowed: SUMMARY: This section provides summary of AP scores Metric RunID Hop AP SF-ALL-Macro STANFORD_KB_ENG_2 0 0.0000 SF-ALL-Macro STANFORD_KB_ENG_2 ALL 0.0000 LDC-MEAN-ALL-Macro STANFORD_KB_ENG_2 0 0.0000 LDC-MEAN-ALL-Macro STANFORD_KB_ENG_2 ALL 0.0000 (PRIMARY METRIC) *ALL-Macro AP refer to mean of corresponding AP values. Scores based on confidence values and Average Precision (AP); only k=1 justification allowed: SUMMARY: This section provides summary of AP scores Metric RunID Hop AP SF-ALL-Macro STANFORD_KB_ENG_2 0 0.0000 SF-ALL-Macro STANFORD_KB_ENG_2 ALL 0.0000 LDC-MEAN-ALL-Macro STANFORD_KB_ENG_2 0 0.0000 LDC-MEAN-ALL-Macro STANFORD_KB_ENG_2 ALL 0.0000 *ALL-Macro AP refer to mean of corresponding AP values. Scores based on P/R/F1; only k=1 justification allowed: Metric RunID Hop Prec Recall F1 SF-ALL-Micro STANFORD_KB_ENG_2 0 0.0000 0.0000 0.0000 SF-ALL-Micro STANFORD_KB_ENG_2 ALL 0.0000 0.0000 0.0000 SF-ALL-Macro STANFORD_KB_ENG_2 0 0.0000 0.0000 0.0000 SF-ALL-Macro STANFORD_KB_ENG_2 ALL 0.0000 0.0000 0.0000 LDC-MAX-ALL-Micro STANFORD_KB_ENG_2 0 0.0000 0.0000 0.0000 LDC-MAX-ALL-Micro STANFORD_KB_ENG_2 ALL 0.0000 0.0000 0.0000 LDC-MAX-ALL-Macro STANFORD_KB_ENG_2 0 0.0000 0.0000 0.0000 LDC-MAX-ALL-Macro STANFORD_KB_ENG_2 ALL 0.0000 0.0000 0.0000 LDC-MEAN-ALL-Macro STANFORD_KB_ENG_2 0 0.0000 0.0000 0.0000 LDC-MEAN-ALL-Macro STANFORD_KB_ENG_2 ALL 0.0000 0.0000 0.0000 *ALL-Macro Prec, Recall and F1 refer to mean-precision, mean-recall and mean-F1. --------------------- Composite KB Evaluation (SENTIMENT-SLOTS only): Scores based on confidence values and Average Precision (AP); k=3 justifications allowed: Metric RunID Hop AP SF-ALL-Macro STANFORD_KB_ENG_2 0 0.0000 SF-ALL-Macro STANFORD_KB_ENG_2 1 0.0000 SF-ALL-Macro STANFORD_KB_ENG_2 ALL 0.0000 LDC-MEAN-ALL-Macro STANFORD_KB_ENG_2 0 0.0000 LDC-MEAN-ALL-Macro STANFORD_KB_ENG_2 1 0.0000 LDC-MEAN-ALL-Macro STANFORD_KB_ENG_2 ALL 0.0000 (PRIMARY METRIC) *ALL-Macro AP refer to mean of corresponding AP values. Scores based on confidence values and Average Precision (AP); only k=1 justification allowed: Metric RunID Hop AP SF-ALL-Macro STANFORD_KB_ENG_2 0 0.0000 SF-ALL-Macro STANFORD_KB_ENG_2 1 0.0000 SF-ALL-Macro STANFORD_KB_ENG_2 ALL 0.0000 LDC-MEAN-ALL-Macro STANFORD_KB_ENG_2 0 0.0000 LDC-MEAN-ALL-Macro STANFORD_KB_ENG_2 1 0.0000 LDC-MEAN-ALL-Macro STANFORD_KB_ENG_2 ALL 0.0000 *ALL-Macro AP refer to mean of corresponding AP values. Scores based on P/R/F1; only k=1 justification allowed: Metric RunID Hop Prec Recall F1 SF-ALL-Micro STANFORD_KB_ENG_2 0 0.0000 0.0000 0.0000 SF-ALL-Micro STANFORD_KB_ENG_2 1 0.0000 0.0000 0.0000 SF-ALL-Micro STANFORD_KB_ENG_2 ALL 0.0000 0.0000 0.0000 SF-ALL-Macro STANFORD_KB_ENG_2 0 0.0000 0.0000 0.0000 SF-ALL-Macro STANFORD_KB_ENG_2 1 0.0000 0.0000 0.0000 SF-ALL-Macro STANFORD_KB_ENG_2 ALL 0.0000 0.0000 0.0000 LDC-MAX-ALL-Micro STANFORD_KB_ENG_2 0 0.0000 0.0000 0.0000 LDC-MAX-ALL-Micro STANFORD_KB_ENG_2 1 0.0000 0.0000 0.0000 LDC-MAX-ALL-Micro STANFORD_KB_ENG_2 ALL 0.0000 0.0000 0.0000 LDC-MAX-ALL-Macro STANFORD_KB_ENG_2 0 0.0000 0.0000 0.0000 LDC-MAX-ALL-Macro STANFORD_KB_ENG_2 1 0.0000 0.0000 0.0000 LDC-MAX-ALL-Macro STANFORD_KB_ENG_2 ALL 0.0000 0.0000 0.0000 LDC-MEAN-ALL-Macro STANFORD_KB_ENG_2 0 0.0000 0.0000 0.0000 LDC-MEAN-ALL-Macro STANFORD_KB_ENG_2 1 0.0000 0.0000 0.0000 LDC-MEAN-ALL-Macro STANFORD_KB_ENG_2 ALL 0.0000 0.0000 0.0000 *ALL-Macro Prec, Recall and F1 refer to mean-precision, mean-recall and mean-F1. ************************************************************* Run ID: STANFORD_KB_ENG_3 Did the run access the live Web during the evaluation window: No Did the run extract relations from the Cold Start source corpus: Yes Did the run generate meaningful confidence values: Yes Did the run attempt the full Entity Discovery and Linking task: No Did the run attempt the full Event Nugget Detection and Coreference task: No Did the run include SF relations: Yes Did the run attempt the Event Argument Extraction and Linking task: No Did the run include Sentiment relations: No --------------------- Composite KB Evaluation (All slot types): Scores based on confidence values and Average Precision (AP); k=3 justifications allowed: Metric RunID Hop AP SF-ALL-Macro STANFORD_KB_ENG_3 0 0.1002 SF-ALL-Macro STANFORD_KB_ENG_3 1 0.0265 SF-ALL-Macro STANFORD_KB_ENG_3 ALL 0.0792 LDC-MEAN-ALL-Macro STANFORD_KB_ENG_3 0 0.1177 LDC-MEAN-ALL-Macro STANFORD_KB_ENG_3 1 0.0331 LDC-MEAN-ALL-Macro STANFORD_KB_ENG_3 ALL 0.0936 (PRIMARY METRIC) *ALL-Macro AP refer to mean of corresponding AP values. Scores based on confidence values and Average Precision (AP); only k=1 justification allowed: Metric RunID Hop AP SF-ALL-Macro STANFORD_KB_ENG_3 0 0.1314 SF-ALL-Macro STANFORD_KB_ENG_3 1 0.0304 SF-ALL-Macro STANFORD_KB_ENG_3 ALL 0.1075 LDC-MEAN-ALL-Macro STANFORD_KB_ENG_3 0 0.1548 LDC-MEAN-ALL-Macro STANFORD_KB_ENG_3 1 0.0404 LDC-MEAN-ALL-Macro STANFORD_KB_ENG_3 ALL 0.1274 *ALL-Macro AP refer to mean of corresponding AP values. Scores based on P/R/F1; only k=1 justification allowed: Metric RunID Hop Prec Recall F1 SF-ALL-Micro STANFORD_KB_ENG_3 0 0.2293 0.1600 0.1885 SF-ALL-Micro STANFORD_KB_ENG_3 1 0.0376 0.1221 0.0575 SF-ALL-Micro STANFORD_KB_ENG_3 ALL 0.1185 0.1514 0.1330 SF-ALL-Macro STANFORD_KB_ENG_3 0 0.1040 0.1504 0.1122 SF-ALL-Macro STANFORD_KB_ENG_3 1 0.1047 0.1256 0.1048 SF-ALL-Macro STANFORD_KB_ENG_3 ALL 0.1043 0.1419 0.1097 LDC-MAX-ALL-Micro STANFORD_KB_ENG_3 0 0.2417 0.1734 0.2020 LDC-MAX-ALL-Micro STANFORD_KB_ENG_3 1 0.0418 0.1331 0.0636 LDC-MAX-ALL-Micro STANFORD_KB_ENG_3 ALL 0.1317 0.1647 0.1464 LDC-MAX-ALL-Macro STANFORD_KB_ENG_3 0 0.1294 0.1912 0.1420 LDC-MAX-ALL-Macro STANFORD_KB_ENG_3 1 0.1129 0.1435 0.1143 LDC-MAX-ALL-Macro STANFORD_KB_ENG_3 ALL 0.1238 0.1749 0.1325 LDC-MEAN-ALL-Macro STANFORD_KB_ENG_3 0 0.1212 0.1767 0.1323 LDC-MEAN-ALL-Macro STANFORD_KB_ENG_3 1 0.1137 0.1451 0.1165 LDC-MEAN-ALL-Macro STANFORD_KB_ENG_3 ALL 0.1187 0.1659 0.1269 *ALL-Macro Prec, Recall and F1 refer to mean-precision, mean-recall and mean-F1. --------------------- Composite KB Evaluation (SF-SLOTS only): Scores based on confidence values and Average Precision (AP); k=3 justifications allowed: Metric RunID Hop AP SF-ALL-Macro STANFORD_KB_ENG_3 0 0.2313 SF-ALL-Macro STANFORD_KB_ENG_3 1 0.0743 SF-ALL-Macro STANFORD_KB_ENG_3 ALL 0.1828 LDC-MEAN-ALL-Macro STANFORD_KB_ENG_3 0 0.2537 LDC-MEAN-ALL-Macro STANFORD_KB_ENG_3 1 0.0872 LDC-MEAN-ALL-Macro STANFORD_KB_ENG_3 ALL 0.2018 (PRIMARY METRIC) *ALL-Macro AP refer to mean of corresponding AP values. Scores based on confidence values and Average Precision (AP); only k=1 justification allowed: Metric RunID Hop AP SF-ALL-Macro STANFORD_KB_ENG_3 0 0.3034 SF-ALL-Macro STANFORD_KB_ENG_3 1 0.0851 SF-ALL-Macro STANFORD_KB_ENG_3 ALL 0.2480 LDC-MEAN-ALL-Macro STANFORD_KB_ENG_3 0 0.3336 LDC-MEAN-ALL-Macro STANFORD_KB_ENG_3 1 0.1065 LDC-MEAN-ALL-Macro STANFORD_KB_ENG_3 ALL 0.2746 *ALL-Macro AP refer to mean of corresponding AP values. Scores based on P/R/F1; only k=1 justification allowed: Metric RunID Hop Prec Recall F1 SF-ALL-Micro STANFORD_KB_ENG_3 0 0.2293 0.3230 0.2682 SF-ALL-Micro STANFORD_KB_ENG_3 1 0.0376 0.2295 0.0646 SF-ALL-Micro STANFORD_KB_ENG_3 ALL 0.1185 0.3005 0.1700 SF-ALL-Macro STANFORD_KB_ENG_3 0 0.2401 0.3472 0.2590 SF-ALL-Macro STANFORD_KB_ENG_3 1 0.1930 0.2316 0.1931 SF-ALL-Macro STANFORD_KB_ENG_3 ALL 0.2214 0.3014 0.2329 LDC-MAX-ALL-Micro STANFORD_KB_ENG_3 0 0.2401 0.3340 0.2793 LDC-MAX-ALL-Micro STANFORD_KB_ENG_3 1 0.0407 0.2342 0.0693 LDC-MAX-ALL-Micro STANFORD_KB_ENG_3 ALL 0.1293 0.3109 0.1827 LDC-MAX-ALL-Macro STANFORD_KB_ENG_3 0 0.2790 0.4121 0.3061 LDC-MAX-ALL-Macro STANFORD_KB_ENG_3 1 0.1991 0.2531 0.2017 LDC-MAX-ALL-Macro STANFORD_KB_ENG_3 ALL 0.2480 0.3504 0.2655 LDC-MEAN-ALL-Macro STANFORD_KB_ENG_3 0 0.2614 0.3810 0.2851 LDC-MEAN-ALL-Macro STANFORD_KB_ENG_3 1 0.2005 0.2560 0.2056 LDC-MEAN-ALL-Macro STANFORD_KB_ENG_3 ALL 0.2377 0.3325 0.2542 *ALL-Macro Prec, Recall and F1 refer to mean-precision, mean-recall and mean-F1. --------------------- Composite KB Evaluation (EVENT-SLOTS only): Scores based on confidence values and Average Precision (AP); k=3 justifications allowed: SUMMARY: This section provides summary of AP scores Metric RunID Hop AP SF-ALL-Macro STANFORD_KB_ENG_3 0 0.0000 SF-ALL-Macro STANFORD_KB_ENG_3 ALL 0.0000 LDC-MEAN-ALL-Macro STANFORD_KB_ENG_3 0 0.0000 LDC-MEAN-ALL-Macro STANFORD_KB_ENG_3 ALL 0.0000 (PRIMARY METRIC) *ALL-Macro AP refer to mean of corresponding AP values. Scores based on confidence values and Average Precision (AP); only k=1 justification allowed: SUMMARY: This section provides summary of AP scores Metric RunID Hop AP SF-ALL-Macro STANFORD_KB_ENG_3 0 0.0000 SF-ALL-Macro STANFORD_KB_ENG_3 ALL 0.0000 LDC-MEAN-ALL-Macro STANFORD_KB_ENG_3 0 0.0000 LDC-MEAN-ALL-Macro STANFORD_KB_ENG_3 ALL 0.0000 *ALL-Macro AP refer to mean of corresponding AP values. Scores based on P/R/F1; only k=1 justification allowed: Metric RunID Hop Prec Recall F1 SF-ALL-Micro STANFORD_KB_ENG_3 0 0.0000 0.0000 0.0000 SF-ALL-Micro STANFORD_KB_ENG_3 ALL 0.0000 0.0000 0.0000 SF-ALL-Macro STANFORD_KB_ENG_3 0 0.0000 0.0000 0.0000 SF-ALL-Macro STANFORD_KB_ENG_3 ALL 0.0000 0.0000 0.0000 LDC-MAX-ALL-Micro STANFORD_KB_ENG_3 0 0.0000 0.0000 0.0000 LDC-MAX-ALL-Micro STANFORD_KB_ENG_3 ALL 0.0000 0.0000 0.0000 LDC-MAX-ALL-Macro STANFORD_KB_ENG_3 0 0.0000 0.0000 0.0000 LDC-MAX-ALL-Macro STANFORD_KB_ENG_3 ALL 0.0000 0.0000 0.0000 LDC-MEAN-ALL-Macro STANFORD_KB_ENG_3 0 0.0000 0.0000 0.0000 LDC-MEAN-ALL-Macro STANFORD_KB_ENG_3 ALL 0.0000 0.0000 0.0000 *ALL-Macro Prec, Recall and F1 refer to mean-precision, mean-recall and mean-F1. --------------------- Composite KB Evaluation (SENTIMENT-SLOTS only): Scores based on confidence values and Average Precision (AP); k=3 justifications allowed: Metric RunID Hop AP SF-ALL-Macro STANFORD_KB_ENG_3 0 0.0000 SF-ALL-Macro STANFORD_KB_ENG_3 1 0.0000 SF-ALL-Macro STANFORD_KB_ENG_3 ALL 0.0000 LDC-MEAN-ALL-Macro STANFORD_KB_ENG_3 0 0.0000 LDC-MEAN-ALL-Macro STANFORD_KB_ENG_3 1 0.0000 LDC-MEAN-ALL-Macro STANFORD_KB_ENG_3 ALL 0.0000 (PRIMARY METRIC) *ALL-Macro AP refer to mean of corresponding AP values. Scores based on confidence values and Average Precision (AP); only k=1 justification allowed: Metric RunID Hop AP SF-ALL-Macro STANFORD_KB_ENG_3 0 0.0000 SF-ALL-Macro STANFORD_KB_ENG_3 1 0.0000 SF-ALL-Macro STANFORD_KB_ENG_3 ALL 0.0000 LDC-MEAN-ALL-Macro STANFORD_KB_ENG_3 0 0.0000 LDC-MEAN-ALL-Macro STANFORD_KB_ENG_3 1 0.0000 LDC-MEAN-ALL-Macro STANFORD_KB_ENG_3 ALL 0.0000 *ALL-Macro AP refer to mean of corresponding AP values. Scores based on P/R/F1; only k=1 justification allowed: Metric RunID Hop Prec Recall F1 SF-ALL-Micro STANFORD_KB_ENG_3 0 0.0000 0.0000 0.0000 SF-ALL-Micro STANFORD_KB_ENG_3 1 0.0000 0.0000 0.0000 SF-ALL-Micro STANFORD_KB_ENG_3 ALL 0.0000 0.0000 0.0000 SF-ALL-Macro STANFORD_KB_ENG_3 0 0.0000 0.0000 0.0000 SF-ALL-Macro STANFORD_KB_ENG_3 1 0.0000 0.0000 0.0000 SF-ALL-Macro STANFORD_KB_ENG_3 ALL 0.0000 0.0000 0.0000 LDC-MAX-ALL-Micro STANFORD_KB_ENG_3 0 0.0000 0.0000 0.0000 LDC-MAX-ALL-Micro STANFORD_KB_ENG_3 1 0.0000 0.0000 0.0000 LDC-MAX-ALL-Micro STANFORD_KB_ENG_3 ALL 0.0000 0.0000 0.0000 LDC-MAX-ALL-Macro STANFORD_KB_ENG_3 0 0.0000 0.0000 0.0000 LDC-MAX-ALL-Macro STANFORD_KB_ENG_3 1 0.0000 0.0000 0.0000 LDC-MAX-ALL-Macro STANFORD_KB_ENG_3 ALL 0.0000 0.0000 0.0000 LDC-MEAN-ALL-Macro STANFORD_KB_ENG_3 0 0.0000 0.0000 0.0000 LDC-MEAN-ALL-Macro STANFORD_KB_ENG_3 1 0.0000 0.0000 0.0000 LDC-MEAN-ALL-Macro STANFORD_KB_ENG_3 ALL 0.0000 0.0000 0.0000 *ALL-Macro Prec, Recall and F1 refer to mean-precision, mean-recall and mean-F1. ************************************************************* Run ID: STANFORD_KB_ENG_4 Did the run access the live Web during the evaluation window: Yes Did the run extract relations from the Cold Start source corpus: Yes Did the run generate meaningful confidence values: Yes Did the run attempt the full Entity Discovery and Linking task: No Did the run attempt the full Event Nugget Detection and Coreference task: No Did the run include SF relations: Yes Did the run attempt the Event Argument Extraction and Linking task: No Did the run include Sentiment relations: No --------------------- Composite KB Evaluation (All slot types): Scores based on confidence values and Average Precision (AP); k=3 justifications allowed: Metric RunID Hop AP SF-ALL-Macro STANFORD_KB_ENG_4 0 0.0841 SF-ALL-Macro STANFORD_KB_ENG_4 1 0.0132 SF-ALL-Macro STANFORD_KB_ENG_4 ALL 0.0677 LDC-MEAN-ALL-Macro STANFORD_KB_ENG_4 0 0.1015 LDC-MEAN-ALL-Macro STANFORD_KB_ENG_4 1 0.0174 LDC-MEAN-ALL-Macro STANFORD_KB_ENG_4 ALL 0.0832 (PRIMARY METRIC) *ALL-Macro AP refer to mean of corresponding AP values. Scores based on confidence values and Average Precision (AP); only k=1 justification allowed: Metric RunID Hop AP SF-ALL-Macro STANFORD_KB_ENG_4 0 0.1183 SF-ALL-Macro STANFORD_KB_ENG_4 1 0.0175 SF-ALL-Macro STANFORD_KB_ENG_4 ALL 0.1000 LDC-MEAN-ALL-Macro STANFORD_KB_ENG_4 0 0.1417 LDC-MEAN-ALL-Macro STANFORD_KB_ENG_4 1 0.0255 LDC-MEAN-ALL-Macro STANFORD_KB_ENG_4 ALL 0.1215 *ALL-Macro AP refer to mean of corresponding AP values. Scores based on P/R/F1; only k=1 justification allowed: Metric RunID Hop Prec Recall F1 SF-ALL-Micro STANFORD_KB_ENG_4 0 0.2180 0.1433 0.1729 SF-ALL-Micro STANFORD_KB_ENG_4 1 0.0303 0.0916 0.0455 SF-ALL-Micro STANFORD_KB_ENG_4 ALL 0.1101 0.1316 0.1199 SF-ALL-Macro STANFORD_KB_ENG_4 0 0.0966 0.1338 0.1005 SF-ALL-Macro STANFORD_KB_ENG_4 1 0.0849 0.0966 0.0820 SF-ALL-Macro STANFORD_KB_ENG_4 ALL 0.0926 0.1210 0.0941 LDC-MAX-ALL-Micro STANFORD_KB_ENG_4 0 0.2284 0.1596 0.1879 LDC-MAX-ALL-Micro STANFORD_KB_ENG_4 1 0.0354 0.1079 0.0533 LDC-MAX-ALL-Micro STANFORD_KB_ENG_4 ALL 0.1230 0.1484 0.1345 LDC-MAX-ALL-Macro STANFORD_KB_ENG_4 0 0.1222 0.1754 0.1304 LDC-MAX-ALL-Macro STANFORD_KB_ENG_4 1 0.0961 0.1150 0.0934 LDC-MAX-ALL-Macro STANFORD_KB_ENG_4 ALL 0.1133 0.1548 0.1178 LDC-MEAN-ALL-Macro STANFORD_KB_ENG_4 0 0.1131 0.1621 0.1206 LDC-MEAN-ALL-Macro STANFORD_KB_ENG_4 1 0.0927 0.1126 0.0920 LDC-MEAN-ALL-Macro STANFORD_KB_ENG_4 ALL 0.1061 0.1452 0.1108 *ALL-Macro Prec, Recall and F1 refer to mean-precision, mean-recall and mean-F1. --------------------- Composite KB Evaluation (SF-SLOTS only): Scores based on confidence values and Average Precision (AP); k=3 justifications allowed: Metric RunID Hop AP SF-ALL-Macro STANFORD_KB_ENG_4 0 0.1942 SF-ALL-Macro STANFORD_KB_ENG_4 1 0.0371 SF-ALL-Macro STANFORD_KB_ENG_4 ALL 0.1563 LDC-MEAN-ALL-Macro STANFORD_KB_ENG_4 0 0.2187 LDC-MEAN-ALL-Macro STANFORD_KB_ENG_4 1 0.0459 LDC-MEAN-ALL-Macro STANFORD_KB_ENG_4 ALL 0.1793 (PRIMARY METRIC) *ALL-Macro AP refer to mean of corresponding AP values. Scores based on confidence values and Average Precision (AP); only k=1 justification allowed: Metric RunID Hop AP SF-ALL-Macro STANFORD_KB_ENG_4 0 0.2731 SF-ALL-Macro STANFORD_KB_ENG_4 1 0.0490 SF-ALL-Macro STANFORD_KB_ENG_4 ALL 0.2309 LDC-MEAN-ALL-Macro STANFORD_KB_ENG_4 0 0.3054 LDC-MEAN-ALL-Macro STANFORD_KB_ENG_4 1 0.0673 LDC-MEAN-ALL-Macro STANFORD_KB_ENG_4 ALL 0.2619 *ALL-Macro AP refer to mean of corresponding AP values. Scores based on P/R/F1; only k=1 justification allowed: Metric RunID Hop Prec Recall F1 SF-ALL-Micro STANFORD_KB_ENG_4 0 0.2180 0.2893 0.2486 SF-ALL-Micro STANFORD_KB_ENG_4 1 0.0303 0.1721 0.0515 SF-ALL-Micro STANFORD_KB_ENG_4 ALL 0.1101 0.2612 0.1549 SF-ALL-Macro STANFORD_KB_ENG_4 0 0.2230 0.3088 0.2320 SF-ALL-Macro STANFORD_KB_ENG_4 1 0.1566 0.1781 0.1511 SF-ALL-Macro STANFORD_KB_ENG_4 ALL 0.1967 0.2570 0.1999 LDC-MAX-ALL-Micro STANFORD_KB_ENG_4 0 0.2287 0.3073 0.2622 LDC-MAX-ALL-Micro STANFORD_KB_ENG_4 1 0.0354 0.1899 0.0596 LDC-MAX-ALL-Micro STANFORD_KB_ENG_4 ALL 0.1231 0.2801 0.1710 LDC-MAX-ALL-Macro STANFORD_KB_ENG_4 0 0.2635 0.3782 0.2812 LDC-MAX-ALL-Macro STANFORD_KB_ENG_4 1 0.1695 0.2028 0.1648 LDC-MAX-ALL-Macro STANFORD_KB_ENG_4 ALL 0.2270 0.3101 0.2360 LDC-MEAN-ALL-Macro STANFORD_KB_ENG_4 0 0.2438 0.3495 0.2599 LDC-MEAN-ALL-Macro STANFORD_KB_ENG_4 1 0.1635 0.1986 0.1623 LDC-MEAN-ALL-Macro STANFORD_KB_ENG_4 ALL 0.2126 0.2909 0.2220 *ALL-Macro Prec, Recall and F1 refer to mean-precision, mean-recall and mean-F1. --------------------- Composite KB Evaluation (EVENT-SLOTS only): Scores based on confidence values and Average Precision (AP); k=3 justifications allowed: SUMMARY: This section provides summary of AP scores Metric RunID Hop AP SF-ALL-Macro STANFORD_KB_ENG_4 0 0.0000 SF-ALL-Macro STANFORD_KB_ENG_4 ALL 0.0000 LDC-MEAN-ALL-Macro STANFORD_KB_ENG_4 0 0.0000 LDC-MEAN-ALL-Macro STANFORD_KB_ENG_4 ALL 0.0000 (PRIMARY METRIC) *ALL-Macro AP refer to mean of corresponding AP values. Scores based on confidence values and Average Precision (AP); only k=1 justification allowed: SUMMARY: This section provides summary of AP scores Metric RunID Hop AP SF-ALL-Macro STANFORD_KB_ENG_4 0 0.0000 SF-ALL-Macro STANFORD_KB_ENG_4 ALL 0.0000 LDC-MEAN-ALL-Macro STANFORD_KB_ENG_4 0 0.0000 LDC-MEAN-ALL-Macro STANFORD_KB_ENG_4 ALL 0.0000 *ALL-Macro AP refer to mean of corresponding AP values. Scores based on P/R/F1; only k=1 justification allowed: Metric RunID Hop Prec Recall F1 SF-ALL-Micro STANFORD_KB_ENG_4 0 0.0000 0.0000 0.0000 SF-ALL-Micro STANFORD_KB_ENG_4 ALL 0.0000 0.0000 0.0000 SF-ALL-Macro STANFORD_KB_ENG_4 0 0.0000 0.0000 0.0000 SF-ALL-Macro STANFORD_KB_ENG_4 ALL 0.0000 0.0000 0.0000 LDC-MAX-ALL-Micro STANFORD_KB_ENG_4 0 0.0000 0.0000 0.0000 LDC-MAX-ALL-Micro STANFORD_KB_ENG_4 ALL 0.0000 0.0000 0.0000 LDC-MAX-ALL-Macro STANFORD_KB_ENG_4 0 0.0000 0.0000 0.0000 LDC-MAX-ALL-Macro STANFORD_KB_ENG_4 ALL 0.0000 0.0000 0.0000 LDC-MEAN-ALL-Macro STANFORD_KB_ENG_4 0 0.0000 0.0000 0.0000 LDC-MEAN-ALL-Macro STANFORD_KB_ENG_4 ALL 0.0000 0.0000 0.0000 *ALL-Macro Prec, Recall and F1 refer to mean-precision, mean-recall and mean-F1. --------------------- Composite KB Evaluation (SENTIMENT-SLOTS only): Scores based on confidence values and Average Precision (AP); k=3 justifications allowed: Metric RunID Hop AP SF-ALL-Macro STANFORD_KB_ENG_4 0 0.0000 SF-ALL-Macro STANFORD_KB_ENG_4 1 0.0000 SF-ALL-Macro STANFORD_KB_ENG_4 ALL 0.0000 LDC-MEAN-ALL-Macro STANFORD_KB_ENG_4 0 0.0000 LDC-MEAN-ALL-Macro STANFORD_KB_ENG_4 1 0.0000 LDC-MEAN-ALL-Macro STANFORD_KB_ENG_4 ALL 0.0000 (PRIMARY METRIC) *ALL-Macro AP refer to mean of corresponding AP values. Scores based on confidence values and Average Precision (AP); only k=1 justification allowed: Metric RunID Hop AP SF-ALL-Macro STANFORD_KB_ENG_4 0 0.0000 SF-ALL-Macro STANFORD_KB_ENG_4 1 0.0000 SF-ALL-Macro STANFORD_KB_ENG_4 ALL 0.0000 LDC-MEAN-ALL-Macro STANFORD_KB_ENG_4 0 0.0000 LDC-MEAN-ALL-Macro STANFORD_KB_ENG_4 1 0.0000 LDC-MEAN-ALL-Macro STANFORD_KB_ENG_4 ALL 0.0000 *ALL-Macro AP refer to mean of corresponding AP values. Scores based on P/R/F1; only k=1 justification allowed: Metric RunID Hop Prec Recall F1 SF-ALL-Micro STANFORD_KB_ENG_4 0 0.0000 0.0000 0.0000 SF-ALL-Micro STANFORD_KB_ENG_4 1 0.0000 0.0000 0.0000 SF-ALL-Micro STANFORD_KB_ENG_4 ALL 0.0000 0.0000 0.0000 SF-ALL-Macro STANFORD_KB_ENG_4 0 0.0000 0.0000 0.0000 SF-ALL-Macro STANFORD_KB_ENG_4 1 0.0000 0.0000 0.0000 SF-ALL-Macro STANFORD_KB_ENG_4 ALL 0.0000 0.0000 0.0000 LDC-MAX-ALL-Micro STANFORD_KB_ENG_4 0 0.0000 0.0000 0.0000 LDC-MAX-ALL-Micro STANFORD_KB_ENG_4 1 0.0000 0.0000 0.0000 LDC-MAX-ALL-Micro STANFORD_KB_ENG_4 ALL 0.0000 0.0000 0.0000 LDC-MAX-ALL-Macro STANFORD_KB_ENG_4 0 0.0000 0.0000 0.0000 LDC-MAX-ALL-Macro STANFORD_KB_ENG_4 1 0.0000 0.0000 0.0000 LDC-MAX-ALL-Macro STANFORD_KB_ENG_4 ALL 0.0000 0.0000 0.0000 LDC-MEAN-ALL-Macro STANFORD_KB_ENG_4 0 0.0000 0.0000 0.0000 LDC-MEAN-ALL-Macro STANFORD_KB_ENG_4 1 0.0000 0.0000 0.0000 LDC-MEAN-ALL-Macro STANFORD_KB_ENG_4 ALL 0.0000 0.0000 0.0000 *ALL-Macro Prec, Recall and F1 refer to mean-precision, mean-recall and mean-F1. ************************************************************* Run ID: STANFORD_KB_ENG_5 Did the run access the live Web during the evaluation window: No Did the run extract relations from the Cold Start source corpus: Yes Did the run generate meaningful confidence values: Yes Did the run attempt the full Entity Discovery and Linking task: No Did the run attempt the full Event Nugget Detection and Coreference task: No Did the run include SF relations: Yes Did the run attempt the Event Argument Extraction and Linking task: No Did the run include Sentiment relations: No --------------------- Composite KB Evaluation (All slot types): Scores based on confidence values and Average Precision (AP); k=3 justifications allowed: Metric RunID Hop AP SF-ALL-Macro STANFORD_KB_ENG_5 0 0.0846 SF-ALL-Macro STANFORD_KB_ENG_5 1 0.0126 SF-ALL-Macro STANFORD_KB_ENG_5 ALL 0.0673 LDC-MEAN-ALL-Macro STANFORD_KB_ENG_5 0 0.1028 LDC-MEAN-ALL-Macro STANFORD_KB_ENG_5 1 0.0166 LDC-MEAN-ALL-Macro STANFORD_KB_ENG_5 ALL 0.0833 (PRIMARY METRIC) *ALL-Macro AP refer to mean of corresponding AP values. Scores based on confidence values and Average Precision (AP); only k=1 justification allowed: Metric RunID Hop AP SF-ALL-Macro STANFORD_KB_ENG_5 0 0.1192 SF-ALL-Macro STANFORD_KB_ENG_5 1 0.0171 SF-ALL-Macro STANFORD_KB_ENG_5 ALL 0.0996 LDC-MEAN-ALL-Macro STANFORD_KB_ENG_5 0 0.1442 LDC-MEAN-ALL-Macro STANFORD_KB_ENG_5 1 0.0250 LDC-MEAN-ALL-Macro STANFORD_KB_ENG_5 ALL 0.1221 *ALL-Macro AP refer to mean of corresponding AP values. Scores based on P/R/F1; only k=1 justification allowed: Metric RunID Hop Prec Recall F1 SF-ALL-Micro STANFORD_KB_ENG_5 0 0.2181 0.1399 0.1705 SF-ALL-Micro STANFORD_KB_ENG_5 1 0.0290 0.0843 0.0431 SF-ALL-Micro STANFORD_KB_ENG_5 ALL 0.1099 0.1273 0.1180 SF-ALL-Macro STANFORD_KB_ENG_5 0 0.1001 0.1343 0.1028 SF-ALL-Macro STANFORD_KB_ENG_5 1 0.0777 0.0890 0.0762 SF-ALL-Macro STANFORD_KB_ENG_5 ALL 0.0924 0.1187 0.0936 LDC-MAX-ALL-Micro STANFORD_KB_ENG_5 0 0.2313 0.1566 0.1868 LDC-MAX-ALL-Micro STANFORD_KB_ENG_5 1 0.0315 0.0971 0.0476 LDC-MAX-ALL-Micro STANFORD_KB_ENG_5 ALL 0.1202 0.1437 0.1309 LDC-MAX-ALL-Macro STANFORD_KB_ENG_5 0 0.1270 0.1767 0.1339 LDC-MAX-ALL-Macro STANFORD_KB_ENG_5 1 0.0841 0.1025 0.0833 LDC-MAX-ALL-Macro STANFORD_KB_ENG_5 ALL 0.1123 0.1513 0.1166 LDC-MEAN-ALL-Macro STANFORD_KB_ENG_5 0 0.1179 0.1637 0.1243 LDC-MEAN-ALL-Macro STANFORD_KB_ENG_5 1 0.0851 0.1034 0.0854 LDC-MEAN-ALL-Macro STANFORD_KB_ENG_5 ALL 0.1067 0.1431 0.1110 *ALL-Macro Prec, Recall and F1 refer to mean-precision, mean-recall and mean-F1. --------------------- Composite KB Evaluation (SF-SLOTS only): Scores based on confidence values and Average Precision (AP); k=3 justifications allowed: Metric RunID Hop AP SF-ALL-Macro STANFORD_KB_ENG_5 0 0.1953 SF-ALL-Macro STANFORD_KB_ENG_5 1 0.0352 SF-ALL-Macro STANFORD_KB_ENG_5 ALL 0.1554 LDC-MEAN-ALL-Macro STANFORD_KB_ENG_5 0 0.2217 LDC-MEAN-ALL-Macro STANFORD_KB_ENG_5 1 0.0436 LDC-MEAN-ALL-Macro STANFORD_KB_ENG_5 ALL 0.1795 (PRIMARY METRIC) *ALL-Macro AP refer to mean of corresponding AP values. Scores based on confidence values and Average Precision (AP); only k=1 justification allowed: Metric RunID Hop AP SF-ALL-Macro STANFORD_KB_ENG_5 0 0.2751 SF-ALL-Macro STANFORD_KB_ENG_5 1 0.0479 SF-ALL-Macro STANFORD_KB_ENG_5 ALL 0.2300 LDC-MEAN-ALL-Macro STANFORD_KB_ENG_5 0 0.3109 LDC-MEAN-ALL-Macro STANFORD_KB_ENG_5 1 0.0659 LDC-MEAN-ALL-Macro STANFORD_KB_ENG_5 ALL 0.2633 *ALL-Macro AP refer to mean of corresponding AP values. Scores based on P/R/F1; only k=1 justification allowed: Metric RunID Hop Prec Recall F1 SF-ALL-Micro STANFORD_KB_ENG_5 0 0.2181 0.2824 0.2461 SF-ALL-Micro STANFORD_KB_ENG_5 1 0.0290 0.1585 0.0490 SF-ALL-Micro STANFORD_KB_ENG_5 ALL 0.1099 0.2526 0.1532 SF-ALL-Macro STANFORD_KB_ENG_5 0 0.2310 0.3101 0.2372 SF-ALL-Macro STANFORD_KB_ENG_5 1 0.1432 0.1641 0.1404 SF-ALL-Macro STANFORD_KB_ENG_5 ALL 0.1962 0.2522 0.1989 LDC-MAX-ALL-Micro STANFORD_KB_ENG_5 0 0.2324 0.3015 0.2625 LDC-MAX-ALL-Micro STANFORD_KB_ENG_5 1 0.0324 0.1709 0.0544 LDC-MAX-ALL-Micro STANFORD_KB_ENG_5 ALL 0.1222 0.2713 0.1685 LDC-MAX-ALL-Macro STANFORD_KB_ENG_5 0 0.2738 0.3809 0.2886 LDC-MAX-ALL-Macro STANFORD_KB_ENG_5 1 0.1483 0.1808 0.1469 LDC-MAX-ALL-Macro STANFORD_KB_ENG_5 ALL 0.2251 0.3032 0.2336 LDC-MEAN-ALL-Macro STANFORD_KB_ENG_5 0 0.2542 0.3530 0.2679 LDC-MEAN-ALL-Macro STANFORD_KB_ENG_5 1 0.1502 0.1824 0.1507 LDC-MEAN-ALL-Macro STANFORD_KB_ENG_5 ALL 0.2138 0.2867 0.2224 *ALL-Macro Prec, Recall and F1 refer to mean-precision, mean-recall and mean-F1. --------------------- Composite KB Evaluation (EVENT-SLOTS only): Scores based on confidence values and Average Precision (AP); k=3 justifications allowed: SUMMARY: This section provides summary of AP scores Metric RunID Hop AP SF-ALL-Macro STANFORD_KB_ENG_5 0 0.0000 SF-ALL-Macro STANFORD_KB_ENG_5 ALL 0.0000 LDC-MEAN-ALL-Macro STANFORD_KB_ENG_5 0 0.0000 LDC-MEAN-ALL-Macro STANFORD_KB_ENG_5 ALL 0.0000 (PRIMARY METRIC) *ALL-Macro AP refer to mean of corresponding AP values. Scores based on confidence values and Average Precision (AP); only k=1 justification allowed: SUMMARY: This section provides summary of AP scores Metric RunID Hop AP SF-ALL-Macro STANFORD_KB_ENG_5 0 0.0000 SF-ALL-Macro STANFORD_KB_ENG_5 ALL 0.0000 LDC-MEAN-ALL-Macro STANFORD_KB_ENG_5 0 0.0000 LDC-MEAN-ALL-Macro STANFORD_KB_ENG_5 ALL 0.0000 *ALL-Macro AP refer to mean of corresponding AP values. Scores based on P/R/F1; only k=1 justification allowed: Metric RunID Hop Prec Recall F1 SF-ALL-Micro STANFORD_KB_ENG_5 0 0.0000 0.0000 0.0000 SF-ALL-Micro STANFORD_KB_ENG_5 ALL 0.0000 0.0000 0.0000 SF-ALL-Macro STANFORD_KB_ENG_5 0 0.0000 0.0000 0.0000 SF-ALL-Macro STANFORD_KB_ENG_5 ALL 0.0000 0.0000 0.0000 LDC-MAX-ALL-Micro STANFORD_KB_ENG_5 0 0.0000 0.0000 0.0000 LDC-MAX-ALL-Micro STANFORD_KB_ENG_5 ALL 0.0000 0.0000 0.0000 LDC-MAX-ALL-Macro STANFORD_KB_ENG_5 0 0.0000 0.0000 0.0000 LDC-MAX-ALL-Macro STANFORD_KB_ENG_5 ALL 0.0000 0.0000 0.0000 LDC-MEAN-ALL-Macro STANFORD_KB_ENG_5 0 0.0000 0.0000 0.0000 LDC-MEAN-ALL-Macro STANFORD_KB_ENG_5 ALL 0.0000 0.0000 0.0000 *ALL-Macro Prec, Recall and F1 refer to mean-precision, mean-recall and mean-F1. --------------------- Composite KB Evaluation (SENTIMENT-SLOTS only): Scores based on confidence values and Average Precision (AP); k=3 justifications allowed: Metric RunID Hop AP SF-ALL-Macro STANFORD_KB_ENG_5 0 0.0000 SF-ALL-Macro STANFORD_KB_ENG_5 1 0.0000 SF-ALL-Macro STANFORD_KB_ENG_5 ALL 0.0000 LDC-MEAN-ALL-Macro STANFORD_KB_ENG_5 0 0.0000 LDC-MEAN-ALL-Macro STANFORD_KB_ENG_5 1 0.0000 LDC-MEAN-ALL-Macro STANFORD_KB_ENG_5 ALL 0.0000 (PRIMARY METRIC) *ALL-Macro AP refer to mean of corresponding AP values. Scores based on confidence values and Average Precision (AP); only k=1 justification allowed: Metric RunID Hop AP SF-ALL-Macro STANFORD_KB_ENG_5 0 0.0000 SF-ALL-Macro STANFORD_KB_ENG_5 1 0.0000 SF-ALL-Macro STANFORD_KB_ENG_5 ALL 0.0000 LDC-MEAN-ALL-Macro STANFORD_KB_ENG_5 0 0.0000 LDC-MEAN-ALL-Macro STANFORD_KB_ENG_5 1 0.0000 LDC-MEAN-ALL-Macro STANFORD_KB_ENG_5 ALL 0.0000 *ALL-Macro AP refer to mean of corresponding AP values. Scores based on P/R/F1; only k=1 justification allowed: Metric RunID Hop Prec Recall F1 SF-ALL-Micro STANFORD_KB_ENG_5 0 0.0000 0.0000 0.0000 SF-ALL-Micro STANFORD_KB_ENG_5 1 0.0000 0.0000 0.0000 SF-ALL-Micro STANFORD_KB_ENG_5 ALL 0.0000 0.0000 0.0000 SF-ALL-Macro STANFORD_KB_ENG_5 0 0.0000 0.0000 0.0000 SF-ALL-Macro STANFORD_KB_ENG_5 1 0.0000 0.0000 0.0000 SF-ALL-Macro STANFORD_KB_ENG_5 ALL 0.0000 0.0000 0.0000 LDC-MAX-ALL-Micro STANFORD_KB_ENG_5 0 0.0000 0.0000 0.0000 LDC-MAX-ALL-Micro STANFORD_KB_ENG_5 1 0.0000 0.0000 0.0000 LDC-MAX-ALL-Micro STANFORD_KB_ENG_5 ALL 0.0000 0.0000 0.0000 LDC-MAX-ALL-Macro STANFORD_KB_ENG_5 0 0.0000 0.0000 0.0000 LDC-MAX-ALL-Macro STANFORD_KB_ENG_5 1 0.0000 0.0000 0.0000 LDC-MAX-ALL-Macro STANFORD_KB_ENG_5 ALL 0.0000 0.0000 0.0000 LDC-MEAN-ALL-Macro STANFORD_KB_ENG_5 0 0.0000 0.0000 0.0000 LDC-MEAN-ALL-Macro STANFORD_KB_ENG_5 1 0.0000 0.0000 0.0000 LDC-MEAN-ALL-Macro STANFORD_KB_ENG_5 ALL 0.0000 0.0000 0.0000 *ALL-Macro Prec, Recall and F1 refer to mean-precision, mean-recall and mean-F1.