===================================================================== TAC KBP 2017 CHINESE KB CONSTRUCTION: COMPOSITE KB EVALUATION RESULTS ===================================================================== Team ID: STANFORD Organization: Stanford University ************************************************************* Run ID: STANFORD_KB_CMN_1 Did the run access the live Web during the evaluation window: No Did the run extract relations from the Cold Start source corpus: Yes Did the run generate meaningful confidence values: Yes Did the run attempt the full Entity Discovery and Linking task: No Did the run attempt the full Event Nugget Detection and Coreference task: No Did the run include SF relations: Yes Did the run attempt the Event Argument Extraction and Linking task: No Did the run include Sentiment relations: No --------------------- Composite KB Evaluation (All slot types): Scores based on confidence values and Average Precision (AP); k=3 justifications allowed: Metric RunID Hop AP SF-ALL-Macro STANFORD_KB_CMN_1 0 0.0741 SF-ALL-Macro STANFORD_KB_CMN_1 1 0.0125 SF-ALL-Macro STANFORD_KB_CMN_1 ALL 0.0551 LDC-MEAN-ALL-Macro STANFORD_KB_CMN_1 0 0.0795 LDC-MEAN-ALL-Macro STANFORD_KB_CMN_1 1 0.0098 LDC-MEAN-ALL-Macro STANFORD_KB_CMN_1 ALL 0.0601 (PRIMARY METRIC) *ALL-Macro AP refer to mean of corresponding AP values. Scores based on confidence values and Average Precision (AP); only k=1 justification allowed: Metric RunID Hop AP SF-ALL-Macro STANFORD_KB_CMN_1 0 0.1107 SF-ALL-Macro STANFORD_KB_CMN_1 1 0.0227 SF-ALL-Macro STANFORD_KB_CMN_1 ALL 0.0847 LDC-MEAN-ALL-Macro STANFORD_KB_CMN_1 0 0.1133 LDC-MEAN-ALL-Macro STANFORD_KB_CMN_1 1 0.0167 LDC-MEAN-ALL-Macro STANFORD_KB_CMN_1 ALL 0.0903 *ALL-Macro AP refer to mean of corresponding AP values. Scores based on P/R/F1; only k=1 justification allowed: Metric RunID Hop Prec Recall F1 SF-ALL-Micro STANFORD_KB_CMN_1 0 0.5135 0.0701 0.1233 SF-ALL-Micro STANFORD_KB_CMN_1 1 0.2558 0.0299 0.0536 SF-ALL-Micro STANFORD_KB_CMN_1 ALL 0.4416 0.0576 0.1019 SF-ALL-Macro STANFORD_KB_CMN_1 0 0.1251 0.1134 0.1152 SF-ALL-Macro STANFORD_KB_CMN_1 1 0.0307 0.0313 0.0280 SF-ALL-Macro STANFORD_KB_CMN_1 ALL 0.0899 0.0828 0.0828 LDC-MAX-ALL-Micro STANFORD_KB_CMN_1 0 0.5212 0.0833 0.1437 LDC-MAX-ALL-Micro STANFORD_KB_CMN_1 1 0.2500 0.0312 0.0556 LDC-MAX-ALL-Micro STANFORD_KB_CMN_1 ALL 0.4489 0.0668 0.1163 LDC-MAX-ALL-Macro STANFORD_KB_CMN_1 0 0.1392 0.1245 0.1272 LDC-MAX-ALL-Macro STANFORD_KB_CMN_1 1 0.0376 0.0424 0.0373 LDC-MAX-ALL-Macro STANFORD_KB_CMN_1 ALL 0.1037 0.0958 0.0957 LDC-MEAN-ALL-Macro STANFORD_KB_CMN_1 0 0.1322 0.1172 0.1197 LDC-MEAN-ALL-Macro STANFORD_KB_CMN_1 1 0.0337 0.0389 0.0337 LDC-MEAN-ALL-Macro STANFORD_KB_CMN_1 ALL 0.0977 0.0898 0.0896 *ALL-Macro Prec, Recall and F1 refer to mean-precision, mean-recall and mean-F1. --------------------- Composite KB Evaluation (SF-SLOTS only): Scores based on confidence values and Average Precision (AP); k=3 justifications allowed: Metric RunID Hop AP SF-ALL-Macro STANFORD_KB_CMN_1 0 0.1540 SF-ALL-Macro STANFORD_KB_CMN_1 1 0.0249 SF-ALL-Macro STANFORD_KB_CMN_1 ALL 0.1145 LDC-MEAN-ALL-Macro STANFORD_KB_CMN_1 0 0.1618 LDC-MEAN-ALL-Macro STANFORD_KB_CMN_1 1 0.0193 LDC-MEAN-ALL-Macro STANFORD_KB_CMN_1 ALL 0.1224 (PRIMARY METRIC) *ALL-Macro AP refer to mean of corresponding AP values. Scores based on confidence values and Average Precision (AP); only k=1 justification allowed: Metric RunID Hop AP SF-ALL-Macro STANFORD_KB_CMN_1 0 0.2302 SF-ALL-Macro STANFORD_KB_CMN_1 1 0.0450 SF-ALL-Macro STANFORD_KB_CMN_1 ALL 0.1761 LDC-MEAN-ALL-Macro STANFORD_KB_CMN_1 0 0.2308 LDC-MEAN-ALL-Macro STANFORD_KB_CMN_1 1 0.0331 LDC-MEAN-ALL-Macro STANFORD_KB_CMN_1 ALL 0.1838 *ALL-Macro AP refer to mean of corresponding AP values. Scores based on P/R/F1; only k=1 justification allowed: Metric RunID Hop Prec Recall F1 SF-ALL-Micro STANFORD_KB_CMN_1 0 0.5135 0.1836 0.2705 SF-ALL-Micro STANFORD_KB_CMN_1 1 0.2558 0.1215 0.1648 SF-ALL-Micro STANFORD_KB_CMN_1 ALL 0.4416 0.1696 0.2450 SF-ALL-Macro STANFORD_KB_CMN_1 0 0.2600 0.2357 0.2396 SF-ALL-Macro STANFORD_KB_CMN_1 1 0.0732 0.0748 0.0669 SF-ALL-Macro STANFORD_KB_CMN_1 ALL 0.1964 0.1809 0.1808 LDC-MAX-ALL-Micro STANFORD_KB_CMN_1 0 0.5181 0.1903 0.2783 LDC-MAX-ALL-Micro STANFORD_KB_CMN_1 1 0.2500 0.1316 0.1724 LDC-MAX-ALL-Micro STANFORD_KB_CMN_1 ALL 0.4469 0.1784 0.2551 LDC-MAX-ALL-Macro STANFORD_KB_CMN_1 0 0.2834 0.2535 0.2589 LDC-MAX-ALL-Macro STANFORD_KB_CMN_1 1 0.0737 0.0831 0.0731 LDC-MAX-ALL-Macro STANFORD_KB_CMN_1 ALL 0.2083 0.1925 0.1924 LDC-MEAN-ALL-Macro STANFORD_KB_CMN_1 0 0.2691 0.2386 0.2438 LDC-MEAN-ALL-Macro STANFORD_KB_CMN_1 1 0.0661 0.0762 0.0660 LDC-MEAN-ALL-Macro STANFORD_KB_CMN_1 ALL 0.1964 0.1805 0.1801 *ALL-Macro Prec, Recall and F1 refer to mean-precision, mean-recall and mean-F1. --------------------- Composite KB Evaluation (EVENT-SLOTS only): Scores based on confidence values and Average Precision (AP); k=3 justifications allowed: SUMMARY: This section provides summary of AP scores Metric RunID Hop AP SF-ALL-Macro STANFORD_KB_CMN_1 0 0.0000 SF-ALL-Macro STANFORD_KB_CMN_1 ALL 0.0000 LDC-MEAN-ALL-Macro STANFORD_KB_CMN_1 0 0.0000 LDC-MEAN-ALL-Macro STANFORD_KB_CMN_1 ALL 0.0000 (PRIMARY METRIC) *ALL-Macro AP refer to mean of corresponding AP values. Scores based on confidence values and Average Precision (AP); only k=1 justification allowed: SUMMARY: This section provides summary of AP scores Metric RunID Hop AP SF-ALL-Macro STANFORD_KB_CMN_1 0 0.0000 SF-ALL-Macro STANFORD_KB_CMN_1 ALL 0.0000 LDC-MEAN-ALL-Macro STANFORD_KB_CMN_1 0 0.0000 LDC-MEAN-ALL-Macro STANFORD_KB_CMN_1 ALL 0.0000 *ALL-Macro AP refer to mean of corresponding AP values. Scores based on P/R/F1; only k=1 justification allowed: Metric RunID Hop Prec Recall F1 SF-ALL-Micro STANFORD_KB_CMN_1 0 0.0000 0.0000 0.0000 SF-ALL-Micro STANFORD_KB_CMN_1 ALL 0.0000 0.0000 0.0000 SF-ALL-Macro STANFORD_KB_CMN_1 0 0.0000 0.0000 0.0000 SF-ALL-Macro STANFORD_KB_CMN_1 ALL 0.0000 0.0000 0.0000 LDC-MAX-ALL-Micro STANFORD_KB_CMN_1 0 0.0000 0.0000 0.0000 LDC-MAX-ALL-Micro STANFORD_KB_CMN_1 ALL 0.0000 0.0000 0.0000 LDC-MAX-ALL-Macro STANFORD_KB_CMN_1 0 0.0000 0.0000 0.0000 LDC-MAX-ALL-Macro STANFORD_KB_CMN_1 ALL 0.0000 0.0000 0.0000 LDC-MEAN-ALL-Macro STANFORD_KB_CMN_1 0 0.0000 0.0000 0.0000 LDC-MEAN-ALL-Macro STANFORD_KB_CMN_1 ALL 0.0000 0.0000 0.0000 *ALL-Macro Prec, Recall and F1 refer to mean-precision, mean-recall and mean-F1. --------------------- Composite KB Evaluation (SENTIMENT-SLOTS only): Scores based on confidence values and Average Precision (AP); k=3 justifications allowed: Metric RunID Hop AP SF-ALL-Macro STANFORD_KB_CMN_1 0 0.0000 SF-ALL-Macro STANFORD_KB_CMN_1 1 0.0000 SF-ALL-Macro STANFORD_KB_CMN_1 ALL 0.0000 LDC-MEAN-ALL-Macro STANFORD_KB_CMN_1 0 0.0000 LDC-MEAN-ALL-Macro STANFORD_KB_CMN_1 1 0.0000 LDC-MEAN-ALL-Macro STANFORD_KB_CMN_1 ALL 0.0000 (PRIMARY METRIC) *ALL-Macro AP refer to mean of corresponding AP values. Scores based on confidence values and Average Precision (AP); only k=1 justification allowed: Metric RunID Hop AP SF-ALL-Macro STANFORD_KB_CMN_1 0 0.0000 SF-ALL-Macro STANFORD_KB_CMN_1 1 0.0000 SF-ALL-Macro STANFORD_KB_CMN_1 ALL 0.0000 LDC-MEAN-ALL-Macro STANFORD_KB_CMN_1 0 0.0000 LDC-MEAN-ALL-Macro STANFORD_KB_CMN_1 1 0.0000 LDC-MEAN-ALL-Macro STANFORD_KB_CMN_1 ALL 0.0000 *ALL-Macro AP refer to mean of corresponding AP values. Scores based on P/R/F1; only k=1 justification allowed: Metric RunID Hop Prec Recall F1 SF-ALL-Micro STANFORD_KB_CMN_1 0 0.0000 0.0000 0.0000 SF-ALL-Micro STANFORD_KB_CMN_1 1 0.0000 0.0000 0.0000 SF-ALL-Micro STANFORD_KB_CMN_1 ALL 0.0000 0.0000 0.0000 SF-ALL-Macro STANFORD_KB_CMN_1 0 0.0000 0.0000 0.0000 SF-ALL-Macro STANFORD_KB_CMN_1 1 0.0000 0.0000 0.0000 SF-ALL-Macro STANFORD_KB_CMN_1 ALL 0.0000 0.0000 0.0000 LDC-MAX-ALL-Micro STANFORD_KB_CMN_1 0 0.0000 0.0000 0.0000 LDC-MAX-ALL-Micro STANFORD_KB_CMN_1 1 0.0000 0.0000 0.0000 LDC-MAX-ALL-Micro STANFORD_KB_CMN_1 ALL 0.0000 0.0000 0.0000 LDC-MAX-ALL-Macro STANFORD_KB_CMN_1 0 0.0000 0.0000 0.0000 LDC-MAX-ALL-Macro STANFORD_KB_CMN_1 1 0.0000 0.0000 0.0000 LDC-MAX-ALL-Macro STANFORD_KB_CMN_1 ALL 0.0000 0.0000 0.0000 LDC-MEAN-ALL-Macro STANFORD_KB_CMN_1 0 0.0000 0.0000 0.0000 LDC-MEAN-ALL-Macro STANFORD_KB_CMN_1 1 0.0000 0.0000 0.0000 LDC-MEAN-ALL-Macro STANFORD_KB_CMN_1 ALL 0.0000 0.0000 0.0000 *ALL-Macro Prec, Recall and F1 refer to mean-precision, mean-recall and mean-F1. ************************************************************* Run ID: STANFORD_KB_CMN_2 Did the run access the live Web during the evaluation window: No Did the run extract relations from the Cold Start source corpus: Yes Did the run generate meaningful confidence values: Yes Did the run attempt the full Entity Discovery and Linking task: No Did the run attempt the full Event Nugget Detection and Coreference task: No Did the run include SF relations: Yes Did the run attempt the Event Argument Extraction and Linking task: No Did the run include Sentiment relations: No --------------------- Composite KB Evaluation (All slot types): Scores based on confidence values and Average Precision (AP); k=3 justifications allowed: Metric RunID Hop AP SF-ALL-Macro STANFORD_KB_CMN_2 0 0.0434 SF-ALL-Macro STANFORD_KB_CMN_2 1 0.0084 SF-ALL-Macro STANFORD_KB_CMN_2 ALL 0.0303 LDC-MEAN-ALL-Macro STANFORD_KB_CMN_2 0 0.0399 LDC-MEAN-ALL-Macro STANFORD_KB_CMN_2 1 0.0075 LDC-MEAN-ALL-Macro STANFORD_KB_CMN_2 ALL 0.0288 (PRIMARY METRIC) *ALL-Macro AP refer to mean of corresponding AP values. Scores based on confidence values and Average Precision (AP); only k=1 justification allowed: Metric RunID Hop AP SF-ALL-Macro STANFORD_KB_CMN_2 0 0.0735 SF-ALL-Macro STANFORD_KB_CMN_2 1 0.0094 SF-ALL-Macro STANFORD_KB_CMN_2 ALL 0.0506 LDC-MEAN-ALL-Macro STANFORD_KB_CMN_2 0 0.0673 LDC-MEAN-ALL-Macro STANFORD_KB_CMN_2 1 0.0080 LDC-MEAN-ALL-Macro STANFORD_KB_CMN_2 ALL 0.0502 *ALL-Macro AP refer to mean of corresponding AP values. Scores based on P/R/F1; only k=1 justification allowed: Metric RunID Hop Prec Recall F1 SF-ALL-Micro STANFORD_KB_CMN_2 0 0.5562 0.0578 0.1047 SF-ALL-Micro STANFORD_KB_CMN_2 1 0.2449 0.0163 0.0306 SF-ALL-Micro STANFORD_KB_CMN_2 ALL 0.4862 0.0449 0.0822 SF-ALL-Macro STANFORD_KB_CMN_2 0 0.0873 0.0759 0.0785 SF-ALL-Macro STANFORD_KB_CMN_2 1 0.0153 0.0153 0.0142 SF-ALL-Macro STANFORD_KB_CMN_2 ALL 0.0605 0.0534 0.0546 LDC-MAX-ALL-Micro STANFORD_KB_CMN_2 0 0.5000 0.0552 0.0995 LDC-MAX-ALL-Micro STANFORD_KB_CMN_2 1 0.2381 0.0208 0.0383 LDC-MAX-ALL-Micro STANFORD_KB_CMN_2 ALL 0.4295 0.0443 0.0803 LDC-MAX-ALL-Macro STANFORD_KB_CMN_2 0 0.0916 0.0754 0.0795 LDC-MAX-ALL-Macro STANFORD_KB_CMN_2 1 0.0224 0.0255 0.0229 LDC-MAX-ALL-Macro STANFORD_KB_CMN_2 ALL 0.0674 0.0580 0.0597 LDC-MEAN-ALL-Macro STANFORD_KB_CMN_2 0 0.0862 0.0700 0.0740 LDC-MEAN-ALL-Macro STANFORD_KB_CMN_2 1 0.0190 0.0222 0.0196 LDC-MEAN-ALL-Macro STANFORD_KB_CMN_2 ALL 0.0627 0.0533 0.0550 *ALL-Macro Prec, Recall and F1 refer to mean-precision, mean-recall and mean-F1. --------------------- Composite KB Evaluation (SF-SLOTS only): Scores based on confidence values and Average Precision (AP); k=3 justifications allowed: Metric RunID Hop AP SF-ALL-Macro STANFORD_KB_CMN_2 0 0.0903 SF-ALL-Macro STANFORD_KB_CMN_2 1 0.0166 SF-ALL-Macro STANFORD_KB_CMN_2 ALL 0.0629 LDC-MEAN-ALL-Macro STANFORD_KB_CMN_2 0 0.0813 LDC-MEAN-ALL-Macro STANFORD_KB_CMN_2 1 0.0148 LDC-MEAN-ALL-Macro STANFORD_KB_CMN_2 ALL 0.0586 (PRIMARY METRIC) *ALL-Macro AP refer to mean of corresponding AP values. Scores based on confidence values and Average Precision (AP); only k=1 justification allowed: Metric RunID Hop AP SF-ALL-Macro STANFORD_KB_CMN_2 0 0.1528 SF-ALL-Macro STANFORD_KB_CMN_2 1 0.0186 SF-ALL-Macro STANFORD_KB_CMN_2 ALL 0.1052 LDC-MEAN-ALL-Macro STANFORD_KB_CMN_2 0 0.1370 LDC-MEAN-ALL-Macro STANFORD_KB_CMN_2 1 0.0159 LDC-MEAN-ALL-Macro STANFORD_KB_CMN_2 ALL 0.1022 *ALL-Macro AP refer to mean of corresponding AP values. Scores based on P/R/F1; only k=1 justification allowed: Metric RunID Hop Prec Recall F1 SF-ALL-Micro STANFORD_KB_CMN_2 0 0.5562 0.1514 0.2380 SF-ALL-Micro STANFORD_KB_CMN_2 1 0.2449 0.0663 0.1043 SF-ALL-Micro STANFORD_KB_CMN_2 ALL 0.4862 0.1322 0.2078 SF-ALL-Macro STANFORD_KB_CMN_2 0 0.1814 0.1578 0.1633 SF-ALL-Macro STANFORD_KB_CMN_2 1 0.0364 0.0365 0.0339 SF-ALL-Macro STANFORD_KB_CMN_2 ALL 0.1320 0.1165 0.1192 LDC-MAX-ALL-Micro STANFORD_KB_CMN_2 0 0.4957 0.1261 0.2011 LDC-MAX-ALL-Micro STANFORD_KB_CMN_2 1 0.2326 0.0877 0.1274 LDC-MAX-ALL-Micro STANFORD_KB_CMN_2 ALL 0.4241 0.1184 0.1851 LDC-MAX-ALL-Macro STANFORD_KB_CMN_2 0 0.1866 0.1536 0.1619 LDC-MAX-ALL-Macro STANFORD_KB_CMN_2 1 0.0438 0.0500 0.0449 LDC-MAX-ALL-Macro STANFORD_KB_CMN_2 ALL 0.1355 0.1165 0.1200 LDC-MEAN-ALL-Macro STANFORD_KB_CMN_2 0 0.1755 0.1424 0.1508 LDC-MEAN-ALL-Macro STANFORD_KB_CMN_2 1 0.0373 0.0435 0.0384 LDC-MEAN-ALL-Macro STANFORD_KB_CMN_2 ALL 0.1260 0.1070 0.1105 *ALL-Macro Prec, Recall and F1 refer to mean-precision, mean-recall and mean-F1. --------------------- Composite KB Evaluation (EVENT-SLOTS only): Scores based on confidence values and Average Precision (AP); k=3 justifications allowed: SUMMARY: This section provides summary of AP scores Metric RunID Hop AP SF-ALL-Macro STANFORD_KB_CMN_2 0 0.0000 SF-ALL-Macro STANFORD_KB_CMN_2 ALL 0.0000 LDC-MEAN-ALL-Macro STANFORD_KB_CMN_2 0 0.0000 LDC-MEAN-ALL-Macro STANFORD_KB_CMN_2 ALL 0.0000 (PRIMARY METRIC) *ALL-Macro AP refer to mean of corresponding AP values. Scores based on confidence values and Average Precision (AP); only k=1 justification allowed: SUMMARY: This section provides summary of AP scores Metric RunID Hop AP SF-ALL-Macro STANFORD_KB_CMN_2 0 0.0000 SF-ALL-Macro STANFORD_KB_CMN_2 ALL 0.0000 LDC-MEAN-ALL-Macro STANFORD_KB_CMN_2 0 0.0000 LDC-MEAN-ALL-Macro STANFORD_KB_CMN_2 ALL 0.0000 *ALL-Macro AP refer to mean of corresponding AP values. Scores based on P/R/F1; only k=1 justification allowed: Metric RunID Hop Prec Recall F1 SF-ALL-Micro STANFORD_KB_CMN_2 0 0.0000 0.0000 0.0000 SF-ALL-Micro STANFORD_KB_CMN_2 ALL 0.0000 0.0000 0.0000 SF-ALL-Macro STANFORD_KB_CMN_2 0 0.0000 0.0000 0.0000 SF-ALL-Macro STANFORD_KB_CMN_2 ALL 0.0000 0.0000 0.0000 LDC-MAX-ALL-Micro STANFORD_KB_CMN_2 0 0.0000 0.0000 0.0000 LDC-MAX-ALL-Micro STANFORD_KB_CMN_2 ALL 0.0000 0.0000 0.0000 LDC-MAX-ALL-Macro STANFORD_KB_CMN_2 0 0.0000 0.0000 0.0000 LDC-MAX-ALL-Macro STANFORD_KB_CMN_2 ALL 0.0000 0.0000 0.0000 LDC-MEAN-ALL-Macro STANFORD_KB_CMN_2 0 0.0000 0.0000 0.0000 LDC-MEAN-ALL-Macro STANFORD_KB_CMN_2 ALL 0.0000 0.0000 0.0000 *ALL-Macro Prec, Recall and F1 refer to mean-precision, mean-recall and mean-F1. --------------------- Composite KB Evaluation (SENTIMENT-SLOTS only): Scores based on confidence values and Average Precision (AP); k=3 justifications allowed: Metric RunID Hop AP SF-ALL-Macro STANFORD_KB_CMN_2 0 0.0000 SF-ALL-Macro STANFORD_KB_CMN_2 1 0.0000 SF-ALL-Macro STANFORD_KB_CMN_2 ALL 0.0000 LDC-MEAN-ALL-Macro STANFORD_KB_CMN_2 0 0.0000 LDC-MEAN-ALL-Macro STANFORD_KB_CMN_2 1 0.0000 LDC-MEAN-ALL-Macro STANFORD_KB_CMN_2 ALL 0.0000 (PRIMARY METRIC) *ALL-Macro AP refer to mean of corresponding AP values. Scores based on confidence values and Average Precision (AP); only k=1 justification allowed: Metric RunID Hop AP SF-ALL-Macro STANFORD_KB_CMN_2 0 0.0000 SF-ALL-Macro STANFORD_KB_CMN_2 1 0.0000 SF-ALL-Macro STANFORD_KB_CMN_2 ALL 0.0000 LDC-MEAN-ALL-Macro STANFORD_KB_CMN_2 0 0.0000 LDC-MEAN-ALL-Macro STANFORD_KB_CMN_2 1 0.0000 LDC-MEAN-ALL-Macro STANFORD_KB_CMN_2 ALL 0.0000 *ALL-Macro AP refer to mean of corresponding AP values. Scores based on P/R/F1; only k=1 justification allowed: Metric RunID Hop Prec Recall F1 SF-ALL-Micro STANFORD_KB_CMN_2 0 0.0000 0.0000 0.0000 SF-ALL-Micro STANFORD_KB_CMN_2 1 0.0000 0.0000 0.0000 SF-ALL-Micro STANFORD_KB_CMN_2 ALL 0.0000 0.0000 0.0000 SF-ALL-Macro STANFORD_KB_CMN_2 0 0.0000 0.0000 0.0000 SF-ALL-Macro STANFORD_KB_CMN_2 1 0.0000 0.0000 0.0000 SF-ALL-Macro STANFORD_KB_CMN_2 ALL 0.0000 0.0000 0.0000 LDC-MAX-ALL-Micro STANFORD_KB_CMN_2 0 0.0000 0.0000 0.0000 LDC-MAX-ALL-Micro STANFORD_KB_CMN_2 1 0.0000 0.0000 0.0000 LDC-MAX-ALL-Micro STANFORD_KB_CMN_2 ALL 0.0000 0.0000 0.0000 LDC-MAX-ALL-Macro STANFORD_KB_CMN_2 0 0.0000 0.0000 0.0000 LDC-MAX-ALL-Macro STANFORD_KB_CMN_2 1 0.0000 0.0000 0.0000 LDC-MAX-ALL-Macro STANFORD_KB_CMN_2 ALL 0.0000 0.0000 0.0000 LDC-MEAN-ALL-Macro STANFORD_KB_CMN_2 0 0.0000 0.0000 0.0000 LDC-MEAN-ALL-Macro STANFORD_KB_CMN_2 1 0.0000 0.0000 0.0000 LDC-MEAN-ALL-Macro STANFORD_KB_CMN_2 ALL 0.0000 0.0000 0.0000 *ALL-Macro Prec, Recall and F1 refer to mean-precision, mean-recall and mean-F1. ************************************************************* Run ID: STANFORD_KB_CMN_3 Did the run access the live Web during the evaluation window: No Did the run extract relations from the Cold Start source corpus: Yes Did the run generate meaningful confidence values: Yes Did the run attempt the full Entity Discovery and Linking task: No Did the run attempt the full Event Nugget Detection and Coreference task: No Did the run include SF relations: Yes Did the run attempt the Event Argument Extraction and Linking task: No Did the run include Sentiment relations: No --------------------- Composite KB Evaluation (All slot types): Scores based on confidence values and Average Precision (AP); k=3 justifications allowed: Metric RunID Hop AP SF-ALL-Macro STANFORD_KB_CMN_3 0 0.0000 SF-ALL-Macro STANFORD_KB_CMN_3 1 0.0000 SF-ALL-Macro STANFORD_KB_CMN_3 ALL 0.0000 LDC-MEAN-ALL-Macro STANFORD_KB_CMN_3 0 0.0000 LDC-MEAN-ALL-Macro STANFORD_KB_CMN_3 1 0.0000 LDC-MEAN-ALL-Macro STANFORD_KB_CMN_3 ALL 0.0000 (PRIMARY METRIC) *ALL-Macro AP refer to mean of corresponding AP values. Scores based on confidence values and Average Precision (AP); only k=1 justification allowed: Metric RunID Hop AP SF-ALL-Macro STANFORD_KB_CMN_3 0 0.0000 SF-ALL-Macro STANFORD_KB_CMN_3 1 0.0000 SF-ALL-Macro STANFORD_KB_CMN_3 ALL 0.0000 LDC-MEAN-ALL-Macro STANFORD_KB_CMN_3 0 0.0000 LDC-MEAN-ALL-Macro STANFORD_KB_CMN_3 1 0.0000 LDC-MEAN-ALL-Macro STANFORD_KB_CMN_3 ALL 0.0000 *ALL-Macro AP refer to mean of corresponding AP values. Scores based on P/R/F1; only k=1 justification allowed: Metric RunID Hop Prec Recall F1 SF-ALL-Micro STANFORD_KB_CMN_3 0 0.0000 0.0000 0.0000 SF-ALL-Micro STANFORD_KB_CMN_3 1 0.0000 0.0000 0.0000 SF-ALL-Micro STANFORD_KB_CMN_3 ALL 0.0000 0.0000 0.0000 SF-ALL-Macro STANFORD_KB_CMN_3 0 0.0000 0.0000 0.0000 SF-ALL-Macro STANFORD_KB_CMN_3 1 0.0000 0.0000 0.0000 SF-ALL-Macro STANFORD_KB_CMN_3 ALL 0.0000 0.0000 0.0000 LDC-MAX-ALL-Micro STANFORD_KB_CMN_3 0 0.0000 0.0000 0.0000 LDC-MAX-ALL-Micro STANFORD_KB_CMN_3 1 0.0000 0.0000 0.0000 LDC-MAX-ALL-Micro STANFORD_KB_CMN_3 ALL 0.0000 0.0000 0.0000 LDC-MAX-ALL-Macro STANFORD_KB_CMN_3 0 0.0000 0.0000 0.0000 LDC-MAX-ALL-Macro STANFORD_KB_CMN_3 1 0.0000 0.0000 0.0000 LDC-MAX-ALL-Macro STANFORD_KB_CMN_3 ALL 0.0000 0.0000 0.0000 LDC-MEAN-ALL-Macro STANFORD_KB_CMN_3 0 0.0000 0.0000 0.0000 LDC-MEAN-ALL-Macro STANFORD_KB_CMN_3 1 0.0000 0.0000 0.0000 LDC-MEAN-ALL-Macro STANFORD_KB_CMN_3 ALL 0.0000 0.0000 0.0000 *ALL-Macro Prec, Recall and F1 refer to mean-precision, mean-recall and mean-F1. --------------------- Composite KB Evaluation (SF-SLOTS only): Scores based on confidence values and Average Precision (AP); k=3 justifications allowed: Metric RunID Hop AP SF-ALL-Macro STANFORD_KB_CMN_3 0 0.0000 SF-ALL-Macro STANFORD_KB_CMN_3 1 0.0000 SF-ALL-Macro STANFORD_KB_CMN_3 ALL 0.0000 LDC-MEAN-ALL-Macro STANFORD_KB_CMN_3 0 0.0000 LDC-MEAN-ALL-Macro STANFORD_KB_CMN_3 1 0.0000 LDC-MEAN-ALL-Macro STANFORD_KB_CMN_3 ALL 0.0000 (PRIMARY METRIC) *ALL-Macro AP refer to mean of corresponding AP values. Scores based on confidence values and Average Precision (AP); only k=1 justification allowed: Metric RunID Hop AP SF-ALL-Macro STANFORD_KB_CMN_3 0 0.0000 SF-ALL-Macro STANFORD_KB_CMN_3 1 0.0000 SF-ALL-Macro STANFORD_KB_CMN_3 ALL 0.0000 LDC-MEAN-ALL-Macro STANFORD_KB_CMN_3 0 0.0000 LDC-MEAN-ALL-Macro STANFORD_KB_CMN_3 1 0.0000 LDC-MEAN-ALL-Macro STANFORD_KB_CMN_3 ALL 0.0000 *ALL-Macro AP refer to mean of corresponding AP values. Scores based on P/R/F1; only k=1 justification allowed: Metric RunID Hop Prec Recall F1 SF-ALL-Micro STANFORD_KB_CMN_3 0 0.0000 0.0000 0.0000 SF-ALL-Micro STANFORD_KB_CMN_3 1 0.0000 0.0000 0.0000 SF-ALL-Micro STANFORD_KB_CMN_3 ALL 0.0000 0.0000 0.0000 SF-ALL-Macro STANFORD_KB_CMN_3 0 0.0000 0.0000 0.0000 SF-ALL-Macro STANFORD_KB_CMN_3 1 0.0000 0.0000 0.0000 SF-ALL-Macro STANFORD_KB_CMN_3 ALL 0.0000 0.0000 0.0000 LDC-MAX-ALL-Micro STANFORD_KB_CMN_3 0 0.0000 0.0000 0.0000 LDC-MAX-ALL-Micro STANFORD_KB_CMN_3 1 0.0000 0.0000 0.0000 LDC-MAX-ALL-Micro STANFORD_KB_CMN_3 ALL 0.0000 0.0000 0.0000 LDC-MAX-ALL-Macro STANFORD_KB_CMN_3 0 0.0000 0.0000 0.0000 LDC-MAX-ALL-Macro STANFORD_KB_CMN_3 1 0.0000 0.0000 0.0000 LDC-MAX-ALL-Macro STANFORD_KB_CMN_3 ALL 0.0000 0.0000 0.0000 LDC-MEAN-ALL-Macro STANFORD_KB_CMN_3 0 0.0000 0.0000 0.0000 LDC-MEAN-ALL-Macro STANFORD_KB_CMN_3 1 0.0000 0.0000 0.0000 LDC-MEAN-ALL-Macro STANFORD_KB_CMN_3 ALL 0.0000 0.0000 0.0000 *ALL-Macro Prec, Recall and F1 refer to mean-precision, mean-recall and mean-F1. --------------------- Composite KB Evaluation (EVENT-SLOTS only): Scores based on confidence values and Average Precision (AP); k=3 justifications allowed: SUMMARY: This section provides summary of AP scores Metric RunID Hop AP SF-ALL-Macro STANFORD_KB_CMN_3 0 0.0000 SF-ALL-Macro STANFORD_KB_CMN_3 ALL 0.0000 LDC-MEAN-ALL-Macro STANFORD_KB_CMN_3 0 0.0000 LDC-MEAN-ALL-Macro STANFORD_KB_CMN_3 ALL 0.0000 (PRIMARY METRIC) *ALL-Macro AP refer to mean of corresponding AP values. Scores based on confidence values and Average Precision (AP); only k=1 justification allowed: SUMMARY: This section provides summary of AP scores Metric RunID Hop AP SF-ALL-Macro STANFORD_KB_CMN_3 0 0.0000 SF-ALL-Macro STANFORD_KB_CMN_3 ALL 0.0000 LDC-MEAN-ALL-Macro STANFORD_KB_CMN_3 0 0.0000 LDC-MEAN-ALL-Macro STANFORD_KB_CMN_3 ALL 0.0000 *ALL-Macro AP refer to mean of corresponding AP values. Scores based on P/R/F1; only k=1 justification allowed: Metric RunID Hop Prec Recall F1 SF-ALL-Micro STANFORD_KB_CMN_3 0 0.0000 0.0000 0.0000 SF-ALL-Micro STANFORD_KB_CMN_3 ALL 0.0000 0.0000 0.0000 SF-ALL-Macro STANFORD_KB_CMN_3 0 0.0000 0.0000 0.0000 SF-ALL-Macro STANFORD_KB_CMN_3 ALL 0.0000 0.0000 0.0000 LDC-MAX-ALL-Micro STANFORD_KB_CMN_3 0 0.0000 0.0000 0.0000 LDC-MAX-ALL-Micro STANFORD_KB_CMN_3 ALL 0.0000 0.0000 0.0000 LDC-MAX-ALL-Macro STANFORD_KB_CMN_3 0 0.0000 0.0000 0.0000 LDC-MAX-ALL-Macro STANFORD_KB_CMN_3 ALL 0.0000 0.0000 0.0000 LDC-MEAN-ALL-Macro STANFORD_KB_CMN_3 0 0.0000 0.0000 0.0000 LDC-MEAN-ALL-Macro STANFORD_KB_CMN_3 ALL 0.0000 0.0000 0.0000 *ALL-Macro Prec, Recall and F1 refer to mean-precision, mean-recall and mean-F1. --------------------- Composite KB Evaluation (SENTIMENT-SLOTS only): Scores based on confidence values and Average Precision (AP); k=3 justifications allowed: Metric RunID Hop AP SF-ALL-Macro STANFORD_KB_CMN_3 0 0.0000 SF-ALL-Macro STANFORD_KB_CMN_3 1 0.0000 SF-ALL-Macro STANFORD_KB_CMN_3 ALL 0.0000 LDC-MEAN-ALL-Macro STANFORD_KB_CMN_3 0 0.0000 LDC-MEAN-ALL-Macro STANFORD_KB_CMN_3 1 0.0000 LDC-MEAN-ALL-Macro STANFORD_KB_CMN_3 ALL 0.0000 (PRIMARY METRIC) *ALL-Macro AP refer to mean of corresponding AP values. Scores based on confidence values and Average Precision (AP); only k=1 justification allowed: Metric RunID Hop AP SF-ALL-Macro STANFORD_KB_CMN_3 0 0.0000 SF-ALL-Macro STANFORD_KB_CMN_3 1 0.0000 SF-ALL-Macro STANFORD_KB_CMN_3 ALL 0.0000 LDC-MEAN-ALL-Macro STANFORD_KB_CMN_3 0 0.0000 LDC-MEAN-ALL-Macro STANFORD_KB_CMN_3 1 0.0000 LDC-MEAN-ALL-Macro STANFORD_KB_CMN_3 ALL 0.0000 *ALL-Macro AP refer to mean of corresponding AP values. Scores based on P/R/F1; only k=1 justification allowed: Metric RunID Hop Prec Recall F1 SF-ALL-Micro STANFORD_KB_CMN_3 0 0.0000 0.0000 0.0000 SF-ALL-Micro STANFORD_KB_CMN_3 1 0.0000 0.0000 0.0000 SF-ALL-Micro STANFORD_KB_CMN_3 ALL 0.0000 0.0000 0.0000 SF-ALL-Macro STANFORD_KB_CMN_3 0 0.0000 0.0000 0.0000 SF-ALL-Macro STANFORD_KB_CMN_3 1 0.0000 0.0000 0.0000 SF-ALL-Macro STANFORD_KB_CMN_3 ALL 0.0000 0.0000 0.0000 LDC-MAX-ALL-Micro STANFORD_KB_CMN_3 0 0.0000 0.0000 0.0000 LDC-MAX-ALL-Micro STANFORD_KB_CMN_3 1 0.0000 0.0000 0.0000 LDC-MAX-ALL-Micro STANFORD_KB_CMN_3 ALL 0.0000 0.0000 0.0000 LDC-MAX-ALL-Macro STANFORD_KB_CMN_3 0 0.0000 0.0000 0.0000 LDC-MAX-ALL-Macro STANFORD_KB_CMN_3 1 0.0000 0.0000 0.0000 LDC-MAX-ALL-Macro STANFORD_KB_CMN_3 ALL 0.0000 0.0000 0.0000 LDC-MEAN-ALL-Macro STANFORD_KB_CMN_3 0 0.0000 0.0000 0.0000 LDC-MEAN-ALL-Macro STANFORD_KB_CMN_3 1 0.0000 0.0000 0.0000 LDC-MEAN-ALL-Macro STANFORD_KB_CMN_3 ALL 0.0000 0.0000 0.0000 *ALL-Macro Prec, Recall and F1 refer to mean-precision, mean-recall and mean-F1. ************************************************************* Run ID: STANFORD_KB_CMN_4 Did the run access the live Web during the evaluation window: No Did the run extract relations from the Cold Start source corpus: Yes Did the run generate meaningful confidence values: Yes Did the run attempt the full Entity Discovery and Linking task: No Did the run attempt the full Event Nugget Detection and Coreference task: No Did the run include SF relations: Yes Did the run attempt the Event Argument Extraction and Linking task: No Did the run include Sentiment relations: No --------------------- Composite KB Evaluation (All slot types): Scores based on confidence values and Average Precision (AP); k=3 justifications allowed: Metric RunID Hop AP SF-ALL-Macro STANFORD_KB_CMN_4 0 0.0710 SF-ALL-Macro STANFORD_KB_CMN_4 1 0.0018 SF-ALL-Macro STANFORD_KB_CMN_4 ALL 0.0546 LDC-MEAN-ALL-Macro STANFORD_KB_CMN_4 0 0.0757 LDC-MEAN-ALL-Macro STANFORD_KB_CMN_4 1 0.0027 LDC-MEAN-ALL-Macro STANFORD_KB_CMN_4 ALL 0.0609 (PRIMARY METRIC) *ALL-Macro AP refer to mean of corresponding AP values. Scores based on confidence values and Average Precision (AP); only k=1 justification allowed: Metric RunID Hop AP SF-ALL-Macro STANFORD_KB_CMN_4 0 0.0989 SF-ALL-Macro STANFORD_KB_CMN_4 1 0.0024 SF-ALL-Macro STANFORD_KB_CMN_4 ALL 0.0748 LDC-MEAN-ALL-Macro STANFORD_KB_CMN_4 0 0.1015 LDC-MEAN-ALL-Macro STANFORD_KB_CMN_4 1 0.0033 LDC-MEAN-ALL-Macro STANFORD_KB_CMN_4 ALL 0.0824 *ALL-Macro AP refer to mean of corresponding AP values. Scores based on P/R/F1; only k=1 justification allowed: Metric RunID Hop Prec Recall F1 SF-ALL-Micro STANFORD_KB_CMN_4 0 0.5188 0.0424 0.0784 SF-ALL-Micro STANFORD_KB_CMN_4 1 0.1270 0.0109 0.0201 SF-ALL-Micro STANFORD_KB_CMN_4 ALL 0.3929 0.0326 0.0602 SF-ALL-Macro STANFORD_KB_CMN_4 0 0.1135 0.1000 0.1030 SF-ALL-Macro STANFORD_KB_CMN_4 1 0.0100 0.0121 0.0092 SF-ALL-Macro STANFORD_KB_CMN_4 ALL 0.0750 0.0673 0.0681 LDC-MAX-ALL-Micro STANFORD_KB_CMN_4 0 0.4898 0.0465 0.0850 LDC-MAX-ALL-Micro STANFORD_KB_CMN_4 1 0.1750 0.0146 0.0269 LDC-MAX-ALL-Micro STANFORD_KB_CMN_4 ALL 0.3986 0.0364 0.0667 LDC-MAX-ALL-Macro STANFORD_KB_CMN_4 0 0.1266 0.1109 0.1144 LDC-MAX-ALL-Macro STANFORD_KB_CMN_4 1 0.0153 0.0205 0.0152 LDC-MAX-ALL-Macro STANFORD_KB_CMN_4 ALL 0.0877 0.0793 0.0797 LDC-MEAN-ALL-Macro STANFORD_KB_CMN_4 0 0.1171 0.1027 0.1060 LDC-MEAN-ALL-Macro STANFORD_KB_CMN_4 1 0.0146 0.0203 0.0148 LDC-MEAN-ALL-Macro STANFORD_KB_CMN_4 ALL 0.0813 0.0739 0.0742 *ALL-Macro Prec, Recall and F1 refer to mean-precision, mean-recall and mean-F1. --------------------- Composite KB Evaluation (SF-SLOTS only): Scores based on confidence values and Average Precision (AP); k=3 justifications allowed: Metric RunID Hop AP SF-ALL-Macro STANFORD_KB_CMN_4 0 0.1477 SF-ALL-Macro STANFORD_KB_CMN_4 1 0.0037 SF-ALL-Macro STANFORD_KB_CMN_4 ALL 0.1135 LDC-MEAN-ALL-Macro STANFORD_KB_CMN_4 0 0.1541 LDC-MEAN-ALL-Macro STANFORD_KB_CMN_4 1 0.0054 LDC-MEAN-ALL-Macro STANFORD_KB_CMN_4 ALL 0.1239 (PRIMARY METRIC) *ALL-Macro AP refer to mean of corresponding AP values. Scores based on confidence values and Average Precision (AP); only k=1 justification allowed: Metric RunID Hop AP SF-ALL-Macro STANFORD_KB_CMN_4 0 0.2056 SF-ALL-Macro STANFORD_KB_CMN_4 1 0.0048 SF-ALL-Macro STANFORD_KB_CMN_4 ALL 0.1554 LDC-MEAN-ALL-Macro STANFORD_KB_CMN_4 0 0.2066 LDC-MEAN-ALL-Macro STANFORD_KB_CMN_4 1 0.0065 LDC-MEAN-ALL-Macro STANFORD_KB_CMN_4 ALL 0.1677 *ALL-Macro AP refer to mean of corresponding AP values. Scores based on P/R/F1; only k=1 justification allowed: Metric RunID Hop Prec Recall F1 SF-ALL-Micro STANFORD_KB_CMN_4 0 0.5188 0.1111 0.1830 SF-ALL-Micro STANFORD_KB_CMN_4 1 0.1270 0.0442 0.0656 SF-ALL-Micro STANFORD_KB_CMN_4 ALL 0.3929 0.0960 0.1543 SF-ALL-Macro STANFORD_KB_CMN_4 0 0.2360 0.2078 0.2142 SF-ALL-Macro STANFORD_KB_CMN_4 1 0.0238 0.0288 0.0220 SF-ALL-Macro STANFORD_KB_CMN_4 ALL 0.1638 0.1469 0.1488 LDC-MAX-ALL-Micro STANFORD_KB_CMN_4 0 0.4948 0.1062 0.1749 LDC-MAX-ALL-Micro STANFORD_KB_CMN_4 1 0.1228 0.0614 0.0819 LDC-MAX-ALL-Micro STANFORD_KB_CMN_4 ALL 0.3571 0.0972 0.1528 LDC-MAX-ALL-Macro STANFORD_KB_CMN_4 0 0.2579 0.2257 0.2330 LDC-MAX-ALL-Macro STANFORD_KB_CMN_4 1 0.0300 0.0403 0.0298 LDC-MAX-ALL-Macro STANFORD_KB_CMN_4 ALL 0.1762 0.1593 0.1602 LDC-MEAN-ALL-Macro STANFORD_KB_CMN_4 0 0.2385 0.2091 0.2159 LDC-MEAN-ALL-Macro STANFORD_KB_CMN_4 1 0.0285 0.0398 0.0291 LDC-MEAN-ALL-Macro STANFORD_KB_CMN_4 ALL 0.1633 0.1485 0.1490 *ALL-Macro Prec, Recall and F1 refer to mean-precision, mean-recall and mean-F1. --------------------- Composite KB Evaluation (EVENT-SLOTS only): Scores based on confidence values and Average Precision (AP); k=3 justifications allowed: SUMMARY: This section provides summary of AP scores Metric RunID Hop AP SF-ALL-Macro STANFORD_KB_CMN_4 0 0.0000 SF-ALL-Macro STANFORD_KB_CMN_4 ALL 0.0000 LDC-MEAN-ALL-Macro STANFORD_KB_CMN_4 0 0.0000 LDC-MEAN-ALL-Macro STANFORD_KB_CMN_4 ALL 0.0000 (PRIMARY METRIC) *ALL-Macro AP refer to mean of corresponding AP values. Scores based on confidence values and Average Precision (AP); only k=1 justification allowed: SUMMARY: This section provides summary of AP scores Metric RunID Hop AP SF-ALL-Macro STANFORD_KB_CMN_4 0 0.0000 SF-ALL-Macro STANFORD_KB_CMN_4 ALL 0.0000 LDC-MEAN-ALL-Macro STANFORD_KB_CMN_4 0 0.0000 LDC-MEAN-ALL-Macro STANFORD_KB_CMN_4 ALL 0.0000 *ALL-Macro AP refer to mean of corresponding AP values. Scores based on P/R/F1; only k=1 justification allowed: Metric RunID Hop Prec Recall F1 SF-ALL-Micro STANFORD_KB_CMN_4 0 0.0000 0.0000 0.0000 SF-ALL-Micro STANFORD_KB_CMN_4 ALL 0.0000 0.0000 0.0000 SF-ALL-Macro STANFORD_KB_CMN_4 0 0.0000 0.0000 0.0000 SF-ALL-Macro STANFORD_KB_CMN_4 ALL 0.0000 0.0000 0.0000 LDC-MAX-ALL-Micro STANFORD_KB_CMN_4 0 0.0000 0.0000 0.0000 LDC-MAX-ALL-Micro STANFORD_KB_CMN_4 ALL 0.0000 0.0000 0.0000 LDC-MAX-ALL-Macro STANFORD_KB_CMN_4 0 0.0000 0.0000 0.0000 LDC-MAX-ALL-Macro STANFORD_KB_CMN_4 ALL 0.0000 0.0000 0.0000 LDC-MEAN-ALL-Macro STANFORD_KB_CMN_4 0 0.0000 0.0000 0.0000 LDC-MEAN-ALL-Macro STANFORD_KB_CMN_4 ALL 0.0000 0.0000 0.0000 *ALL-Macro Prec, Recall and F1 refer to mean-precision, mean-recall and mean-F1. --------------------- Composite KB Evaluation (SENTIMENT-SLOTS only): Scores based on confidence values and Average Precision (AP); k=3 justifications allowed: Metric RunID Hop AP SF-ALL-Macro STANFORD_KB_CMN_4 0 0.0000 SF-ALL-Macro STANFORD_KB_CMN_4 1 0.0000 SF-ALL-Macro STANFORD_KB_CMN_4 ALL 0.0000 LDC-MEAN-ALL-Macro STANFORD_KB_CMN_4 0 0.0000 LDC-MEAN-ALL-Macro STANFORD_KB_CMN_4 1 0.0000 LDC-MEAN-ALL-Macro STANFORD_KB_CMN_4 ALL 0.0000 (PRIMARY METRIC) *ALL-Macro AP refer to mean of corresponding AP values. Scores based on confidence values and Average Precision (AP); only k=1 justification allowed: Metric RunID Hop AP SF-ALL-Macro STANFORD_KB_CMN_4 0 0.0000 SF-ALL-Macro STANFORD_KB_CMN_4 1 0.0000 SF-ALL-Macro STANFORD_KB_CMN_4 ALL 0.0000 LDC-MEAN-ALL-Macro STANFORD_KB_CMN_4 0 0.0000 LDC-MEAN-ALL-Macro STANFORD_KB_CMN_4 1 0.0000 LDC-MEAN-ALL-Macro STANFORD_KB_CMN_4 ALL 0.0000 *ALL-Macro AP refer to mean of corresponding AP values. Scores based on P/R/F1; only k=1 justification allowed: Metric RunID Hop Prec Recall F1 SF-ALL-Micro STANFORD_KB_CMN_4 0 0.0000 0.0000 0.0000 SF-ALL-Micro STANFORD_KB_CMN_4 1 0.0000 0.0000 0.0000 SF-ALL-Micro STANFORD_KB_CMN_4 ALL 0.0000 0.0000 0.0000 SF-ALL-Macro STANFORD_KB_CMN_4 0 0.0000 0.0000 0.0000 SF-ALL-Macro STANFORD_KB_CMN_4 1 0.0000 0.0000 0.0000 SF-ALL-Macro STANFORD_KB_CMN_4 ALL 0.0000 0.0000 0.0000 LDC-MAX-ALL-Micro STANFORD_KB_CMN_4 0 0.0000 0.0000 0.0000 LDC-MAX-ALL-Micro STANFORD_KB_CMN_4 1 0.0000 0.0000 0.0000 LDC-MAX-ALL-Micro STANFORD_KB_CMN_4 ALL 0.0000 0.0000 0.0000 LDC-MAX-ALL-Macro STANFORD_KB_CMN_4 0 0.0000 0.0000 0.0000 LDC-MAX-ALL-Macro STANFORD_KB_CMN_4 1 0.0000 0.0000 0.0000 LDC-MAX-ALL-Macro STANFORD_KB_CMN_4 ALL 0.0000 0.0000 0.0000 LDC-MEAN-ALL-Macro STANFORD_KB_CMN_4 0 0.0000 0.0000 0.0000 LDC-MEAN-ALL-Macro STANFORD_KB_CMN_4 1 0.0000 0.0000 0.0000 LDC-MEAN-ALL-Macro STANFORD_KB_CMN_4 ALL 0.0000 0.0000 0.0000 *ALL-Macro Prec, Recall and F1 refer to mean-precision, mean-recall and mean-F1.