===================================================================== TAC KBP 2017 ENGLISH KB CONSTRUCTION: COMPOSITE KB EVALUATION RESULTS ===================================================================== Team ID: newbie_mr_01 Organization: Machine Reading Co ************************************************************* Run ID: newbie_mr_01_KB_ENG_1 Did the run access the live Web during the evaluation window: No Did the run extract relations from the Cold Start source corpus: Yes Did the run generate meaningful confidence values: No Did the run attempt the full Entity Discovery and Linking task: No Did the run attempt the full Event Nugget Detection and Coreference task: No Did the run include SF relations: Yes Did the run attempt the Event Argument Extraction and Linking task: No Did the run include Sentiment relations: No --------------------- Composite KB Evaluation (All slot types): Scores based on confidence values and Average Precision (AP); k=3 justifications allowed: Metric RunID Hop AP SF-ALL-Macro newbie_mr_01_KB_ENG_1 0 0.0011 SF-ALL-Macro newbie_mr_01_KB_ENG_1 1 0.0000 SF-ALL-Macro newbie_mr_01_KB_ENG_1 ALL 0.0011 LDC-MEAN-ALL-Macro newbie_mr_01_KB_ENG_1 0 0.0014 LDC-MEAN-ALL-Macro newbie_mr_01_KB_ENG_1 1 0.0000 LDC-MEAN-ALL-Macro newbie_mr_01_KB_ENG_1 ALL 0.0014 (PRIMARY METRIC) *ALL-Macro AP refer to mean of corresponding AP values. Scores based on confidence values and Average Precision (AP); only k=1 justification allowed: Metric RunID Hop AP SF-ALL-Macro newbie_mr_01_KB_ENG_1 0 0.0011 SF-ALL-Macro newbie_mr_01_KB_ENG_1 1 0.0000 SF-ALL-Macro newbie_mr_01_KB_ENG_1 ALL 0.0011 LDC-MEAN-ALL-Macro newbie_mr_01_KB_ENG_1 0 0.0014 LDC-MEAN-ALL-Macro newbie_mr_01_KB_ENG_1 1 0.0000 LDC-MEAN-ALL-Macro newbie_mr_01_KB_ENG_1 ALL 0.0014 *ALL-Macro AP refer to mean of corresponding AP values. Scores based on P/R/F1; only k=1 justification allowed: Metric RunID Hop Prec Recall F1 SF-ALL-Micro newbie_mr_01_KB_ENG_1 0 0.3333 0.0004 0.0009 SF-ALL-Micro newbie_mr_01_KB_ENG_1 1 0.0000 0.0000 0.0000 SF-ALL-Micro newbie_mr_01_KB_ENG_1 ALL 0.3333 0.0003 0.0007 SF-ALL-Macro newbie_mr_01_KB_ENG_1 0 0.0011 0.0011 0.0011 SF-ALL-Macro newbie_mr_01_KB_ENG_1 1 0.0000 0.0000 0.0000 SF-ALL-Macro newbie_mr_01_KB_ENG_1 ALL 0.0007 0.0007 0.0007 LDC-MAX-ALL-Micro newbie_mr_01_KB_ENG_1 0 1.0000 0.0010 0.0020 LDC-MAX-ALL-Micro newbie_mr_01_KB_ENG_1 1 0.0000 0.0000 0.0000 LDC-MAX-ALL-Micro newbie_mr_01_KB_ENG_1 ALL 1.0000 0.0008 0.0016 LDC-MAX-ALL-Macro newbie_mr_01_KB_ENG_1 0 0.0028 0.0028 0.0028 LDC-MAX-ALL-Macro newbie_mr_01_KB_ENG_1 1 0.0000 0.0000 0.0000 LDC-MAX-ALL-Macro newbie_mr_01_KB_ENG_1 ALL 0.0018 0.0018 0.0018 LDC-MEAN-ALL-Macro newbie_mr_01_KB_ENG_1 0 0.0014 0.0014 0.0014 LDC-MEAN-ALL-Macro newbie_mr_01_KB_ENG_1 1 0.0000 0.0000 0.0000 LDC-MEAN-ALL-Macro newbie_mr_01_KB_ENG_1 ALL 0.0009 0.0009 0.0009 *ALL-Macro Prec, Recall and F1 refer to mean-precision, mean-recall and mean-F1. --------------------- Composite KB Evaluation (SF-SLOTS only): Scores based on confidence values and Average Precision (AP); k=3 justifications allowed: Metric RunID Hop AP SF-ALL-Macro newbie_mr_01_KB_ENG_1 0 0.0025 SF-ALL-Macro newbie_mr_01_KB_ENG_1 1 0.0000 SF-ALL-Macro newbie_mr_01_KB_ENG_1 ALL 0.0025 LDC-MEAN-ALL-Macro newbie_mr_01_KB_ENG_1 0 0.0030 LDC-MEAN-ALL-Macro newbie_mr_01_KB_ENG_1 1 0.0000 LDC-MEAN-ALL-Macro newbie_mr_01_KB_ENG_1 ALL 0.0030 (PRIMARY METRIC) *ALL-Macro AP refer to mean of corresponding AP values. Scores based on confidence values and Average Precision (AP); only k=1 justification allowed: Metric RunID Hop AP SF-ALL-Macro newbie_mr_01_KB_ENG_1 0 0.0025 SF-ALL-Macro newbie_mr_01_KB_ENG_1 1 0.0000 SF-ALL-Macro newbie_mr_01_KB_ENG_1 ALL 0.0025 LDC-MEAN-ALL-Macro newbie_mr_01_KB_ENG_1 0 0.0030 LDC-MEAN-ALL-Macro newbie_mr_01_KB_ENG_1 1 0.0000 LDC-MEAN-ALL-Macro newbie_mr_01_KB_ENG_1 ALL 0.0030 *ALL-Macro AP refer to mean of corresponding AP values. Scores based on P/R/F1; only k=1 justification allowed: Metric RunID Hop Prec Recall F1 SF-ALL-Micro newbie_mr_01_KB_ENG_1 0 0.3333 0.0009 0.0017 SF-ALL-Micro newbie_mr_01_KB_ENG_1 1 0.0000 0.0000 0.0000 SF-ALL-Micro newbie_mr_01_KB_ENG_1 ALL 0.3333 0.0007 0.0013 SF-ALL-Macro newbie_mr_01_KB_ENG_1 0 0.0025 0.0025 0.0025 SF-ALL-Macro newbie_mr_01_KB_ENG_1 1 0.0000 0.0000 0.0000 SF-ALL-Macro newbie_mr_01_KB_ENG_1 ALL 0.0015 0.0015 0.0015 LDC-MAX-ALL-Micro newbie_mr_01_KB_ENG_1 0 0.5000 0.0019 0.0038 LDC-MAX-ALL-Micro newbie_mr_01_KB_ENG_1 1 0.0000 0.0000 0.0000 LDC-MAX-ALL-Micro newbie_mr_01_KB_ENG_1 ALL 0.5000 0.0015 0.0029 LDC-MAX-ALL-Macro newbie_mr_01_KB_ENG_1 0 0.0060 0.0060 0.0060 LDC-MAX-ALL-Macro newbie_mr_01_KB_ENG_1 1 0.0000 0.0000 0.0000 LDC-MAX-ALL-Macro newbie_mr_01_KB_ENG_1 ALL 0.0037 0.0037 0.0037 LDC-MEAN-ALL-Macro newbie_mr_01_KB_ENG_1 0 0.0030 0.0030 0.0030 LDC-MEAN-ALL-Macro newbie_mr_01_KB_ENG_1 1 0.0000 0.0000 0.0000 LDC-MEAN-ALL-Macro newbie_mr_01_KB_ENG_1 ALL 0.0018 0.0018 0.0018 *ALL-Macro Prec, Recall and F1 refer to mean-precision, mean-recall and mean-F1. --------------------- Composite KB Evaluation (EVENT-SLOTS only): Scores based on confidence values and Average Precision (AP); k=3 justifications allowed: SUMMARY: This section provides summary of AP scores Metric RunID Hop AP SF-ALL-Macro newbie_mr_01_KB_ENG_1 0 0.0000 SF-ALL-Macro newbie_mr_01_KB_ENG_1 ALL 0.0000 LDC-MEAN-ALL-Macro newbie_mr_01_KB_ENG_1 0 0.0000 LDC-MEAN-ALL-Macro newbie_mr_01_KB_ENG_1 ALL 0.0000 (PRIMARY METRIC) *ALL-Macro AP refer to mean of corresponding AP values. Scores based on confidence values and Average Precision (AP); only k=1 justification allowed: SUMMARY: This section provides summary of AP scores Metric RunID Hop AP SF-ALL-Macro newbie_mr_01_KB_ENG_1 0 0.0000 SF-ALL-Macro newbie_mr_01_KB_ENG_1 ALL 0.0000 LDC-MEAN-ALL-Macro newbie_mr_01_KB_ENG_1 0 0.0000 LDC-MEAN-ALL-Macro newbie_mr_01_KB_ENG_1 ALL 0.0000 *ALL-Macro AP refer to mean of corresponding AP values. Scores based on P/R/F1; only k=1 justification allowed: Metric RunID Hop Prec Recall F1 SF-ALL-Micro newbie_mr_01_KB_ENG_1 0 0.0000 0.0000 0.0000 SF-ALL-Micro newbie_mr_01_KB_ENG_1 ALL 0.0000 0.0000 0.0000 SF-ALL-Macro newbie_mr_01_KB_ENG_1 0 0.0000 0.0000 0.0000 SF-ALL-Macro newbie_mr_01_KB_ENG_1 ALL 0.0000 0.0000 0.0000 LDC-MAX-ALL-Micro newbie_mr_01_KB_ENG_1 0 0.0000 0.0000 0.0000 LDC-MAX-ALL-Micro newbie_mr_01_KB_ENG_1 ALL 0.0000 0.0000 0.0000 LDC-MAX-ALL-Macro newbie_mr_01_KB_ENG_1 0 0.0000 0.0000 0.0000 LDC-MAX-ALL-Macro newbie_mr_01_KB_ENG_1 ALL 0.0000 0.0000 0.0000 LDC-MEAN-ALL-Macro newbie_mr_01_KB_ENG_1 0 0.0000 0.0000 0.0000 LDC-MEAN-ALL-Macro newbie_mr_01_KB_ENG_1 ALL 0.0000 0.0000 0.0000 *ALL-Macro Prec, Recall and F1 refer to mean-precision, mean-recall and mean-F1. --------------------- Composite KB Evaluation (SENTIMENT-SLOTS only): Scores based on confidence values and Average Precision (AP); k=3 justifications allowed: Metric RunID Hop AP SF-ALL-Macro newbie_mr_01_KB_ENG_1 0 0.0000 SF-ALL-Macro newbie_mr_01_KB_ENG_1 1 0.0000 SF-ALL-Macro newbie_mr_01_KB_ENG_1 ALL 0.0000 LDC-MEAN-ALL-Macro newbie_mr_01_KB_ENG_1 0 0.0000 LDC-MEAN-ALL-Macro newbie_mr_01_KB_ENG_1 1 0.0000 LDC-MEAN-ALL-Macro newbie_mr_01_KB_ENG_1 ALL 0.0000 (PRIMARY METRIC) *ALL-Macro AP refer to mean of corresponding AP values. Scores based on confidence values and Average Precision (AP); only k=1 justification allowed: Metric RunID Hop AP SF-ALL-Macro newbie_mr_01_KB_ENG_1 0 0.0000 SF-ALL-Macro newbie_mr_01_KB_ENG_1 1 0.0000 SF-ALL-Macro newbie_mr_01_KB_ENG_1 ALL 0.0000 LDC-MEAN-ALL-Macro newbie_mr_01_KB_ENG_1 0 0.0000 LDC-MEAN-ALL-Macro newbie_mr_01_KB_ENG_1 1 0.0000 LDC-MEAN-ALL-Macro newbie_mr_01_KB_ENG_1 ALL 0.0000 *ALL-Macro AP refer to mean of corresponding AP values. Scores based on P/R/F1; only k=1 justification allowed: Metric RunID Hop Prec Recall F1 SF-ALL-Micro newbie_mr_01_KB_ENG_1 0 0.0000 0.0000 0.0000 SF-ALL-Micro newbie_mr_01_KB_ENG_1 1 0.0000 0.0000 0.0000 SF-ALL-Micro newbie_mr_01_KB_ENG_1 ALL 0.0000 0.0000 0.0000 SF-ALL-Macro newbie_mr_01_KB_ENG_1 0 0.0000 0.0000 0.0000 SF-ALL-Macro newbie_mr_01_KB_ENG_1 1 0.0000 0.0000 0.0000 SF-ALL-Macro newbie_mr_01_KB_ENG_1 ALL 0.0000 0.0000 0.0000 LDC-MAX-ALL-Micro newbie_mr_01_KB_ENG_1 0 0.0000 0.0000 0.0000 LDC-MAX-ALL-Micro newbie_mr_01_KB_ENG_1 1 0.0000 0.0000 0.0000 LDC-MAX-ALL-Micro newbie_mr_01_KB_ENG_1 ALL 0.0000 0.0000 0.0000 LDC-MAX-ALL-Macro newbie_mr_01_KB_ENG_1 0 0.0000 0.0000 0.0000 LDC-MAX-ALL-Macro newbie_mr_01_KB_ENG_1 1 0.0000 0.0000 0.0000 LDC-MAX-ALL-Macro newbie_mr_01_KB_ENG_1 ALL 0.0000 0.0000 0.0000 LDC-MEAN-ALL-Macro newbie_mr_01_KB_ENG_1 0 0.0000 0.0000 0.0000 LDC-MEAN-ALL-Macro newbie_mr_01_KB_ENG_1 1 0.0000 0.0000 0.0000 LDC-MEAN-ALL-Macro newbie_mr_01_KB_ENG_1 ALL 0.0000 0.0000 0.0000 *ALL-Macro Prec, Recall and F1 refer to mean-precision, mean-recall and mean-F1.