===================================================================== TAC KBP 2017 CHINESE KB CONSTRUCTION: COMPOSITE KB EVALUATION RESULTS ===================================================================== Team ID: SAFT_ISI Organization: USC Information Sciences Institute ************************************************************* Run ID: SAFT_ISI_KB_CMN_1 Did the run access the live Web during the evaluation window: No Did the run extract relations from the Cold Start source corpus: Yes Did the run generate meaningful confidence values: Yes Did the run attempt the full Entity Discovery and Linking task: Yes Did the run attempt the full Event Nugget Detection and Coreference task: Yes Did the run include SF relations: Yes Did the run attempt the Event Argument Extraction and Linking task: Yes Did the run include Sentiment relations: Yes --------------------- Composite KB Evaluation (All slot types): Scores based on confidence values and Average Precision (AP); k=3 justifications allowed: Metric RunID Hop AP SF-ALL-Macro SAFT_ISI_KB_CMN_1 0 0.0033 SF-ALL-Macro SAFT_ISI_KB_CMN_1 1 0.0007 SF-ALL-Macro SAFT_ISI_KB_CMN_1 ALL 0.0013 LDC-MEAN-ALL-Macro SAFT_ISI_KB_CMN_1 0 0.0017 LDC-MEAN-ALL-Macro SAFT_ISI_KB_CMN_1 1 0.0004 LDC-MEAN-ALL-Macro SAFT_ISI_KB_CMN_1 ALL 0.0007 (PRIMARY METRIC) *ALL-Macro AP refer to mean of corresponding AP values. Scores based on confidence values and Average Precision (AP); only k=1 justification allowed: Metric RunID Hop AP SF-ALL-Macro SAFT_ISI_KB_CMN_1 0 0.0030 SF-ALL-Macro SAFT_ISI_KB_CMN_1 1 0.0005 SF-ALL-Macro SAFT_ISI_KB_CMN_1 ALL 0.0010 LDC-MEAN-ALL-Macro SAFT_ISI_KB_CMN_1 0 0.0016 LDC-MEAN-ALL-Macro SAFT_ISI_KB_CMN_1 1 0.0003 LDC-MEAN-ALL-Macro SAFT_ISI_KB_CMN_1 ALL 0.0005 *ALL-Macro AP refer to mean of corresponding AP values. Scores based on P/R/F1; only k=1 justification allowed: Metric RunID Hop Prec Recall F1 SF-ALL-Micro SAFT_ISI_KB_CMN_1 0 0.2003 0.0719 0.1058 SF-ALL-Micro SAFT_ISI_KB_CMN_1 1 0.0188 0.1347 0.0331 SF-ALL-Micro SAFT_ISI_KB_CMN_1 ALL 0.0370 0.0914 0.0527 SF-ALL-Macro SAFT_ISI_KB_CMN_1 0 0.0017 0.0050 0.0025 SF-ALL-Macro SAFT_ISI_KB_CMN_1 1 0.0505 0.1263 0.0600 SF-ALL-Macro SAFT_ISI_KB_CMN_1 ALL 0.0198 0.0502 0.0239 LDC-MAX-ALL-Micro SAFT_ISI_KB_CMN_1 0 0.1840 0.0378 0.0627 LDC-MAX-ALL-Micro SAFT_ISI_KB_CMN_1 1 0.0188 0.0688 0.0296 LDC-MAX-ALL-Micro SAFT_ISI_KB_CMN_1 ALL 0.0367 0.0476 0.0414 LDC-MAX-ALL-Macro SAFT_ISI_KB_CMN_1 0 0.0009 0.0027 0.0013 LDC-MAX-ALL-Macro SAFT_ISI_KB_CMN_1 1 0.0295 0.0739 0.0351 LDC-MAX-ALL-Macro SAFT_ISI_KB_CMN_1 ALL 0.0109 0.0276 0.0131 LDC-MEAN-ALL-Macro SAFT_ISI_KB_CMN_1 0 0.0009 0.0027 0.0013 LDC-MEAN-ALL-Macro SAFT_ISI_KB_CMN_1 1 0.0295 0.0739 0.0351 LDC-MEAN-ALL-Macro SAFT_ISI_KB_CMN_1 ALL 0.0109 0.0276 0.0131 *ALL-Macro Prec, Recall and F1 refer to mean-precision, mean-recall and mean-F1. --------------------- Composite KB Evaluation (SF-SLOTS only): Scores based on confidence values and Average Precision (AP); k=3 justifications allowed: Metric RunID Hop AP SF-ALL-Macro SAFT_ISI_KB_CMN_1 0 0.0000 SF-ALL-Macro SAFT_ISI_KB_CMN_1 1 0.0000 SF-ALL-Macro SAFT_ISI_KB_CMN_1 ALL 0.0000 LDC-MEAN-ALL-Macro SAFT_ISI_KB_CMN_1 0 0.0000 LDC-MEAN-ALL-Macro SAFT_ISI_KB_CMN_1 1 0.0000 LDC-MEAN-ALL-Macro SAFT_ISI_KB_CMN_1 ALL 0.0000 (PRIMARY METRIC) *ALL-Macro AP refer to mean of corresponding AP values. Scores based on confidence values and Average Precision (AP); only k=1 justification allowed: Metric RunID Hop AP SF-ALL-Macro SAFT_ISI_KB_CMN_1 0 0.0000 SF-ALL-Macro SAFT_ISI_KB_CMN_1 1 0.0000 SF-ALL-Macro SAFT_ISI_KB_CMN_1 ALL 0.0000 LDC-MEAN-ALL-Macro SAFT_ISI_KB_CMN_1 0 0.0000 LDC-MEAN-ALL-Macro SAFT_ISI_KB_CMN_1 1 0.0000 LDC-MEAN-ALL-Macro SAFT_ISI_KB_CMN_1 ALL 0.0000 *ALL-Macro AP refer to mean of corresponding AP values. Scores based on P/R/F1; only k=1 justification allowed: Metric RunID Hop Prec Recall F1 SF-ALL-Micro SAFT_ISI_KB_CMN_1 0 0.0000 0.0000 0.0000 SF-ALL-Micro SAFT_ISI_KB_CMN_1 1 0.0000 0.0000 0.0000 SF-ALL-Micro SAFT_ISI_KB_CMN_1 ALL 0.0000 0.0000 0.0000 SF-ALL-Macro SAFT_ISI_KB_CMN_1 0 0.0000 0.0000 0.0000 SF-ALL-Macro SAFT_ISI_KB_CMN_1 1 0.0000 0.0000 0.0000 SF-ALL-Macro SAFT_ISI_KB_CMN_1 ALL 0.0000 0.0000 0.0000 LDC-MAX-ALL-Micro SAFT_ISI_KB_CMN_1 0 0.0000 0.0000 0.0000 LDC-MAX-ALL-Micro SAFT_ISI_KB_CMN_1 1 0.0000 0.0000 0.0000 LDC-MAX-ALL-Micro SAFT_ISI_KB_CMN_1 ALL 0.0000 0.0000 0.0000 LDC-MAX-ALL-Macro SAFT_ISI_KB_CMN_1 0 0.0000 0.0000 0.0000 LDC-MAX-ALL-Macro SAFT_ISI_KB_CMN_1 1 0.0000 0.0000 0.0000 LDC-MAX-ALL-Macro SAFT_ISI_KB_CMN_1 ALL 0.0000 0.0000 0.0000 LDC-MEAN-ALL-Macro SAFT_ISI_KB_CMN_1 0 0.0000 0.0000 0.0000 LDC-MEAN-ALL-Macro SAFT_ISI_KB_CMN_1 1 0.0000 0.0000 0.0000 LDC-MEAN-ALL-Macro SAFT_ISI_KB_CMN_1 ALL 0.0000 0.0000 0.0000 *ALL-Macro Prec, Recall and F1 refer to mean-precision, mean-recall and mean-F1. --------------------- Composite KB Evaluation (EVENT-SLOTS only): Scores based on confidence values and Average Precision (AP); k=3 justifications allowed: SUMMARY: This section provides summary of AP scores Metric RunID Hop AP SF-ALL-Macro SAFT_ISI_KB_CMN_1 0 0.0000 SF-ALL-Macro SAFT_ISI_KB_CMN_1 ALL 0.0000 LDC-MEAN-ALL-Macro SAFT_ISI_KB_CMN_1 0 0.0000 LDC-MEAN-ALL-Macro SAFT_ISI_KB_CMN_1 ALL 0.0000 (PRIMARY METRIC) *ALL-Macro AP refer to mean of corresponding AP values. Scores based on confidence values and Average Precision (AP); only k=1 justification allowed: SUMMARY: This section provides summary of AP scores Metric RunID Hop AP SF-ALL-Macro SAFT_ISI_KB_CMN_1 0 0.0000 SF-ALL-Macro SAFT_ISI_KB_CMN_1 ALL 0.0000 LDC-MEAN-ALL-Macro SAFT_ISI_KB_CMN_1 0 0.0000 LDC-MEAN-ALL-Macro SAFT_ISI_KB_CMN_1 ALL 0.0000 *ALL-Macro AP refer to mean of corresponding AP values. Scores based on P/R/F1; only k=1 justification allowed: Metric RunID Hop Prec Recall F1 SF-ALL-Micro SAFT_ISI_KB_CMN_1 0 0.0000 0.0000 0.0000 SF-ALL-Micro SAFT_ISI_KB_CMN_1 ALL 0.0000 0.0000 0.0000 SF-ALL-Macro SAFT_ISI_KB_CMN_1 0 0.0000 0.0000 0.0000 SF-ALL-Macro SAFT_ISI_KB_CMN_1 ALL 0.0000 0.0000 0.0000 LDC-MAX-ALL-Micro SAFT_ISI_KB_CMN_1 0 0.0000 0.0000 0.0000 LDC-MAX-ALL-Micro SAFT_ISI_KB_CMN_1 ALL 0.0000 0.0000 0.0000 LDC-MAX-ALL-Macro SAFT_ISI_KB_CMN_1 0 0.0000 0.0000 0.0000 LDC-MAX-ALL-Macro SAFT_ISI_KB_CMN_1 ALL 0.0000 0.0000 0.0000 LDC-MEAN-ALL-Macro SAFT_ISI_KB_CMN_1 0 0.0000 0.0000 0.0000 LDC-MEAN-ALL-Macro SAFT_ISI_KB_CMN_1 ALL 0.0000 0.0000 0.0000 *ALL-Macro Prec, Recall and F1 refer to mean-precision, mean-recall and mean-F1. --------------------- Composite KB Evaluation (SENTIMENT-SLOTS only): Scores based on confidence values and Average Precision (AP); k=3 justifications allowed: Metric RunID Hop AP SF-ALL-Macro SAFT_ISI_KB_CMN_1 0 0.0178 SF-ALL-Macro SAFT_ISI_KB_CMN_1 1 0.0042 SF-ALL-Macro SAFT_ISI_KB_CMN_1 ALL 0.0068 LDC-MEAN-ALL-Macro SAFT_ISI_KB_CMN_1 0 0.0097 LDC-MEAN-ALL-Macro SAFT_ISI_KB_CMN_1 1 0.0022 LDC-MEAN-ALL-Macro SAFT_ISI_KB_CMN_1 ALL 0.0037 (PRIMARY METRIC) *ALL-Macro AP refer to mean of corresponding AP values. Scores based on confidence values and Average Precision (AP); only k=1 justification allowed: Metric RunID Hop AP SF-ALL-Macro SAFT_ISI_KB_CMN_1 0 0.0161 SF-ALL-Macro SAFT_ISI_KB_CMN_1 1 0.0026 SF-ALL-Macro SAFT_ISI_KB_CMN_1 ALL 0.0055 LDC-MEAN-ALL-Macro SAFT_ISI_KB_CMN_1 0 0.0088 LDC-MEAN-ALL-Macro SAFT_ISI_KB_CMN_1 1 0.0014 LDC-MEAN-ALL-Macro SAFT_ISI_KB_CMN_1 ALL 0.0030 *ALL-Macro AP refer to mean of corresponding AP values. Scores based on P/R/F1; only k=1 justification allowed: Metric RunID Hop Prec Recall F1 SF-ALL-Micro SAFT_ISI_KB_CMN_1 0 0.2294 0.2617 0.2445 SF-ALL-Micro SAFT_ISI_KB_CMN_1 1 0.0188 0.3680 0.0359 SF-ALL-Micro SAFT_ISI_KB_CMN_1 ALL 0.0375 0.3017 0.0667 SF-ALL-Macro SAFT_ISI_KB_CMN_1 0 0.0091 0.0274 0.0137 SF-ALL-Macro SAFT_ISI_KB_CMN_1 1 0.1351 0.3381 0.1606 SF-ALL-Macro SAFT_ISI_KB_CMN_1 ALL 0.0780 0.1974 0.0941 LDC-MAX-ALL-Micro SAFT_ISI_KB_CMN_1 0 0.2179 0.1681 0.1898 LDC-MAX-ALL-Micro SAFT_ISI_KB_CMN_1 1 0.0188 0.3113 0.0355 LDC-MAX-ALL-Micro SAFT_ISI_KB_CMN_1 ALL 0.0373 0.2130 0.0635 LDC-MAX-ALL-Macro SAFT_ISI_KB_CMN_1 0 0.0050 0.0150 0.0075 LDC-MAX-ALL-Macro SAFT_ISI_KB_CMN_1 1 0.1061 0.2657 0.1262 LDC-MAX-ALL-Macro SAFT_ISI_KB_CMN_1 ALL 0.0512 0.1294 0.0617 LDC-MEAN-ALL-Macro SAFT_ISI_KB_CMN_1 0 0.0050 0.0150 0.0075 LDC-MEAN-ALL-Macro SAFT_ISI_KB_CMN_1 1 0.1061 0.2657 0.1262 LDC-MEAN-ALL-Macro SAFT_ISI_KB_CMN_1 ALL 0.0512 0.1294 0.0617 *ALL-Macro Prec, Recall and F1 refer to mean-precision, mean-recall and mean-F1.