===================================================================== TAC KBP 2017 SPANISH KB CONSTRUCTION: COMPOSITE KB EVALUATION RESULTS ===================================================================== Team ID: SAFT_ISI Organization: USC Information Sciences Institute ************************************************************* Run ID: SAFT_ISI_KB_SPA_1 Did the run access the live Web during the evaluation window: No Did the run extract relations from the Cold Start source corpus: Yes Did the run generate meaningful confidence values: Yes Did the run attempt the full Entity Discovery and Linking task: Yes Did the run attempt the full Event Nugget Detection and Coreference task: Yes Did the run include SF relations: Yes Did the run attempt the Event Argument Extraction and Linking task: Yes Did the run include Sentiment relations: Yes --------------------- Composite KB Evaluation (All slot types): Scores based on confidence values and Average Precision (AP); k=3 justifications allowed: Metric RunID Hop AP SF-ALL-Macro SAFT_ISI_KB_SPA_1 0 0.0128 SF-ALL-Macro SAFT_ISI_KB_SPA_1 1 0.0000 SF-ALL-Macro SAFT_ISI_KB_SPA_1 ALL 0.0128 LDC-MEAN-ALL-Macro SAFT_ISI_KB_SPA_1 0 0.0111 LDC-MEAN-ALL-Macro SAFT_ISI_KB_SPA_1 1 0.0000 LDC-MEAN-ALL-Macro SAFT_ISI_KB_SPA_1 ALL 0.0111 (PRIMARY METRIC) *ALL-Macro AP refer to mean of corresponding AP values. Scores based on confidence values and Average Precision (AP); only k=1 justification allowed: Metric RunID Hop AP SF-ALL-Macro SAFT_ISI_KB_SPA_1 0 0.0136 SF-ALL-Macro SAFT_ISI_KB_SPA_1 1 0.0000 SF-ALL-Macro SAFT_ISI_KB_SPA_1 ALL 0.0136 LDC-MEAN-ALL-Macro SAFT_ISI_KB_SPA_1 0 0.0126 LDC-MEAN-ALL-Macro SAFT_ISI_KB_SPA_1 1 0.0000 LDC-MEAN-ALL-Macro SAFT_ISI_KB_SPA_1 ALL 0.0126 *ALL-Macro AP refer to mean of corresponding AP values. Scores based on P/R/F1; only k=1 justification allowed: Metric RunID Hop Prec Recall F1 SF-ALL-Micro SAFT_ISI_KB_SPA_1 0 0.0621 0.0184 0.0284 SF-ALL-Micro SAFT_ISI_KB_SPA_1 1 0.0000 0.0000 0.0000 SF-ALL-Micro SAFT_ISI_KB_SPA_1 ALL 0.0621 0.0121 0.0202 SF-ALL-Macro SAFT_ISI_KB_SPA_1 0 0.0151 0.0160 0.0142 SF-ALL-Macro SAFT_ISI_KB_SPA_1 1 0.0000 0.0000 0.0000 SF-ALL-Macro SAFT_ISI_KB_SPA_1 ALL 0.0095 0.0100 0.0089 LDC-MAX-ALL-Micro SAFT_ISI_KB_SPA_1 0 0.0580 0.0274 0.0372 LDC-MAX-ALL-Micro SAFT_ISI_KB_SPA_1 1 0.0000 0.0000 0.0000 LDC-MAX-ALL-Micro SAFT_ISI_KB_SPA_1 ALL 0.0580 0.0175 0.0269 LDC-MAX-ALL-Macro SAFT_ISI_KB_SPA_1 0 0.0210 0.0199 0.0185 LDC-MAX-ALL-Macro SAFT_ISI_KB_SPA_1 1 0.0000 0.0000 0.0000 LDC-MAX-ALL-Macro SAFT_ISI_KB_SPA_1 ALL 0.0133 0.0126 0.0118 LDC-MEAN-ALL-Macro SAFT_ISI_KB_SPA_1 0 0.0148 0.0147 0.0134 LDC-MEAN-ALL-Macro SAFT_ISI_KB_SPA_1 1 0.0000 0.0000 0.0000 LDC-MEAN-ALL-Macro SAFT_ISI_KB_SPA_1 ALL 0.0094 0.0093 0.0085 *ALL-Macro Prec, Recall and F1 refer to mean-precision, mean-recall and mean-F1. --------------------- Composite KB Evaluation (SF-SLOTS only): Scores based on confidence values and Average Precision (AP); k=3 justifications allowed: Metric RunID Hop AP SF-ALL-Macro SAFT_ISI_KB_SPA_1 0 0.0000 SF-ALL-Macro SAFT_ISI_KB_SPA_1 1 0.0000 SF-ALL-Macro SAFT_ISI_KB_SPA_1 ALL 0.0000 LDC-MEAN-ALL-Macro SAFT_ISI_KB_SPA_1 0 0.0000 LDC-MEAN-ALL-Macro SAFT_ISI_KB_SPA_1 1 0.0000 LDC-MEAN-ALL-Macro SAFT_ISI_KB_SPA_1 ALL 0.0000 (PRIMARY METRIC) *ALL-Macro AP refer to mean of corresponding AP values. Scores based on confidence values and Average Precision (AP); only k=1 justification allowed: Metric RunID Hop AP SF-ALL-Macro SAFT_ISI_KB_SPA_1 0 0.0000 SF-ALL-Macro SAFT_ISI_KB_SPA_1 1 0.0000 SF-ALL-Macro SAFT_ISI_KB_SPA_1 ALL 0.0000 LDC-MEAN-ALL-Macro SAFT_ISI_KB_SPA_1 0 0.0000 LDC-MEAN-ALL-Macro SAFT_ISI_KB_SPA_1 1 0.0000 LDC-MEAN-ALL-Macro SAFT_ISI_KB_SPA_1 ALL 0.0000 *ALL-Macro AP refer to mean of corresponding AP values. Scores based on P/R/F1; only k=1 justification allowed: Metric RunID Hop Prec Recall F1 SF-ALL-Micro SAFT_ISI_KB_SPA_1 0 0.0000 0.0000 0.0000 SF-ALL-Micro SAFT_ISI_KB_SPA_1 1 0.0000 0.0000 0.0000 SF-ALL-Micro SAFT_ISI_KB_SPA_1 ALL 0.0000 0.0000 0.0000 SF-ALL-Macro SAFT_ISI_KB_SPA_1 0 0.0000 0.0000 0.0000 SF-ALL-Macro SAFT_ISI_KB_SPA_1 1 0.0000 0.0000 0.0000 SF-ALL-Macro SAFT_ISI_KB_SPA_1 ALL 0.0000 0.0000 0.0000 LDC-MAX-ALL-Micro SAFT_ISI_KB_SPA_1 0 0.0000 0.0000 0.0000 LDC-MAX-ALL-Micro SAFT_ISI_KB_SPA_1 1 0.0000 0.0000 0.0000 LDC-MAX-ALL-Micro SAFT_ISI_KB_SPA_1 ALL 0.0000 0.0000 0.0000 LDC-MAX-ALL-Macro SAFT_ISI_KB_SPA_1 0 0.0000 0.0000 0.0000 LDC-MAX-ALL-Macro SAFT_ISI_KB_SPA_1 1 0.0000 0.0000 0.0000 LDC-MAX-ALL-Macro SAFT_ISI_KB_SPA_1 ALL 0.0000 0.0000 0.0000 LDC-MEAN-ALL-Macro SAFT_ISI_KB_SPA_1 0 0.0000 0.0000 0.0000 LDC-MEAN-ALL-Macro SAFT_ISI_KB_SPA_1 1 0.0000 0.0000 0.0000 LDC-MEAN-ALL-Macro SAFT_ISI_KB_SPA_1 ALL 0.0000 0.0000 0.0000 *ALL-Macro Prec, Recall and F1 refer to mean-precision, mean-recall and mean-F1. --------------------- Composite KB Evaluation (EVENT-SLOTS only): Scores based on confidence values and Average Precision (AP); k=3 justifications allowed: SUMMARY: This section provides summary of AP scores Metric RunID Hop AP SF-ALL-Macro SAFT_ISI_KB_SPA_1 0 0.0473 SF-ALL-Macro SAFT_ISI_KB_SPA_1 ALL 0.0473 LDC-MEAN-ALL-Macro SAFT_ISI_KB_SPA_1 0 0.0425 LDC-MEAN-ALL-Macro SAFT_ISI_KB_SPA_1 ALL 0.0425 (PRIMARY METRIC) *ALL-Macro AP refer to mean of corresponding AP values. Scores based on confidence values and Average Precision (AP); only k=1 justification allowed: SUMMARY: This section provides summary of AP scores Metric RunID Hop AP SF-ALL-Macro SAFT_ISI_KB_SPA_1 0 0.0511 SF-ALL-Macro SAFT_ISI_KB_SPA_1 ALL 0.0511 LDC-MEAN-ALL-Macro SAFT_ISI_KB_SPA_1 0 0.0493 LDC-MEAN-ALL-Macro SAFT_ISI_KB_SPA_1 ALL 0.0493 *ALL-Macro AP refer to mean of corresponding AP values. Scores based on P/R/F1; only k=1 justification allowed: Metric RunID Hop Prec Recall F1 SF-ALL-Micro SAFT_ISI_KB_SPA_1 0 0.1707 0.0603 0.0892 SF-ALL-Micro SAFT_ISI_KB_SPA_1 ALL 0.1707 0.0603 0.0892 SF-ALL-Macro SAFT_ISI_KB_SPA_1 0 0.0568 0.0568 0.0530 SF-ALL-Macro SAFT_ISI_KB_SPA_1 ALL 0.0568 0.0568 0.0530 LDC-MAX-ALL-Micro SAFT_ISI_KB_SPA_1 0 0.2273 0.0758 0.1136 LDC-MAX-ALL-Micro SAFT_ISI_KB_SPA_1 ALL 0.2273 0.0758 0.1136 LDC-MAX-ALL-Macro SAFT_ISI_KB_SPA_1 0 0.0816 0.0714 0.0714 LDC-MAX-ALL-Macro SAFT_ISI_KB_SPA_1 ALL 0.0816 0.0714 0.0714 LDC-MEAN-ALL-Macro SAFT_ISI_KB_SPA_1 0 0.0578 0.0544 0.0522 LDC-MEAN-ALL-Macro SAFT_ISI_KB_SPA_1 ALL 0.0578 0.0544 0.0522 *ALL-Macro Prec, Recall and F1 refer to mean-precision, mean-recall and mean-F1. --------------------- Composite KB Evaluation (SENTIMENT-SLOTS only): Scores based on confidence values and Average Precision (AP); k=3 justifications allowed: Metric RunID Hop AP SF-ALL-Macro SAFT_ISI_KB_SPA_1 0 0.0020 SF-ALL-Macro SAFT_ISI_KB_SPA_1 1 0.0000 SF-ALL-Macro SAFT_ISI_KB_SPA_1 ALL 0.0020 LDC-MEAN-ALL-Macro SAFT_ISI_KB_SPA_1 0 0.0018 LDC-MEAN-ALL-Macro SAFT_ISI_KB_SPA_1 1 0.0000 LDC-MEAN-ALL-Macro SAFT_ISI_KB_SPA_1 ALL 0.0018 (PRIMARY METRIC) *ALL-Macro AP refer to mean of corresponding AP values. Scores based on confidence values and Average Precision (AP); only k=1 justification allowed: Metric RunID Hop AP SF-ALL-Macro SAFT_ISI_KB_SPA_1 0 0.0004 SF-ALL-Macro SAFT_ISI_KB_SPA_1 1 0.0000 SF-ALL-Macro SAFT_ISI_KB_SPA_1 ALL 0.0004 LDC-MEAN-ALL-Macro SAFT_ISI_KB_SPA_1 0 0.0004 LDC-MEAN-ALL-Macro SAFT_ISI_KB_SPA_1 1 0.0000 LDC-MEAN-ALL-Macro SAFT_ISI_KB_SPA_1 ALL 0.0004 *ALL-Macro AP refer to mean of corresponding AP values. Scores based on P/R/F1; only k=1 justification allowed: Metric RunID Hop Prec Recall F1 SF-ALL-Micro SAFT_ISI_KB_SPA_1 0 0.0250 0.0180 0.0209 SF-ALL-Micro SAFT_ISI_KB_SPA_1 1 0.0000 0.0000 0.0000 SF-ALL-Micro SAFT_ISI_KB_SPA_1 ALL 0.0250 0.0155 0.0191 SF-ALL-Macro SAFT_ISI_KB_SPA_1 0 0.0007 0.0054 0.0013 SF-ALL-Macro SAFT_ISI_KB_SPA_1 1 0.0000 0.0000 0.0000 SF-ALL-Macro SAFT_ISI_KB_SPA_1 ALL 0.0005 0.0038 0.0009 LDC-MAX-ALL-Micro SAFT_ISI_KB_SPA_1 0 0.0306 0.0395 0.0345 LDC-MAX-ALL-Micro SAFT_ISI_KB_SPA_1 1 0.0000 0.0000 0.0000 LDC-MAX-ALL-Micro SAFT_ISI_KB_SPA_1 ALL 0.0306 0.0330 0.0317 LDC-MAX-ALL-Macro SAFT_ISI_KB_SPA_1 0 0.0013 0.0098 0.0023 LDC-MAX-ALL-Macro SAFT_ISI_KB_SPA_1 1 0.0000 0.0000 0.0000 LDC-MAX-ALL-Macro SAFT_ISI_KB_SPA_1 ALL 0.0010 0.0071 0.0017 LDC-MEAN-ALL-Macro SAFT_ISI_KB_SPA_1 0 0.0007 0.0049 0.0012 LDC-MEAN-ALL-Macro SAFT_ISI_KB_SPA_1 1 0.0000 0.0000 0.0000 LDC-MEAN-ALL-Macro SAFT_ISI_KB_SPA_1 ALL 0.0005 0.0035 0.0008 *ALL-Macro Prec, Recall and F1 refer to mean-precision, mean-recall and mean-F1.