===================================================================== TAC KBP 2017 ENGLISH KB CONSTRUCTION: COMPOSITE KB EVALUATION RESULTS ===================================================================== Team ID: SAFT_ISI Organization: USC Information Sciences Institute ************************************************************* Run ID: SAFT_ISI_KB_ENG_1 Did the run access the live Web during the evaluation window: No Did the run extract relations from the Cold Start source corpus: Yes Did the run generate meaningful confidence values: Yes Did the run attempt the full Entity Discovery and Linking task: Yes Did the run attempt the full Event Nugget Detection and Coreference task: Yes Did the run include SF relations: Yes Did the run attempt the Event Argument Extraction and Linking task: Yes Did the run include Sentiment relations: Yes --------------------- Composite KB Evaluation (All slot types): Scores based on confidence values and Average Precision (AP); k=3 justifications allowed: Metric RunID Hop AP SF-ALL-Macro SAFT_ISI_KB_ENG_1 0 0.0457 SF-ALL-Macro SAFT_ISI_KB_ENG_1 1 0.0018 SF-ALL-Macro SAFT_ISI_KB_ENG_1 ALL 0.0414 LDC-MEAN-ALL-Macro SAFT_ISI_KB_ENG_1 0 0.0515 LDC-MEAN-ALL-Macro SAFT_ISI_KB_ENG_1 1 0.0026 LDC-MEAN-ALL-Macro SAFT_ISI_KB_ENG_1 ALL 0.0453 (PRIMARY METRIC) *ALL-Macro AP refer to mean of corresponding AP values. Scores based on confidence values and Average Precision (AP); only k=1 justification allowed: Metric RunID Hop AP SF-ALL-Macro SAFT_ISI_KB_ENG_1 0 0.0637 SF-ALL-Macro SAFT_ISI_KB_ENG_1 1 0.0028 SF-ALL-Macro SAFT_ISI_KB_ENG_1 ALL 0.0589 LDC-MEAN-ALL-Macro SAFT_ISI_KB_ENG_1 0 0.0697 LDC-MEAN-ALL-Macro SAFT_ISI_KB_ENG_1 1 0.0027 LDC-MEAN-ALL-Macro SAFT_ISI_KB_ENG_1 ALL 0.0636 *ALL-Macro AP refer to mean of corresponding AP values. Scores based on P/R/F1; only k=1 justification allowed: Metric RunID Hop Prec Recall F1 SF-ALL-Micro SAFT_ISI_KB_ENG_1 0 0.1894 0.0625 0.0940 SF-ALL-Micro SAFT_ISI_KB_ENG_1 1 0.0034 0.0189 0.0058 SF-ALL-Micro SAFT_ISI_KB_ENG_1 ALL 0.0346 0.0526 0.0418 SF-ALL-Macro SAFT_ISI_KB_ENG_1 0 0.0619 0.0766 0.0585 SF-ALL-Macro SAFT_ISI_KB_ENG_1 1 0.0036 0.0083 0.0043 SF-ALL-Macro SAFT_ISI_KB_ENG_1 ALL 0.0418 0.0531 0.0398 LDC-MAX-ALL-Micro SAFT_ISI_KB_ENG_1 0 0.1840 0.0773 0.1089 LDC-MAX-ALL-Micro SAFT_ISI_KB_ENG_1 1 0.0024 0.0252 0.0044 LDC-MAX-ALL-Micro SAFT_ISI_KB_ENG_1 ALL 0.0256 0.0660 0.0369 LDC-MAX-ALL-Macro SAFT_ISI_KB_ENG_1 0 0.0854 0.1066 0.0789 LDC-MAX-ALL-Macro SAFT_ISI_KB_ENG_1 1 0.0070 0.0143 0.0076 LDC-MAX-ALL-Macro SAFT_ISI_KB_ENG_1 ALL 0.0586 0.0750 0.0546 LDC-MEAN-ALL-Macro SAFT_ISI_KB_ENG_1 0 0.0644 0.0842 0.0623 LDC-MEAN-ALL-Macro SAFT_ISI_KB_ENG_1 1 0.0042 0.0098 0.0052 LDC-MEAN-ALL-Macro SAFT_ISI_KB_ENG_1 ALL 0.0438 0.0587 0.0428 *ALL-Macro Prec, Recall and F1 refer to mean-precision, mean-recall and mean-F1. --------------------- Composite KB Evaluation (SF-SLOTS only): Scores based on confidence values and Average Precision (AP); k=3 justifications allowed: Metric RunID Hop AP SF-ALL-Macro SAFT_ISI_KB_ENG_1 0 0.0500 SF-ALL-Macro SAFT_ISI_KB_ENG_1 1 0.0051 SF-ALL-Macro SAFT_ISI_KB_ENG_1 ALL 0.0403 LDC-MEAN-ALL-Macro SAFT_ISI_KB_ENG_1 0 0.0593 LDC-MEAN-ALL-Macro SAFT_ISI_KB_ENG_1 1 0.0069 LDC-MEAN-ALL-Macro SAFT_ISI_KB_ENG_1 ALL 0.0463 (PRIMARY METRIC) *ALL-Macro AP refer to mean of corresponding AP values. Scores based on confidence values and Average Precision (AP); only k=1 justification allowed: Metric RunID Hop AP SF-ALL-Macro SAFT_ISI_KB_ENG_1 0 0.0759 SF-ALL-Macro SAFT_ISI_KB_ENG_1 1 0.0079 SF-ALL-Macro SAFT_ISI_KB_ENG_1 ALL 0.0652 LDC-MEAN-ALL-Macro SAFT_ISI_KB_ENG_1 0 0.0875 LDC-MEAN-ALL-Macro SAFT_ISI_KB_ENG_1 1 0.0071 LDC-MEAN-ALL-Macro SAFT_ISI_KB_ENG_1 ALL 0.0745 *ALL-Macro AP refer to mean of corresponding AP values. Scores based on P/R/F1; only k=1 justification allowed: Metric RunID Hop Prec Recall F1 SF-ALL-Micro SAFT_ISI_KB_ENG_1 0 0.1383 0.0449 0.0678 SF-ALL-Micro SAFT_ISI_KB_ENG_1 1 0.0034 0.0355 0.0062 SF-ALL-Micro SAFT_ISI_KB_ENG_1 ALL 0.0156 0.0427 0.0228 SF-ALL-Macro SAFT_ISI_KB_ENG_1 0 0.0656 0.0877 0.0636 SF-ALL-Macro SAFT_ISI_KB_ENG_1 1 0.0067 0.0153 0.0079 SF-ALL-Macro SAFT_ISI_KB_ENG_1 ALL 0.0423 0.0590 0.0415 LDC-MAX-ALL-Micro SAFT_ISI_KB_ENG_1 0 0.1360 0.0592 0.0824 LDC-MAX-ALL-Micro SAFT_ISI_KB_ENG_1 1 0.0024 0.0443 0.0046 LDC-MAX-ALL-Micro SAFT_ISI_KB_ENG_1 ALL 0.0122 0.0557 0.0200 LDC-MAX-ALL-Macro SAFT_ISI_KB_ENG_1 0 0.0887 0.1230 0.0839 LDC-MAX-ALL-Macro SAFT_ISI_KB_ENG_1 1 0.0124 0.0252 0.0135 LDC-MAX-ALL-Macro SAFT_ISI_KB_ENG_1 ALL 0.0591 0.0850 0.0566 LDC-MEAN-ALL-Macro SAFT_ISI_KB_ENG_1 0 0.0722 0.1013 0.0716 LDC-MEAN-ALL-Macro SAFT_ISI_KB_ENG_1 1 0.0074 0.0173 0.0092 LDC-MEAN-ALL-Macro SAFT_ISI_KB_ENG_1 ALL 0.0470 0.0687 0.0473 *ALL-Macro Prec, Recall and F1 refer to mean-precision, mean-recall and mean-F1. --------------------- Composite KB Evaluation (EVENT-SLOTS only): Scores based on confidence values and Average Precision (AP); k=3 justifications allowed: SUMMARY: This section provides summary of AP scores Metric RunID Hop AP SF-ALL-Macro SAFT_ISI_KB_ENG_1 0 0.0560 SF-ALL-Macro SAFT_ISI_KB_ENG_1 ALL 0.0560 LDC-MEAN-ALL-Macro SAFT_ISI_KB_ENG_1 0 0.0622 LDC-MEAN-ALL-Macro SAFT_ISI_KB_ENG_1 ALL 0.0622 (PRIMARY METRIC) *ALL-Macro AP refer to mean of corresponding AP values. Scores based on confidence values and Average Precision (AP); only k=1 justification allowed: SUMMARY: This section provides summary of AP scores Metric RunID Hop AP SF-ALL-Macro SAFT_ISI_KB_ENG_1 0 0.0744 SF-ALL-Macro SAFT_ISI_KB_ENG_1 ALL 0.0744 LDC-MEAN-ALL-Macro SAFT_ISI_KB_ENG_1 0 0.0769 LDC-MEAN-ALL-Macro SAFT_ISI_KB_ENG_1 ALL 0.0769 *ALL-Macro AP refer to mean of corresponding AP values. Scores based on P/R/F1; only k=1 justification allowed: Metric RunID Hop Prec Recall F1 SF-ALL-Micro SAFT_ISI_KB_ENG_1 0 0.3350 0.0801 0.1293 SF-ALL-Micro SAFT_ISI_KB_ENG_1 ALL 0.3350 0.0801 0.1293 SF-ALL-Macro SAFT_ISI_KB_ENG_1 0 0.0745 0.0899 0.0713 SF-ALL-Macro SAFT_ISI_KB_ENG_1 ALL 0.0745 0.0899 0.0713 LDC-MAX-ALL-Micro SAFT_ISI_KB_ENG_1 0 0.3451 0.1046 0.1605 LDC-MAX-ALL-Micro SAFT_ISI_KB_ENG_1 ALL 0.3451 0.1046 0.1605 LDC-MAX-ALL-Macro SAFT_ISI_KB_ENG_1 0 0.1133 0.1271 0.1025 LDC-MAX-ALL-Macro SAFT_ISI_KB_ENG_1 ALL 0.1133 0.1271 0.1025 LDC-MEAN-ALL-Macro SAFT_ISI_KB_ENG_1 0 0.0778 0.0950 0.0746 LDC-MEAN-ALL-Macro SAFT_ISI_KB_ENG_1 ALL 0.0778 0.0950 0.0746 *ALL-Macro Prec, Recall and F1 refer to mean-precision, mean-recall and mean-F1. --------------------- Composite KB Evaluation (SENTIMENT-SLOTS only): Scores based on confidence values and Average Precision (AP); k=3 justifications allowed: Metric RunID Hop AP SF-ALL-Macro SAFT_ISI_KB_ENG_1 0 0.0158 SF-ALL-Macro SAFT_ISI_KB_ENG_1 1 0.0000 SF-ALL-Macro SAFT_ISI_KB_ENG_1 ALL 0.0148 LDC-MEAN-ALL-Macro SAFT_ISI_KB_ENG_1 0 0.0098 LDC-MEAN-ALL-Macro SAFT_ISI_KB_ENG_1 1 0.0000 LDC-MEAN-ALL-Macro SAFT_ISI_KB_ENG_1 ALL 0.0089 (PRIMARY METRIC) *ALL-Macro AP refer to mean of corresponding AP values. Scores based on confidence values and Average Precision (AP); only k=1 justification allowed: Metric RunID Hop AP SF-ALL-Macro SAFT_ISI_KB_ENG_1 0 0.0145 SF-ALL-Macro SAFT_ISI_KB_ENG_1 1 0.0000 SF-ALL-Macro SAFT_ISI_KB_ENG_1 ALL 0.0139 LDC-MEAN-ALL-Macro SAFT_ISI_KB_ENG_1 0 0.0087 LDC-MEAN-ALL-Macro SAFT_ISI_KB_ENG_1 1 0.0000 LDC-MEAN-ALL-Macro SAFT_ISI_KB_ENG_1 ALL 0.0081 *ALL-Macro AP refer to mean of corresponding AP values. Scores based on P/R/F1; only k=1 justification allowed: Metric RunID Hop Prec Recall F1 SF-ALL-Micro SAFT_ISI_KB_ENG_1 0 0.1323 0.0786 0.0986 SF-ALL-Micro SAFT_ISI_KB_ENG_1 1 0.0000 0.0000 0.0000 SF-ALL-Micro SAFT_ISI_KB_ENG_1 ALL 0.1323 0.0628 0.0852 SF-ALL-Macro SAFT_ISI_KB_ENG_1 0 0.0281 0.0252 0.0212 SF-ALL-Macro SAFT_ISI_KB_ENG_1 1 0.0000 0.0000 0.0000 SF-ALL-Macro SAFT_ISI_KB_ENG_1 ALL 0.0199 0.0178 0.0150 LDC-MAX-ALL-Micro SAFT_ISI_KB_ENG_1 0 0.1176 0.0714 0.0889 LDC-MAX-ALL-Micro SAFT_ISI_KB_ENG_1 1 0.0000 0.0000 0.0000 LDC-MAX-ALL-Micro SAFT_ISI_KB_ENG_1 ALL 0.1176 0.0563 0.0762 LDC-MAX-ALL-Macro SAFT_ISI_KB_ENG_1 0 0.0204 0.0225 0.0185 LDC-MAX-ALL-Macro SAFT_ISI_KB_ENG_1 1 0.0000 0.0000 0.0000 LDC-MAX-ALL-Macro SAFT_ISI_KB_ENG_1 ALL 0.0144 0.0158 0.0130 LDC-MEAN-ALL-Macro SAFT_ISI_KB_ENG_1 0 0.0172 0.0176 0.0132 LDC-MEAN-ALL-Macro SAFT_ISI_KB_ENG_1 1 0.0000 0.0000 0.0000 LDC-MEAN-ALL-Macro SAFT_ISI_KB_ENG_1 ALL 0.0121 0.0124 0.0093 *ALL-Macro Prec, Recall and F1 refer to mean-precision, mean-recall and mean-F1.