===================================================================== TAC KBP 2017 CHINESE KB CONSTRUCTION: COMPOSITE KB EVALUATION RESULTS ===================================================================== Team ID: ISCAS_Sogou Organization: Institute of Software, Chinese Academy of Sciences & Sogou, Inc. ************************************************************* Run ID: ISCAS_Sogou_KB_CMN_1 Did the run access the live Web during the evaluation window: No Did the run extract relations from the Cold Start source corpus: Yes Did the run generate meaningful confidence values: Yes Did the run attempt the full Entity Discovery and Linking task: No Did the run attempt the full Event Nugget Detection and Coreference task: Yes Did the run include SF relations: Yes Did the run attempt the Event Argument Extraction and Linking task: Yes Did the run include Sentiment relations: Yes --------------------- Composite KB Evaluation (All slot types): Scores based on confidence values and Average Precision (AP); k=3 justifications allowed: Metric RunID Hop AP SF-ALL-Macro ISCAS_Sogou_KB_CMN_1 0 0.0734 SF-ALL-Macro ISCAS_Sogou_KB_CMN_1 1 0.0136 SF-ALL-Macro ISCAS_Sogou_KB_CMN_1 ALL 0.0628 LDC-MEAN-ALL-Macro ISCAS_Sogou_KB_CMN_1 0 0.0793 LDC-MEAN-ALL-Macro ISCAS_Sogou_KB_CMN_1 1 0.0111 LDC-MEAN-ALL-Macro ISCAS_Sogou_KB_CMN_1 ALL 0.0699 (PRIMARY METRIC) *ALL-Macro AP refer to mean of corresponding AP values. Scores based on confidence values and Average Precision (AP); only k=1 justification allowed: Metric RunID Hop AP SF-ALL-Macro ISCAS_Sogou_KB_CMN_1 0 0.1081 SF-ALL-Macro ISCAS_Sogou_KB_CMN_1 1 0.0159 SF-ALL-Macro ISCAS_Sogou_KB_CMN_1 ALL 0.0912 LDC-MEAN-ALL-Macro ISCAS_Sogou_KB_CMN_1 0 0.1086 LDC-MEAN-ALL-Macro ISCAS_Sogou_KB_CMN_1 1 0.0141 LDC-MEAN-ALL-Macro ISCAS_Sogou_KB_CMN_1 ALL 0.0959 *ALL-Macro AP refer to mean of corresponding AP values. Scores based on P/R/F1; only k=1 justification allowed: Metric RunID Hop Prec Recall F1 SF-ALL-Micro ISCAS_Sogou_KB_CMN_1 0 0.3610 0.1229 0.1834 SF-ALL-Micro ISCAS_Sogou_KB_CMN_1 1 0.0374 0.0558 0.0448 SF-ALL-Micro ISCAS_Sogou_KB_CMN_1 ALL 0.1461 0.1020 0.1201 SF-ALL-Macro ISCAS_Sogou_KB_CMN_1 0 0.1570 0.1298 0.1154 SF-ALL-Macro ISCAS_Sogou_KB_CMN_1 1 0.0302 0.0442 0.0324 SF-ALL-Macro ISCAS_Sogou_KB_CMN_1 ALL 0.1098 0.0979 0.0845 LDC-MAX-ALL-Micro ISCAS_Sogou_KB_CMN_1 0 0.4224 0.1424 0.2130 LDC-MAX-ALL-Micro ISCAS_Sogou_KB_CMN_1 1 0.0943 0.0625 0.0752 LDC-MAX-ALL-Micro ISCAS_Sogou_KB_CMN_1 ALL 0.2658 0.1171 0.1625 LDC-MAX-ALL-Macro ISCAS_Sogou_KB_CMN_1 0 0.1989 0.1494 0.1447 LDC-MAX-ALL-Macro ISCAS_Sogou_KB_CMN_1 1 0.0336 0.0527 0.0387 LDC-MAX-ALL-Macro ISCAS_Sogou_KB_CMN_1 ALL 0.1411 0.1156 0.1076 LDC-MEAN-ALL-Macro ISCAS_Sogou_KB_CMN_1 0 0.1696 0.1265 0.1207 LDC-MEAN-ALL-Macro ISCAS_Sogou_KB_CMN_1 1 0.0308 0.0392 0.0319 LDC-MEAN-ALL-Macro ISCAS_Sogou_KB_CMN_1 ALL 0.1211 0.0960 0.0896 *ALL-Macro Prec, Recall and F1 refer to mean-precision, mean-recall and mean-F1. --------------------- Composite KB Evaluation (SF-SLOTS only): Scores based on confidence values and Average Precision (AP); k=3 justifications allowed: Metric RunID Hop AP SF-ALL-Macro ISCAS_Sogou_KB_CMN_1 0 0.0699 SF-ALL-Macro ISCAS_Sogou_KB_CMN_1 1 0.0160 SF-ALL-Macro ISCAS_Sogou_KB_CMN_1 ALL 0.0509 LDC-MEAN-ALL-Macro ISCAS_Sogou_KB_CMN_1 0 0.0704 LDC-MEAN-ALL-Macro ISCAS_Sogou_KB_CMN_1 1 0.0099 LDC-MEAN-ALL-Macro ISCAS_Sogou_KB_CMN_1 ALL 0.0531 (PRIMARY METRIC) *ALL-Macro AP refer to mean of corresponding AP values. Scores based on confidence values and Average Precision (AP); only k=1 justification allowed: Metric RunID Hop AP SF-ALL-Macro ISCAS_Sogou_KB_CMN_1 0 0.1075 SF-ALL-Macro ISCAS_Sogou_KB_CMN_1 1 0.0215 SF-ALL-Macro ISCAS_Sogou_KB_CMN_1 ALL 0.0783 LDC-MEAN-ALL-Macro ISCAS_Sogou_KB_CMN_1 0 0.1013 LDC-MEAN-ALL-Macro ISCAS_Sogou_KB_CMN_1 1 0.0144 LDC-MEAN-ALL-Macro ISCAS_Sogou_KB_CMN_1 ALL 0.0793 *ALL-Macro AP refer to mean of corresponding AP values. Scores based on P/R/F1; only k=1 justification allowed: Metric RunID Hop Prec Recall F1 SF-ALL-Micro ISCAS_Sogou_KB_CMN_1 0 0.2500 0.0789 0.1200 SF-ALL-Micro ISCAS_Sogou_KB_CMN_1 1 0.1282 0.0829 0.1007 SF-ALL-Micro ISCAS_Sogou_KB_CMN_1 ALL 0.2045 0.0798 0.1148 SF-ALL-Macro ISCAS_Sogou_KB_CMN_1 0 0.1110 0.1281 0.0981 SF-ALL-Macro ISCAS_Sogou_KB_CMN_1 1 0.0311 0.0378 0.0322 SF-ALL-Macro ISCAS_Sogou_KB_CMN_1 ALL 0.0838 0.0974 0.0756 LDC-MAX-ALL-Micro ISCAS_Sogou_KB_CMN_1 0 0.2903 0.0796 0.1250 LDC-MAX-ALL-Micro ISCAS_Sogou_KB_CMN_1 1 0.2321 0.1140 0.1529 LDC-MAX-ALL-Micro ISCAS_Sogou_KB_CMN_1 ALL 0.2722 0.0866 0.1314 LDC-MAX-ALL-Macro ISCAS_Sogou_KB_CMN_1 0 0.1404 0.1414 0.1198 LDC-MAX-ALL-Macro ISCAS_Sogou_KB_CMN_1 1 0.0254 0.0390 0.0297 LDC-MAX-ALL-Macro ISCAS_Sogou_KB_CMN_1 ALL 0.0992 0.1047 0.0875 LDC-MEAN-ALL-Macro ISCAS_Sogou_KB_CMN_1 0 0.1170 0.1172 0.0974 LDC-MEAN-ALL-Macro ISCAS_Sogou_KB_CMN_1 1 0.0212 0.0251 0.0216 LDC-MEAN-ALL-Macro ISCAS_Sogou_KB_CMN_1 ALL 0.0827 0.0842 0.0703 *ALL-Macro Prec, Recall and F1 refer to mean-precision, mean-recall and mean-F1. --------------------- Composite KB Evaluation (EVENT-SLOTS only): Scores based on confidence values and Average Precision (AP); k=3 justifications allowed: SUMMARY: This section provides summary of AP scores Metric RunID Hop AP SF-ALL-Macro ISCAS_Sogou_KB_CMN_1 0 0.0749 SF-ALL-Macro ISCAS_Sogou_KB_CMN_1 ALL 0.0749 LDC-MEAN-ALL-Macro ISCAS_Sogou_KB_CMN_1 0 0.0914 LDC-MEAN-ALL-Macro ISCAS_Sogou_KB_CMN_1 ALL 0.0914 (PRIMARY METRIC) *ALL-Macro AP refer to mean of corresponding AP values. Scores based on confidence values and Average Precision (AP); only k=1 justification allowed: SUMMARY: This section provides summary of AP scores Metric RunID Hop AP SF-ALL-Macro ISCAS_Sogou_KB_CMN_1 0 0.1120 SF-ALL-Macro ISCAS_Sogou_KB_CMN_1 ALL 0.1120 LDC-MEAN-ALL-Macro ISCAS_Sogou_KB_CMN_1 0 0.1246 LDC-MEAN-ALL-Macro ISCAS_Sogou_KB_CMN_1 ALL 0.1246 *ALL-Macro AP refer to mean of corresponding AP values. Scores based on P/R/F1; only k=1 justification allowed: Metric RunID Hop Prec Recall F1 SF-ALL-Micro ISCAS_Sogou_KB_CMN_1 0 0.5938 0.1699 0.2643 SF-ALL-Micro ISCAS_Sogou_KB_CMN_1 ALL 0.5938 0.1699 0.2643 SF-ALL-Macro ISCAS_Sogou_KB_CMN_1 0 0.2204 0.1187 0.1430 SF-ALL-Macro ISCAS_Sogou_KB_CMN_1 ALL 0.2204 0.1187 0.1430 LDC-MAX-ALL-Micro ISCAS_Sogou_KB_CMN_1 0 0.5935 0.2098 0.3100 LDC-MAX-ALL-Micro ISCAS_Sogou_KB_CMN_1 ALL 0.5935 0.2098 0.3100 LDC-MAX-ALL-Macro ISCAS_Sogou_KB_CMN_1 0 0.2765 0.1512 0.1825 LDC-MAX-ALL-Macro ISCAS_Sogou_KB_CMN_1 ALL 0.2765 0.1512 0.1825 LDC-MEAN-ALL-Macro ISCAS_Sogou_KB_CMN_1 0 0.2497 0.1324 0.1605 LDC-MEAN-ALL-Macro ISCAS_Sogou_KB_CMN_1 ALL 0.2497 0.1324 0.1605 *ALL-Macro Prec, Recall and F1 refer to mean-precision, mean-recall and mean-F1. --------------------- Composite KB Evaluation (SENTIMENT-SLOTS only): Scores based on confidence values and Average Precision (AP); k=3 justifications allowed: Metric RunID Hop AP SF-ALL-Macro ISCAS_Sogou_KB_CMN_1 0 0.0797 SF-ALL-Macro ISCAS_Sogou_KB_CMN_1 1 0.0037 SF-ALL-Macro ISCAS_Sogou_KB_CMN_1 ALL 0.0719 LDC-MEAN-ALL-Macro ISCAS_Sogou_KB_CMN_1 0 0.0812 LDC-MEAN-ALL-Macro ISCAS_Sogou_KB_CMN_1 1 0.0029 LDC-MEAN-ALL-Macro ISCAS_Sogou_KB_CMN_1 ALL 0.0759 (PRIMARY METRIC) *ALL-Macro AP refer to mean of corresponding AP values. Scores based on confidence values and Average Precision (AP); only k=1 justification allowed: Metric RunID Hop AP SF-ALL-Macro ISCAS_Sogou_KB_CMN_1 0 0.1025 SF-ALL-Macro ISCAS_Sogou_KB_CMN_1 1 0.0056 SF-ALL-Macro ISCAS_Sogou_KB_CMN_1 ALL 0.0871 LDC-MEAN-ALL-Macro ISCAS_Sogou_KB_CMN_1 0 0.0991 LDC-MEAN-ALL-Macro ISCAS_Sogou_KB_CMN_1 1 0.0044 LDC-MEAN-ALL-Macro ISCAS_Sogou_KB_CMN_1 ALL 0.0884 *ALL-Macro AP refer to mean of corresponding AP values. Scores based on P/R/F1; only k=1 justification allowed: Metric RunID Hop Prec Recall F1 SF-ALL-Micro ISCAS_Sogou_KB_CMN_1 0 0.2828 0.1253 0.1736 SF-ALL-Micro ISCAS_Sogou_KB_CMN_1 1 0.0313 0.0743 0.0441 SF-ALL-Micro ISCAS_Sogou_KB_CMN_1 ALL 0.0908 0.1061 0.0979 SF-ALL-Macro ISCAS_Sogou_KB_CMN_1 0 0.1619 0.1544 0.1100 SF-ALL-Macro ISCAS_Sogou_KB_CMN_1 1 0.0223 0.0354 0.0247 SF-ALL-Macro ISCAS_Sogou_KB_CMN_1 ALL 0.0855 0.0893 0.0634 LDC-MAX-ALL-Micro ISCAS_Sogou_KB_CMN_1 0 0.3689 0.1638 0.2269 LDC-MAX-ALL-Micro ISCAS_Sogou_KB_CMN_1 1 0.0248 0.1226 0.0413 LDC-MAX-ALL-Micro ISCAS_Sogou_KB_CMN_1 ALL 0.0813 0.1509 0.1057 LDC-MAX-ALL-Macro ISCAS_Sogou_KB_CMN_1 0 0.2171 0.1681 0.1442 LDC-MAX-ALL-Macro ISCAS_Sogou_KB_CMN_1 1 0.0227 0.0465 0.0303 LDC-MAX-ALL-Macro ISCAS_Sogou_KB_CMN_1 ALL 0.1284 0.1126 0.0922 LDC-MEAN-ALL-Macro ISCAS_Sogou_KB_CMN_1 0 0.1657 0.1411 0.1107 LDC-MEAN-ALL-Macro ISCAS_Sogou_KB_CMN_1 1 0.0216 0.0315 0.0228 LDC-MEAN-ALL-Macro ISCAS_Sogou_KB_CMN_1 ALL 0.0999 0.0911 0.0706 *ALL-Macro Prec, Recall and F1 refer to mean-precision, mean-recall and mean-F1. ************************************************************* Run ID: ISCAS_Sogou_KB_CMN_2 Did the run access the live Web during the evaluation window: No Did the run extract relations from the Cold Start source corpus: Yes Did the run generate meaningful confidence values: Yes Did the run attempt the full Entity Discovery and Linking task: No Did the run attempt the full Event Nugget Detection and Coreference task: Yes Did the run include SF relations: Yes Did the run attempt the Event Argument Extraction and Linking task: Yes Did the run include Sentiment relations: Yes --------------------- Composite KB Evaluation (All slot types): Scores based on confidence values and Average Precision (AP); k=3 justifications allowed: Metric RunID Hop AP SF-ALL-Macro ISCAS_Sogou_KB_CMN_2 0 0.0694 SF-ALL-Macro ISCAS_Sogou_KB_CMN_2 1 0.0116 SF-ALL-Macro ISCAS_Sogou_KB_CMN_2 ALL 0.0587 LDC-MEAN-ALL-Macro ISCAS_Sogou_KB_CMN_2 0 0.0761 LDC-MEAN-ALL-Macro ISCAS_Sogou_KB_CMN_2 1 0.0086 LDC-MEAN-ALL-Macro ISCAS_Sogou_KB_CMN_2 ALL 0.0666 (PRIMARY METRIC) *ALL-Macro AP refer to mean of corresponding AP values. Scores based on confidence values and Average Precision (AP); only k=1 justification allowed: Metric RunID Hop AP SF-ALL-Macro ISCAS_Sogou_KB_CMN_2 0 0.1045 SF-ALL-Macro ISCAS_Sogou_KB_CMN_2 1 0.0186 SF-ALL-Macro ISCAS_Sogou_KB_CMN_2 ALL 0.0881 LDC-MEAN-ALL-Macro ISCAS_Sogou_KB_CMN_2 0 0.1057 LDC-MEAN-ALL-Macro ISCAS_Sogou_KB_CMN_2 1 0.0138 LDC-MEAN-ALL-Macro ISCAS_Sogou_KB_CMN_2 ALL 0.0933 *ALL-Macro AP refer to mean of corresponding AP values. Scores based on P/R/F1; only k=1 justification allowed: Metric RunID Hop Prec Recall F1 SF-ALL-Micro ISCAS_Sogou_KB_CMN_2 0 0.2439 0.1346 0.1735 SF-ALL-Micro ISCAS_Sogou_KB_CMN_2 1 0.0131 0.0626 0.0217 SF-ALL-Micro ISCAS_Sogou_KB_CMN_2 ALL 0.0602 0.1122 0.0783 SF-ALL-Macro ISCAS_Sogou_KB_CMN_2 0 0.1443 0.1343 0.1109 SF-ALL-Macro ISCAS_Sogou_KB_CMN_2 1 0.0236 0.0449 0.0272 SF-ALL-Macro ISCAS_Sogou_KB_CMN_2 ALL 0.0994 0.1011 0.0797 LDC-MAX-ALL-Micro ISCAS_Sogou_KB_CMN_2 0 0.2952 0.1550 0.2033 LDC-MAX-ALL-Micro ISCAS_Sogou_KB_CMN_2 1 0.0165 0.0708 0.0268 LDC-MAX-ALL-Micro ISCAS_Sogou_KB_CMN_2 ALL 0.0745 0.1283 0.0943 LDC-MAX-ALL-Macro ISCAS_Sogou_KB_CMN_2 0 0.1809 0.1574 0.1401 LDC-MAX-ALL-Macro ISCAS_Sogou_KB_CMN_2 1 0.0296 0.0544 0.0342 LDC-MAX-ALL-Macro ISCAS_Sogou_KB_CMN_2 ALL 0.1280 0.1214 0.1031 LDC-MEAN-ALL-Macro ISCAS_Sogou_KB_CMN_2 0 0.1570 0.1320 0.1175 LDC-MEAN-ALL-Macro ISCAS_Sogou_KB_CMN_2 1 0.0259 0.0399 0.0283 LDC-MEAN-ALL-Macro ISCAS_Sogou_KB_CMN_2 ALL 0.1112 0.0998 0.0863 *ALL-Macro Prec, Recall and F1 refer to mean-precision, mean-recall and mean-F1. --------------------- Composite KB Evaluation (SF-SLOTS only): Scores based on confidence values and Average Precision (AP); k=3 justifications allowed: Metric RunID Hop AP SF-ALL-Macro ISCAS_Sogou_KB_CMN_2 0 0.0706 SF-ALL-Macro ISCAS_Sogou_KB_CMN_2 1 0.0159 SF-ALL-Macro ISCAS_Sogou_KB_CMN_2 ALL 0.0516 LDC-MEAN-ALL-Macro ISCAS_Sogou_KB_CMN_2 0 0.0709 LDC-MEAN-ALL-Macro ISCAS_Sogou_KB_CMN_2 1 0.0099 LDC-MEAN-ALL-Macro ISCAS_Sogou_KB_CMN_2 ALL 0.0537 (PRIMARY METRIC) *ALL-Macro AP refer to mean of corresponding AP values. Scores based on confidence values and Average Precision (AP); only k=1 justification allowed: Metric RunID Hop AP SF-ALL-Macro ISCAS_Sogou_KB_CMN_2 0 0.1096 SF-ALL-Macro ISCAS_Sogou_KB_CMN_2 1 0.0214 SF-ALL-Macro ISCAS_Sogou_KB_CMN_2 ALL 0.0805 LDC-MEAN-ALL-Macro ISCAS_Sogou_KB_CMN_2 0 0.1029 LDC-MEAN-ALL-Macro ISCAS_Sogou_KB_CMN_2 1 0.0143 LDC-MEAN-ALL-Macro ISCAS_Sogou_KB_CMN_2 ALL 0.0810 *ALL-Macro AP refer to mean of corresponding AP values. Scores based on P/R/F1; only k=1 justification allowed: Metric RunID Hop Prec Recall F1 SF-ALL-Micro ISCAS_Sogou_KB_CMN_2 0 0.2553 0.0773 0.1187 SF-ALL-Micro ISCAS_Sogou_KB_CMN_2 1 0.1443 0.0773 0.1007 SF-ALL-Micro ISCAS_Sogou_KB_CMN_2 ALL 0.2175 0.0773 0.1141 SF-ALL-Macro ISCAS_Sogou_KB_CMN_2 0 0.1097 0.1258 0.0969 SF-ALL-Macro ISCAS_Sogou_KB_CMN_2 1 0.0266 0.0360 0.0296 SF-ALL-Macro ISCAS_Sogou_KB_CMN_2 ALL 0.0814 0.0952 0.0740 LDC-MAX-ALL-Micro ISCAS_Sogou_KB_CMN_2 0 0.3077 0.0796 0.1265 LDC-MAX-ALL-Micro ISCAS_Sogou_KB_CMN_2 1 0.2241 0.1140 0.1512 LDC-MAX-ALL-Micro ISCAS_Sogou_KB_CMN_2 ALL 0.2800 0.0866 0.1323 LDC-MAX-ALL-Macro ISCAS_Sogou_KB_CMN_2 0 0.1404 0.1414 0.1199 LDC-MAX-ALL-Macro ISCAS_Sogou_KB_CMN_2 1 0.0254 0.0390 0.0297 LDC-MAX-ALL-Macro ISCAS_Sogou_KB_CMN_2 ALL 0.0992 0.1047 0.0876 LDC-MEAN-ALL-Macro ISCAS_Sogou_KB_CMN_2 0 0.1164 0.1154 0.0968 LDC-MEAN-ALL-Macro ISCAS_Sogou_KB_CMN_2 1 0.0180 0.0238 0.0198 LDC-MEAN-ALL-Macro ISCAS_Sogou_KB_CMN_2 ALL 0.0811 0.0826 0.0692 *ALL-Macro Prec, Recall and F1 refer to mean-precision, mean-recall and mean-F1. --------------------- Composite KB Evaluation (EVENT-SLOTS only): Scores based on confidence values and Average Precision (AP); k=3 justifications allowed: SUMMARY: This section provides summary of AP scores Metric RunID Hop AP SF-ALL-Macro ISCAS_Sogou_KB_CMN_2 0 0.0753 SF-ALL-Macro ISCAS_Sogou_KB_CMN_2 ALL 0.0753 LDC-MEAN-ALL-Macro ISCAS_Sogou_KB_CMN_2 0 0.0920 LDC-MEAN-ALL-Macro ISCAS_Sogou_KB_CMN_2 ALL 0.0920 (PRIMARY METRIC) *ALL-Macro AP refer to mean of corresponding AP values. Scores based on confidence values and Average Precision (AP); only k=1 justification allowed: SUMMARY: This section provides summary of AP scores Metric RunID Hop AP SF-ALL-Macro ISCAS_Sogou_KB_CMN_2 0 0.1126 SF-ALL-Macro ISCAS_Sogou_KB_CMN_2 ALL 0.1126 LDC-MEAN-ALL-Macro ISCAS_Sogou_KB_CMN_2 0 0.1253 LDC-MEAN-ALL-Macro ISCAS_Sogou_KB_CMN_2 ALL 0.1253 *ALL-Macro AP refer to mean of corresponding AP values. Scores based on P/R/F1; only k=1 justification allowed: Metric RunID Hop Prec Recall F1 SF-ALL-Micro ISCAS_Sogou_KB_CMN_2 0 0.6025 0.1735 0.2694 SF-ALL-Micro ISCAS_Sogou_KB_CMN_2 ALL 0.6025 0.1735 0.2694 SF-ALL-Macro ISCAS_Sogou_KB_CMN_2 0 0.2212 0.1193 0.1437 SF-ALL-Macro ISCAS_Sogou_KB_CMN_2 ALL 0.2212 0.1193 0.1437 LDC-MAX-ALL-Micro ISCAS_Sogou_KB_CMN_2 0 0.6098 0.2155 0.3185 LDC-MAX-ALL-Micro ISCAS_Sogou_KB_CMN_2 ALL 0.6098 0.2155 0.3185 LDC-MAX-ALL-Macro ISCAS_Sogou_KB_CMN_2 0 0.2778 0.1522 0.1837 LDC-MAX-ALL-Macro ISCAS_Sogou_KB_CMN_2 ALL 0.2778 0.1522 0.1837 LDC-MEAN-ALL-Macro ISCAS_Sogou_KB_CMN_2 0 0.2503 0.1329 0.1612 LDC-MEAN-ALL-Macro ISCAS_Sogou_KB_CMN_2 ALL 0.2503 0.1329 0.1612 *ALL-Macro Prec, Recall and F1 refer to mean-precision, mean-recall and mean-F1. --------------------- Composite KB Evaluation (SENTIMENT-SLOTS only): Scores based on confidence values and Average Precision (AP); k=3 justifications allowed: Metric RunID Hop AP SF-ALL-Macro ISCAS_Sogou_KB_CMN_2 0 0.0555 SF-ALL-Macro ISCAS_Sogou_KB_CMN_2 1 0.0029 SF-ALL-Macro ISCAS_Sogou_KB_CMN_2 ALL 0.0468 LDC-MEAN-ALL-Macro ISCAS_Sogou_KB_CMN_2 0 0.0607 LDC-MEAN-ALL-Macro ISCAS_Sogou_KB_CMN_2 1 0.0023 LDC-MEAN-ALL-Macro ISCAS_Sogou_KB_CMN_2 ALL 0.0549 (PRIMARY METRIC) *ALL-Macro AP refer to mean of corresponding AP values. Scores based on confidence values and Average Precision (AP); only k=1 justification allowed: Metric RunID Hop AP SF-ALL-Macro ISCAS_Sogou_KB_CMN_2 0 0.0764 SF-ALL-Macro ISCAS_Sogou_KB_CMN_2 1 0.0075 SF-ALL-Macro ISCAS_Sogou_KB_CMN_2 ALL 0.0635 LDC-MEAN-ALL-Macro ISCAS_Sogou_KB_CMN_2 0 0.0769 LDC-MEAN-ALL-Macro ISCAS_Sogou_KB_CMN_2 1 0.0059 LDC-MEAN-ALL-Macro ISCAS_Sogou_KB_CMN_2 ALL 0.0677 *ALL-Macro AP refer to mean of corresponding AP values. Scores based on P/R/F1; only k=1 justification allowed: Metric RunID Hop Prec Recall F1 SF-ALL-Micro ISCAS_Sogou_KB_CMN_2 0 0.1348 0.1655 0.1486 SF-ALL-Micro ISCAS_Sogou_KB_CMN_2 1 0.0094 0.0967 0.0171 SF-ALL-Micro ISCAS_Sogou_KB_CMN_2 ALL 0.0301 0.1397 0.0496 SF-ALL-Macro ISCAS_Sogou_KB_CMN_2 0 0.0943 0.1842 0.0873 SF-ALL-Macro ISCAS_Sogou_KB_CMN_2 1 0.0100 0.0394 0.0140 SF-ALL-Macro ISCAS_Sogou_KB_CMN_2 ALL 0.0482 0.1050 0.0472 LDC-MAX-ALL-Micro ISCAS_Sogou_KB_CMN_2 0 0.1628 0.2112 0.1839 LDC-MAX-ALL-Micro ISCAS_Sogou_KB_CMN_2 1 0.0203 0.1604 0.0360 LDC-MAX-ALL-Micro ISCAS_Sogou_KB_CMN_2 ALL 0.0579 0.1953 0.0894 LDC-MAX-ALL-Macro ISCAS_Sogou_KB_CMN_2 0 0.1123 0.2112 0.1147 LDC-MAX-ALL-Macro ISCAS_Sogou_KB_CMN_2 1 0.0085 0.0528 0.0146 LDC-MAX-ALL-Macro ISCAS_Sogou_KB_CMN_2 ALL 0.0649 0.1389 0.0690 LDC-MEAN-ALL-Macro ISCAS_Sogou_KB_CMN_2 0 0.0955 0.1758 0.0932 LDC-MEAN-ALL-Macro ISCAS_Sogou_KB_CMN_2 1 0.0102 0.0363 0.0136 LDC-MEAN-ALL-Macro ISCAS_Sogou_KB_CMN_2 ALL 0.0565 0.1121 0.0569 *ALL-Macro Prec, Recall and F1 refer to mean-precision, mean-recall and mean-F1. ************************************************************* Run ID: ISCAS_Sogou_KB_CMN_3 Did the run access the live Web during the evaluation window: No Did the run extract relations from the Cold Start source corpus: Yes Did the run generate meaningful confidence values: Yes Did the run attempt the full Entity Discovery and Linking task: No Did the run attempt the full Event Nugget Detection and Coreference task: Yes Did the run include SF relations: Yes Did the run attempt the Event Argument Extraction and Linking task: Yes Did the run include Sentiment relations: Yes --------------------- Composite KB Evaluation (All slot types): Scores based on confidence values and Average Precision (AP); k=3 justifications allowed: Metric RunID Hop AP SF-ALL-Macro ISCAS_Sogou_KB_CMN_3 0 0.0692 SF-ALL-Macro ISCAS_Sogou_KB_CMN_3 1 0.0116 SF-ALL-Macro ISCAS_Sogou_KB_CMN_3 ALL 0.0585 LDC-MEAN-ALL-Macro ISCAS_Sogou_KB_CMN_3 0 0.0759 LDC-MEAN-ALL-Macro ISCAS_Sogou_KB_CMN_3 1 0.0086 LDC-MEAN-ALL-Macro ISCAS_Sogou_KB_CMN_3 ALL 0.0664 (PRIMARY METRIC) *ALL-Macro AP refer to mean of corresponding AP values. Scores based on confidence values and Average Precision (AP); only k=1 justification allowed: Metric RunID Hop AP SF-ALL-Macro ISCAS_Sogou_KB_CMN_3 0 0.1043 SF-ALL-Macro ISCAS_Sogou_KB_CMN_3 1 0.0186 SF-ALL-Macro ISCAS_Sogou_KB_CMN_3 ALL 0.0880 LDC-MEAN-ALL-Macro ISCAS_Sogou_KB_CMN_3 0 0.1054 LDC-MEAN-ALL-Macro ISCAS_Sogou_KB_CMN_3 1 0.0138 LDC-MEAN-ALL-Macro ISCAS_Sogou_KB_CMN_3 ALL 0.0931 *ALL-Macro AP refer to mean of corresponding AP values. Scores based on P/R/F1; only k=1 justification allowed: Metric RunID Hop Prec Recall F1 SF-ALL-Micro ISCAS_Sogou_KB_CMN_3 0 0.2419 0.1334 0.1719 SF-ALL-Micro ISCAS_Sogou_KB_CMN_3 1 0.0131 0.0626 0.0217 SF-ALL-Micro ISCAS_Sogou_KB_CMN_3 ALL 0.0598 0.1113 0.0778 SF-ALL-Macro ISCAS_Sogou_KB_CMN_3 0 0.1440 0.1342 0.1106 SF-ALL-Macro ISCAS_Sogou_KB_CMN_3 1 0.0236 0.0449 0.0272 SF-ALL-Macro ISCAS_Sogou_KB_CMN_3 ALL 0.0992 0.1009 0.0796 LDC-MAX-ALL-Micro ISCAS_Sogou_KB_CMN_3 0 0.2832 0.1531 0.1987 LDC-MAX-ALL-Micro ISCAS_Sogou_KB_CMN_3 1 0.0130 0.0708 0.0219 LDC-MAX-ALL-Micro ISCAS_Sogou_KB_CMN_3 ALL 0.0605 0.1270 0.0819 LDC-MAX-ALL-Macro ISCAS_Sogou_KB_CMN_3 0 0.1804 0.1571 0.1397 LDC-MAX-ALL-Macro ISCAS_Sogou_KB_CMN_3 1 0.0296 0.0544 0.0342 LDC-MAX-ALL-Macro ISCAS_Sogou_KB_CMN_3 ALL 0.1277 0.1212 0.1028 LDC-MEAN-ALL-Macro ISCAS_Sogou_KB_CMN_3 0 0.1568 0.1318 0.1172 LDC-MEAN-ALL-Macro ISCAS_Sogou_KB_CMN_3 1 0.0259 0.0399 0.0283 LDC-MEAN-ALL-Macro ISCAS_Sogou_KB_CMN_3 ALL 0.1110 0.0997 0.0862 *ALL-Macro Prec, Recall and F1 refer to mean-precision, mean-recall and mean-F1. --------------------- Composite KB Evaluation (SF-SLOTS only): Scores based on confidence values and Average Precision (AP); k=3 justifications allowed: Metric RunID Hop AP SF-ALL-Macro ISCAS_Sogou_KB_CMN_3 0 0.0706 SF-ALL-Macro ISCAS_Sogou_KB_CMN_3 1 0.0159 SF-ALL-Macro ISCAS_Sogou_KB_CMN_3 ALL 0.0516 LDC-MEAN-ALL-Macro ISCAS_Sogou_KB_CMN_3 0 0.0709 LDC-MEAN-ALL-Macro ISCAS_Sogou_KB_CMN_3 1 0.0099 LDC-MEAN-ALL-Macro ISCAS_Sogou_KB_CMN_3 ALL 0.0537 (PRIMARY METRIC) *ALL-Macro AP refer to mean of corresponding AP values. Scores based on confidence values and Average Precision (AP); only k=1 justification allowed: Metric RunID Hop AP SF-ALL-Macro ISCAS_Sogou_KB_CMN_3 0 0.1096 SF-ALL-Macro ISCAS_Sogou_KB_CMN_3 1 0.0214 SF-ALL-Macro ISCAS_Sogou_KB_CMN_3 ALL 0.0805 LDC-MEAN-ALL-Macro ISCAS_Sogou_KB_CMN_3 0 0.1029 LDC-MEAN-ALL-Macro ISCAS_Sogou_KB_CMN_3 1 0.0143 LDC-MEAN-ALL-Macro ISCAS_Sogou_KB_CMN_3 ALL 0.0810 *ALL-Macro AP refer to mean of corresponding AP values. Scores based on P/R/F1; only k=1 justification allowed: Metric RunID Hop Prec Recall F1 SF-ALL-Micro ISCAS_Sogou_KB_CMN_3 0 0.2553 0.0773 0.1187 SF-ALL-Micro ISCAS_Sogou_KB_CMN_3 1 0.1443 0.0773 0.1007 SF-ALL-Micro ISCAS_Sogou_KB_CMN_3 ALL 0.2175 0.0773 0.1141 SF-ALL-Macro ISCAS_Sogou_KB_CMN_3 0 0.1097 0.1258 0.0969 SF-ALL-Macro ISCAS_Sogou_KB_CMN_3 1 0.0266 0.0360 0.0296 SF-ALL-Macro ISCAS_Sogou_KB_CMN_3 ALL 0.0814 0.0952 0.0740 LDC-MAX-ALL-Micro ISCAS_Sogou_KB_CMN_3 0 0.3000 0.0796 0.1259 LDC-MAX-ALL-Micro ISCAS_Sogou_KB_CMN_3 1 0.2241 0.1140 0.1512 LDC-MAX-ALL-Micro ISCAS_Sogou_KB_CMN_3 ALL 0.2753 0.0866 0.1317 LDC-MAX-ALL-Macro ISCAS_Sogou_KB_CMN_3 0 0.1404 0.1414 0.1199 LDC-MAX-ALL-Macro ISCAS_Sogou_KB_CMN_3 1 0.0254 0.0390 0.0297 LDC-MAX-ALL-Macro ISCAS_Sogou_KB_CMN_3 ALL 0.0992 0.1047 0.0876 LDC-MEAN-ALL-Macro ISCAS_Sogou_KB_CMN_3 0 0.1164 0.1154 0.0968 LDC-MEAN-ALL-Macro ISCAS_Sogou_KB_CMN_3 1 0.0180 0.0238 0.0198 LDC-MEAN-ALL-Macro ISCAS_Sogou_KB_CMN_3 ALL 0.0811 0.0826 0.0692 *ALL-Macro Prec, Recall and F1 refer to mean-precision, mean-recall and mean-F1. --------------------- Composite KB Evaluation (EVENT-SLOTS only): Scores based on confidence values and Average Precision (AP); k=3 justifications allowed: SUMMARY: This section provides summary of AP scores Metric RunID Hop AP SF-ALL-Macro ISCAS_Sogou_KB_CMN_3 0 0.0749 SF-ALL-Macro ISCAS_Sogou_KB_CMN_3 ALL 0.0749 LDC-MEAN-ALL-Macro ISCAS_Sogou_KB_CMN_3 0 0.0914 LDC-MEAN-ALL-Macro ISCAS_Sogou_KB_CMN_3 ALL 0.0914 (PRIMARY METRIC) *ALL-Macro AP refer to mean of corresponding AP values. Scores based on confidence values and Average Precision (AP); only k=1 justification allowed: SUMMARY: This section provides summary of AP scores Metric RunID Hop AP SF-ALL-Macro ISCAS_Sogou_KB_CMN_3 0 0.1120 SF-ALL-Macro ISCAS_Sogou_KB_CMN_3 ALL 0.1120 LDC-MEAN-ALL-Macro ISCAS_Sogou_KB_CMN_3 0 0.1246 LDC-MEAN-ALL-Macro ISCAS_Sogou_KB_CMN_3 ALL 0.1246 *ALL-Macro AP refer to mean of corresponding AP values. Scores based on P/R/F1; only k=1 justification allowed: Metric RunID Hop Prec Recall F1 SF-ALL-Micro ISCAS_Sogou_KB_CMN_3 0 0.5938 0.1699 0.2643 SF-ALL-Micro ISCAS_Sogou_KB_CMN_3 ALL 0.5938 0.1699 0.2643 SF-ALL-Macro ISCAS_Sogou_KB_CMN_3 0 0.2204 0.1187 0.1430 SF-ALL-Macro ISCAS_Sogou_KB_CMN_3 ALL 0.2204 0.1187 0.1430 LDC-MAX-ALL-Micro ISCAS_Sogou_KB_CMN_3 0 0.6134 0.2098 0.3126 LDC-MAX-ALL-Micro ISCAS_Sogou_KB_CMN_3 ALL 0.6134 0.2098 0.3126 LDC-MAX-ALL-Macro ISCAS_Sogou_KB_CMN_3 0 0.2765 0.1512 0.1825 LDC-MAX-ALL-Macro ISCAS_Sogou_KB_CMN_3 ALL 0.2765 0.1512 0.1825 LDC-MEAN-ALL-Macro ISCAS_Sogou_KB_CMN_3 0 0.2497 0.1324 0.1605 LDC-MEAN-ALL-Macro ISCAS_Sogou_KB_CMN_3 ALL 0.2497 0.1324 0.1605 *ALL-Macro Prec, Recall and F1 refer to mean-precision, mean-recall and mean-F1. --------------------- Composite KB Evaluation (SENTIMENT-SLOTS only): Scores based on confidence values and Average Precision (AP); k=3 justifications allowed: Metric RunID Hop AP SF-ALL-Macro ISCAS_Sogou_KB_CMN_3 0 0.0555 SF-ALL-Macro ISCAS_Sogou_KB_CMN_3 1 0.0029 SF-ALL-Macro ISCAS_Sogou_KB_CMN_3 ALL 0.0468 LDC-MEAN-ALL-Macro ISCAS_Sogou_KB_CMN_3 0 0.0607 LDC-MEAN-ALL-Macro ISCAS_Sogou_KB_CMN_3 1 0.0023 LDC-MEAN-ALL-Macro ISCAS_Sogou_KB_CMN_3 ALL 0.0549 (PRIMARY METRIC) *ALL-Macro AP refer to mean of corresponding AP values. Scores based on confidence values and Average Precision (AP); only k=1 justification allowed: Metric RunID Hop AP SF-ALL-Macro ISCAS_Sogou_KB_CMN_3 0 0.0764 SF-ALL-Macro ISCAS_Sogou_KB_CMN_3 1 0.0075 SF-ALL-Macro ISCAS_Sogou_KB_CMN_3 ALL 0.0635 LDC-MEAN-ALL-Macro ISCAS_Sogou_KB_CMN_3 0 0.0769 LDC-MEAN-ALL-Macro ISCAS_Sogou_KB_CMN_3 1 0.0059 LDC-MEAN-ALL-Macro ISCAS_Sogou_KB_CMN_3 ALL 0.0677 *ALL-Macro AP refer to mean of corresponding AP values. Scores based on P/R/F1; only k=1 justification allowed: Metric RunID Hop Prec Recall F1 SF-ALL-Micro ISCAS_Sogou_KB_CMN_3 0 0.1348 0.1655 0.1486 SF-ALL-Micro ISCAS_Sogou_KB_CMN_3 1 0.0094 0.0967 0.0171 SF-ALL-Micro ISCAS_Sogou_KB_CMN_3 ALL 0.0301 0.1397 0.0496 SF-ALL-Macro ISCAS_Sogou_KB_CMN_3 0 0.0943 0.1842 0.0873 SF-ALL-Macro ISCAS_Sogou_KB_CMN_3 1 0.0100 0.0394 0.0140 SF-ALL-Macro ISCAS_Sogou_KB_CMN_3 ALL 0.0482 0.1050 0.0472 LDC-MAX-ALL-Micro ISCAS_Sogou_KB_CMN_3 0 0.1522 0.2112 0.1769 LDC-MAX-ALL-Micro ISCAS_Sogou_KB_CMN_3 1 0.0076 0.1604 0.0145 LDC-MAX-ALL-Micro ISCAS_Sogou_KB_CMN_3 ALL 0.0258 0.1953 0.0456 LDC-MAX-ALL-Macro ISCAS_Sogou_KB_CMN_3 0 0.1123 0.2112 0.1147 LDC-MAX-ALL-Macro ISCAS_Sogou_KB_CMN_3 1 0.0085 0.0528 0.0146 LDC-MAX-ALL-Macro ISCAS_Sogou_KB_CMN_3 ALL 0.0649 0.1389 0.0690 LDC-MEAN-ALL-Macro ISCAS_Sogou_KB_CMN_3 0 0.0955 0.1758 0.0932 LDC-MEAN-ALL-Macro ISCAS_Sogou_KB_CMN_3 1 0.0102 0.0363 0.0136 LDC-MEAN-ALL-Macro ISCAS_Sogou_KB_CMN_3 ALL 0.0565 0.1121 0.0569 *ALL-Macro Prec, Recall and F1 refer to mean-precision, mean-recall and mean-F1. ************************************************************* Run ID: ISCAS_Sogou_KB_CMN_4 Did the run access the live Web during the evaluation window: No Did the run extract relations from the Cold Start source corpus: Yes Did the run generate meaningful confidence values: Yes Did the run attempt the full Entity Discovery and Linking task: No Did the run attempt the full Event Nugget Detection and Coreference task: Yes Did the run include SF relations: Yes Did the run attempt the Event Argument Extraction and Linking task: Yes Did the run include Sentiment relations: Yes --------------------- Composite KB Evaluation (All slot types): Scores based on confidence values and Average Precision (AP); k=3 justifications allowed: Metric RunID Hop AP SF-ALL-Macro ISCAS_Sogou_KB_CMN_4 0 0.0737 SF-ALL-Macro ISCAS_Sogou_KB_CMN_4 1 0.0128 SF-ALL-Macro ISCAS_Sogou_KB_CMN_4 ALL 0.0631 LDC-MEAN-ALL-Macro ISCAS_Sogou_KB_CMN_4 0 0.0795 LDC-MEAN-ALL-Macro ISCAS_Sogou_KB_CMN_4 1 0.0106 LDC-MEAN-ALL-Macro ISCAS_Sogou_KB_CMN_4 ALL 0.0701 (PRIMARY METRIC) *ALL-Macro AP refer to mean of corresponding AP values. Scores based on confidence values and Average Precision (AP); only k=1 justification allowed: Metric RunID Hop AP SF-ALL-Macro ISCAS_Sogou_KB_CMN_4 0 0.1091 SF-ALL-Macro ISCAS_Sogou_KB_CMN_4 1 0.0158 SF-ALL-Macro ISCAS_Sogou_KB_CMN_4 ALL 0.0923 LDC-MEAN-ALL-Macro ISCAS_Sogou_KB_CMN_4 0 0.1094 LDC-MEAN-ALL-Macro ISCAS_Sogou_KB_CMN_4 1 0.0140 LDC-MEAN-ALL-Macro ISCAS_Sogou_KB_CMN_4 ALL 0.0967 *ALL-Macro AP refer to mean of corresponding AP values. Scores based on P/R/F1; only k=1 justification allowed: Metric RunID Hop Prec Recall F1 SF-ALL-Micro ISCAS_Sogou_KB_CMN_4 0 0.3645 0.1223 0.1832 SF-ALL-Micro ISCAS_Sogou_KB_CMN_4 1 0.0363 0.0544 0.0435 SF-ALL-Micro ISCAS_Sogou_KB_CMN_4 ALL 0.1449 0.1012 0.1192 SF-ALL-Macro ISCAS_Sogou_KB_CMN_4 0 0.1565 0.1287 0.1148 SF-ALL-Macro ISCAS_Sogou_KB_CMN_4 1 0.0282 0.0434 0.0312 SF-ALL-Macro ISCAS_Sogou_KB_CMN_4 ALL 0.1087 0.0969 0.0837 LDC-MAX-ALL-Micro ISCAS_Sogou_KB_CMN_4 0 0.4249 0.1424 0.2134 LDC-MAX-ALL-Micro ISCAS_Sogou_KB_CMN_4 1 0.0359 0.0625 0.0456 LDC-MAX-ALL-Micro ISCAS_Sogou_KB_CMN_4 ALL 0.1499 0.1171 0.1315 LDC-MAX-ALL-Macro ISCAS_Sogou_KB_CMN_4 0 0.1991 0.1494 0.1449 LDC-MAX-ALL-Macro ISCAS_Sogou_KB_CMN_4 1 0.0335 0.0527 0.0386 LDC-MAX-ALL-Macro ISCAS_Sogou_KB_CMN_4 ALL 0.1412 0.1156 0.1078 LDC-MEAN-ALL-Macro ISCAS_Sogou_KB_CMN_4 0 0.1693 0.1256 0.1204 LDC-MEAN-ALL-Macro ISCAS_Sogou_KB_CMN_4 1 0.0291 0.0386 0.0309 LDC-MEAN-ALL-Macro ISCAS_Sogou_KB_CMN_4 ALL 0.1203 0.0952 0.0891 *ALL-Macro Prec, Recall and F1 refer to mean-precision, mean-recall and mean-F1. --------------------- Composite KB Evaluation (SF-SLOTS only): Scores based on confidence values and Average Precision (AP); k=3 justifications allowed: Metric RunID Hop AP SF-ALL-Macro ISCAS_Sogou_KB_CMN_4 0 0.0706 SF-ALL-Macro ISCAS_Sogou_KB_CMN_4 1 0.0159 SF-ALL-Macro ISCAS_Sogou_KB_CMN_4 ALL 0.0516 LDC-MEAN-ALL-Macro ISCAS_Sogou_KB_CMN_4 0 0.0709 LDC-MEAN-ALL-Macro ISCAS_Sogou_KB_CMN_4 1 0.0099 LDC-MEAN-ALL-Macro ISCAS_Sogou_KB_CMN_4 ALL 0.0537 (PRIMARY METRIC) *ALL-Macro AP refer to mean of corresponding AP values. Scores based on confidence values and Average Precision (AP); only k=1 justification allowed: Metric RunID Hop AP SF-ALL-Macro ISCAS_Sogou_KB_CMN_4 0 0.1096 SF-ALL-Macro ISCAS_Sogou_KB_CMN_4 1 0.0214 SF-ALL-Macro ISCAS_Sogou_KB_CMN_4 ALL 0.0805 LDC-MEAN-ALL-Macro ISCAS_Sogou_KB_CMN_4 0 0.1029 LDC-MEAN-ALL-Macro ISCAS_Sogou_KB_CMN_4 1 0.0143 LDC-MEAN-ALL-Macro ISCAS_Sogou_KB_CMN_4 ALL 0.0810 *ALL-Macro AP refer to mean of corresponding AP values. Scores based on P/R/F1; only k=1 justification allowed: Metric RunID Hop Prec Recall F1 SF-ALL-Micro ISCAS_Sogou_KB_CMN_4 0 0.2553 0.0773 0.1187 SF-ALL-Micro ISCAS_Sogou_KB_CMN_4 1 0.1443 0.0773 0.1007 SF-ALL-Micro ISCAS_Sogou_KB_CMN_4 ALL 0.2175 0.0773 0.1141 SF-ALL-Macro ISCAS_Sogou_KB_CMN_4 0 0.1097 0.1258 0.0969 SF-ALL-Macro ISCAS_Sogou_KB_CMN_4 1 0.0266 0.0360 0.0296 SF-ALL-Macro ISCAS_Sogou_KB_CMN_4 ALL 0.0814 0.0952 0.0740 LDC-MAX-ALL-Micro ISCAS_Sogou_KB_CMN_4 0 0.3051 0.0796 0.1263 LDC-MAX-ALL-Micro ISCAS_Sogou_KB_CMN_4 1 0.2241 0.1140 0.1512 LDC-MAX-ALL-Micro ISCAS_Sogou_KB_CMN_4 ALL 0.2784 0.0866 0.1321 LDC-MAX-ALL-Macro ISCAS_Sogou_KB_CMN_4 0 0.1404 0.1414 0.1199 LDC-MAX-ALL-Macro ISCAS_Sogou_KB_CMN_4 1 0.0254 0.0390 0.0297 LDC-MAX-ALL-Macro ISCAS_Sogou_KB_CMN_4 ALL 0.0992 0.1047 0.0876 LDC-MEAN-ALL-Macro ISCAS_Sogou_KB_CMN_4 0 0.1164 0.1154 0.0968 LDC-MEAN-ALL-Macro ISCAS_Sogou_KB_CMN_4 1 0.0180 0.0238 0.0198 LDC-MEAN-ALL-Macro ISCAS_Sogou_KB_CMN_4 ALL 0.0811 0.0826 0.0692 *ALL-Macro Prec, Recall and F1 refer to mean-precision, mean-recall and mean-F1. --------------------- Composite KB Evaluation (EVENT-SLOTS only): Scores based on confidence values and Average Precision (AP); k=3 justifications allowed: SUMMARY: This section provides summary of AP scores Metric RunID Hop AP SF-ALL-Macro ISCAS_Sogou_KB_CMN_4 0 0.0749 SF-ALL-Macro ISCAS_Sogou_KB_CMN_4 ALL 0.0749 LDC-MEAN-ALL-Macro ISCAS_Sogou_KB_CMN_4 0 0.0914 LDC-MEAN-ALL-Macro ISCAS_Sogou_KB_CMN_4 ALL 0.0914 (PRIMARY METRIC) *ALL-Macro AP refer to mean of corresponding AP values. Scores based on confidence values and Average Precision (AP); only k=1 justification allowed: SUMMARY: This section provides summary of AP scores Metric RunID Hop AP SF-ALL-Macro ISCAS_Sogou_KB_CMN_4 0 0.1120 SF-ALL-Macro ISCAS_Sogou_KB_CMN_4 ALL 0.1120 LDC-MEAN-ALL-Macro ISCAS_Sogou_KB_CMN_4 0 0.1246 LDC-MEAN-ALL-Macro ISCAS_Sogou_KB_CMN_4 ALL 0.1246 *ALL-Macro AP refer to mean of corresponding AP values. Scores based on P/R/F1; only k=1 justification allowed: Metric RunID Hop Prec Recall F1 SF-ALL-Micro ISCAS_Sogou_KB_CMN_4 0 0.5938 0.1699 0.2643 SF-ALL-Micro ISCAS_Sogou_KB_CMN_4 ALL 0.5938 0.1699 0.2643 SF-ALL-Macro ISCAS_Sogou_KB_CMN_4 0 0.2204 0.1187 0.1430 SF-ALL-Macro ISCAS_Sogou_KB_CMN_4 ALL 0.2204 0.1187 0.1430 LDC-MAX-ALL-Micro ISCAS_Sogou_KB_CMN_4 0 0.6186 0.2098 0.3133 LDC-MAX-ALL-Micro ISCAS_Sogou_KB_CMN_4 ALL 0.6186 0.2098 0.3133 LDC-MAX-ALL-Macro ISCAS_Sogou_KB_CMN_4 0 0.2765 0.1512 0.1825 LDC-MAX-ALL-Macro ISCAS_Sogou_KB_CMN_4 ALL 0.2765 0.1512 0.1825 LDC-MEAN-ALL-Macro ISCAS_Sogou_KB_CMN_4 0 0.2497 0.1324 0.1605 LDC-MEAN-ALL-Macro ISCAS_Sogou_KB_CMN_4 ALL 0.2497 0.1324 0.1605 *ALL-Macro Prec, Recall and F1 refer to mean-precision, mean-recall and mean-F1. --------------------- Composite KB Evaluation (SENTIMENT-SLOTS only): Scores based on confidence values and Average Precision (AP); k=3 justifications allowed: Metric RunID Hop AP SF-ALL-Macro ISCAS_Sogou_KB_CMN_4 0 0.0797 SF-ALL-Macro ISCAS_Sogou_KB_CMN_4 1 0.0037 SF-ALL-Macro ISCAS_Sogou_KB_CMN_4 ALL 0.0714 LDC-MEAN-ALL-Macro ISCAS_Sogou_KB_CMN_4 0 0.0812 LDC-MEAN-ALL-Macro ISCAS_Sogou_KB_CMN_4 1 0.0029 LDC-MEAN-ALL-Macro ISCAS_Sogou_KB_CMN_4 ALL 0.0756 (PRIMARY METRIC) *ALL-Macro AP refer to mean of corresponding AP values. Scores based on confidence values and Average Precision (AP); only k=1 justification allowed: Metric RunID Hop AP SF-ALL-Macro ISCAS_Sogou_KB_CMN_4 0 0.1025 SF-ALL-Macro ISCAS_Sogou_KB_CMN_4 1 0.0056 SF-ALL-Macro ISCAS_Sogou_KB_CMN_4 ALL 0.0871 LDC-MEAN-ALL-Macro ISCAS_Sogou_KB_CMN_4 0 0.0991 LDC-MEAN-ALL-Macro ISCAS_Sogou_KB_CMN_4 1 0.0044 LDC-MEAN-ALL-Macro ISCAS_Sogou_KB_CMN_4 ALL 0.0884 *ALL-Macro AP refer to mean of corresponding AP values. Scores based on P/R/F1; only k=1 justification allowed: Metric RunID Hop Prec Recall F1 SF-ALL-Micro ISCAS_Sogou_KB_CMN_4 0 0.2828 0.1253 0.1736 SF-ALL-Micro ISCAS_Sogou_KB_CMN_4 1 0.0313 0.0743 0.0441 SF-ALL-Micro ISCAS_Sogou_KB_CMN_4 ALL 0.0908 0.1061 0.0979 SF-ALL-Macro ISCAS_Sogou_KB_CMN_4 0 0.1619 0.1544 0.1100 SF-ALL-Macro ISCAS_Sogou_KB_CMN_4 1 0.0223 0.0354 0.0247 SF-ALL-Macro ISCAS_Sogou_KB_CMN_4 ALL 0.0855 0.0893 0.0634 LDC-MAX-ALL-Micro ISCAS_Sogou_KB_CMN_4 0 0.3455 0.1638 0.2222 LDC-MAX-ALL-Micro ISCAS_Sogou_KB_CMN_4 1 0.0244 0.1226 0.0408 LDC-MAX-ALL-Micro ISCAS_Sogou_KB_CMN_4 ALL 0.0794 0.1509 0.1041 LDC-MAX-ALL-Macro ISCAS_Sogou_KB_CMN_4 0 0.2171 0.1681 0.1442 LDC-MAX-ALL-Macro ISCAS_Sogou_KB_CMN_4 1 0.0227 0.0465 0.0303 LDC-MAX-ALL-Macro ISCAS_Sogou_KB_CMN_4 ALL 0.1284 0.1126 0.0922 LDC-MEAN-ALL-Macro ISCAS_Sogou_KB_CMN_4 0 0.1657 0.1411 0.1107 LDC-MEAN-ALL-Macro ISCAS_Sogou_KB_CMN_4 1 0.0216 0.0315 0.0228 LDC-MEAN-ALL-Macro ISCAS_Sogou_KB_CMN_4 ALL 0.0999 0.0911 0.0706 *ALL-Macro Prec, Recall and F1 refer to mean-precision, mean-recall and mean-F1. ************************************************************* Run ID: ISCAS_Sogou_KB_CMN_5 Did the run access the live Web during the evaluation window: No Did the run extract relations from the Cold Start source corpus: Yes Did the run generate meaningful confidence values: Yes Did the run attempt the full Entity Discovery and Linking task: No Did the run attempt the full Event Nugget Detection and Coreference task: Yes Did the run include SF relations: Yes Did the run attempt the Event Argument Extraction and Linking task: Yes Did the run include Sentiment relations: Yes --------------------- Composite KB Evaluation (All slot types): Scores based on confidence values and Average Precision (AP); k=3 justifications allowed: Metric RunID Hop AP SF-ALL-Macro ISCAS_Sogou_KB_CMN_5 0 0.0689 SF-ALL-Macro ISCAS_Sogou_KB_CMN_5 1 0.0124 SF-ALL-Macro ISCAS_Sogou_KB_CMN_5 ALL 0.0584 LDC-MEAN-ALL-Macro ISCAS_Sogou_KB_CMN_5 0 0.0756 LDC-MEAN-ALL-Macro ISCAS_Sogou_KB_CMN_5 1 0.0091 LDC-MEAN-ALL-Macro ISCAS_Sogou_KB_CMN_5 ALL 0.0662 (PRIMARY METRIC) *ALL-Macro AP refer to mean of corresponding AP values. Scores based on confidence values and Average Precision (AP); only k=1 justification allowed: Metric RunID Hop AP SF-ALL-Macro ISCAS_Sogou_KB_CMN_5 0 0.1033 SF-ALL-Macro ISCAS_Sogou_KB_CMN_5 1 0.0187 SF-ALL-Macro ISCAS_Sogou_KB_CMN_5 ALL 0.0869 LDC-MEAN-ALL-Macro ISCAS_Sogou_KB_CMN_5 0 0.1046 LDC-MEAN-ALL-Macro ISCAS_Sogou_KB_CMN_5 1 0.0139 LDC-MEAN-ALL-Macro ISCAS_Sogou_KB_CMN_5 ALL 0.0922 *ALL-Macro AP refer to mean of corresponding AP values. Scores based on P/R/F1; only k=1 justification allowed: Metric RunID Hop Prec Recall F1 SF-ALL-Micro ISCAS_Sogou_KB_CMN_5 0 0.2409 0.1340 0.1722 SF-ALL-Micro ISCAS_Sogou_KB_CMN_5 1 0.0134 0.0639 0.0222 SF-ALL-Micro ISCAS_Sogou_KB_CMN_5 ALL 0.0602 0.1122 0.0783 SF-ALL-Macro ISCAS_Sogou_KB_CMN_5 0 0.1446 0.1353 0.1112 SF-ALL-Macro ISCAS_Sogou_KB_CMN_5 1 0.0256 0.0457 0.0284 SF-ALL-Macro ISCAS_Sogou_KB_CMN_5 ALL 0.1003 0.1019 0.0804 LDC-MAX-ALL-Micro ISCAS_Sogou_KB_CMN_5 0 0.2837 0.1531 0.1989 LDC-MAX-ALL-Micro ISCAS_Sogou_KB_CMN_5 1 0.0122 0.0708 0.0208 LDC-MAX-ALL-Micro ISCAS_Sogou_KB_CMN_5 ALL 0.0573 0.1270 0.0789 LDC-MAX-ALL-Macro ISCAS_Sogou_KB_CMN_5 0 0.1805 0.1571 0.1397 LDC-MAX-ALL-Macro ISCAS_Sogou_KB_CMN_5 1 0.0297 0.0544 0.0343 LDC-MAX-ALL-Macro ISCAS_Sogou_KB_CMN_5 ALL 0.1277 0.1212 0.1029 LDC-MEAN-ALL-Macro ISCAS_Sogou_KB_CMN_5 0 0.1571 0.1327 0.1175 LDC-MEAN-ALL-Macro ISCAS_Sogou_KB_CMN_5 1 0.0276 0.0406 0.0293 LDC-MEAN-ALL-Macro ISCAS_Sogou_KB_CMN_5 ALL 0.1118 0.1005 0.0867 *ALL-Macro Prec, Recall and F1 refer to mean-precision, mean-recall and mean-F1. --------------------- Composite KB Evaluation (SF-SLOTS only): Scores based on confidence values and Average Precision (AP); k=3 justifications allowed: Metric RunID Hop AP SF-ALL-Macro ISCAS_Sogou_KB_CMN_5 0 0.0699 SF-ALL-Macro ISCAS_Sogou_KB_CMN_5 1 0.0160 SF-ALL-Macro ISCAS_Sogou_KB_CMN_5 ALL 0.0509 LDC-MEAN-ALL-Macro ISCAS_Sogou_KB_CMN_5 0 0.0704 LDC-MEAN-ALL-Macro ISCAS_Sogou_KB_CMN_5 1 0.0099 LDC-MEAN-ALL-Macro ISCAS_Sogou_KB_CMN_5 ALL 0.0531 (PRIMARY METRIC) *ALL-Macro AP refer to mean of corresponding AP values. Scores based on confidence values and Average Precision (AP); only k=1 justification allowed: Metric RunID Hop AP SF-ALL-Macro ISCAS_Sogou_KB_CMN_5 0 0.1075 SF-ALL-Macro ISCAS_Sogou_KB_CMN_5 1 0.0215 SF-ALL-Macro ISCAS_Sogou_KB_CMN_5 ALL 0.0783 LDC-MEAN-ALL-Macro ISCAS_Sogou_KB_CMN_5 0 0.1013 LDC-MEAN-ALL-Macro ISCAS_Sogou_KB_CMN_5 1 0.0144 LDC-MEAN-ALL-Macro ISCAS_Sogou_KB_CMN_5 ALL 0.0793 *ALL-Macro AP refer to mean of corresponding AP values. Scores based on P/R/F1; only k=1 justification allowed: Metric RunID Hop Prec Recall F1 SF-ALL-Micro ISCAS_Sogou_KB_CMN_5 0 0.2500 0.0789 0.1200 SF-ALL-Micro ISCAS_Sogou_KB_CMN_5 1 0.1282 0.0829 0.1007 SF-ALL-Micro ISCAS_Sogou_KB_CMN_5 ALL 0.2045 0.0798 0.1148 SF-ALL-Macro ISCAS_Sogou_KB_CMN_5 0 0.1110 0.1281 0.0981 SF-ALL-Macro ISCAS_Sogou_KB_CMN_5 1 0.0311 0.0378 0.0322 SF-ALL-Macro ISCAS_Sogou_KB_CMN_5 ALL 0.0838 0.0974 0.0756 LDC-MAX-ALL-Micro ISCAS_Sogou_KB_CMN_5 0 0.2835 0.0796 0.1244 LDC-MAX-ALL-Micro ISCAS_Sogou_KB_CMN_5 1 0.2321 0.1140 0.1529 LDC-MAX-ALL-Micro ISCAS_Sogou_KB_CMN_5 ALL 0.2678 0.0866 0.1308 LDC-MAX-ALL-Macro ISCAS_Sogou_KB_CMN_5 0 0.1404 0.1414 0.1198 LDC-MAX-ALL-Macro ISCAS_Sogou_KB_CMN_5 1 0.0254 0.0390 0.0297 LDC-MAX-ALL-Macro ISCAS_Sogou_KB_CMN_5 ALL 0.0992 0.1047 0.0875 LDC-MEAN-ALL-Macro ISCAS_Sogou_KB_CMN_5 0 0.1170 0.1172 0.0974 LDC-MEAN-ALL-Macro ISCAS_Sogou_KB_CMN_5 1 0.0212 0.0251 0.0216 LDC-MEAN-ALL-Macro ISCAS_Sogou_KB_CMN_5 ALL 0.0827 0.0842 0.0703 *ALL-Macro Prec, Recall and F1 refer to mean-precision, mean-recall and mean-F1. --------------------- Composite KB Evaluation (EVENT-SLOTS only): Scores based on confidence values and Average Precision (AP); k=3 justifications allowed: SUMMARY: This section provides summary of AP scores Metric RunID Hop AP SF-ALL-Macro ISCAS_Sogou_KB_CMN_5 0 0.0749 SF-ALL-Macro ISCAS_Sogou_KB_CMN_5 ALL 0.0749 LDC-MEAN-ALL-Macro ISCAS_Sogou_KB_CMN_5 0 0.0914 LDC-MEAN-ALL-Macro ISCAS_Sogou_KB_CMN_5 ALL 0.0914 (PRIMARY METRIC) *ALL-Macro AP refer to mean of corresponding AP values. Scores based on confidence values and Average Precision (AP); only k=1 justification allowed: SUMMARY: This section provides summary of AP scores Metric RunID Hop AP SF-ALL-Macro ISCAS_Sogou_KB_CMN_5 0 0.1120 SF-ALL-Macro ISCAS_Sogou_KB_CMN_5 ALL 0.1120 LDC-MEAN-ALL-Macro ISCAS_Sogou_KB_CMN_5 0 0.1246 LDC-MEAN-ALL-Macro ISCAS_Sogou_KB_CMN_5 ALL 0.1246 *ALL-Macro AP refer to mean of corresponding AP values. Scores based on P/R/F1; only k=1 justification allowed: Metric RunID Hop Prec Recall F1 SF-ALL-Micro ISCAS_Sogou_KB_CMN_5 0 0.5938 0.1699 0.2643 SF-ALL-Micro ISCAS_Sogou_KB_CMN_5 ALL 0.5938 0.1699 0.2643 SF-ALL-Macro ISCAS_Sogou_KB_CMN_5 0 0.2204 0.1187 0.1430 SF-ALL-Macro ISCAS_Sogou_KB_CMN_5 ALL 0.2204 0.1187 0.1430 LDC-MAX-ALL-Micro ISCAS_Sogou_KB_CMN_5 0 0.5887 0.2098 0.3093 LDC-MAX-ALL-Micro ISCAS_Sogou_KB_CMN_5 ALL 0.5887 0.2098 0.3093 LDC-MAX-ALL-Macro ISCAS_Sogou_KB_CMN_5 0 0.2765 0.1512 0.1825 LDC-MAX-ALL-Macro ISCAS_Sogou_KB_CMN_5 ALL 0.2765 0.1512 0.1825 LDC-MEAN-ALL-Macro ISCAS_Sogou_KB_CMN_5 0 0.2497 0.1324 0.1605 LDC-MEAN-ALL-Macro ISCAS_Sogou_KB_CMN_5 ALL 0.2497 0.1324 0.1605 *ALL-Macro Prec, Recall and F1 refer to mean-precision, mean-recall and mean-F1. --------------------- Composite KB Evaluation (SENTIMENT-SLOTS only): Scores based on confidence values and Average Precision (AP); k=3 justifications allowed: Metric RunID Hop AP SF-ALL-Macro ISCAS_Sogou_KB_CMN_5 0 0.0555 SF-ALL-Macro ISCAS_Sogou_KB_CMN_5 1 0.0029 SF-ALL-Macro ISCAS_Sogou_KB_CMN_5 ALL 0.0479 LDC-MEAN-ALL-Macro ISCAS_Sogou_KB_CMN_5 0 0.0607 LDC-MEAN-ALL-Macro ISCAS_Sogou_KB_CMN_5 1 0.0023 LDC-MEAN-ALL-Macro ISCAS_Sogou_KB_CMN_5 ALL 0.0555 (PRIMARY METRIC) *ALL-Macro AP refer to mean of corresponding AP values. Scores based on confidence values and Average Precision (AP); only k=1 justification allowed: Metric RunID Hop AP SF-ALL-Macro ISCAS_Sogou_KB_CMN_5 0 0.0764 SF-ALL-Macro ISCAS_Sogou_KB_CMN_5 1 0.0075 SF-ALL-Macro ISCAS_Sogou_KB_CMN_5 ALL 0.0636 LDC-MEAN-ALL-Macro ISCAS_Sogou_KB_CMN_5 0 0.0769 LDC-MEAN-ALL-Macro ISCAS_Sogou_KB_CMN_5 1 0.0059 LDC-MEAN-ALL-Macro ISCAS_Sogou_KB_CMN_5 ALL 0.0678 *ALL-Macro AP refer to mean of corresponding AP values. Scores based on P/R/F1; only k=1 justification allowed: Metric RunID Hop Prec Recall F1 SF-ALL-Micro ISCAS_Sogou_KB_CMN_5 0 0.1348 0.1655 0.1486 SF-ALL-Micro ISCAS_Sogou_KB_CMN_5 1 0.0094 0.0967 0.0171 SF-ALL-Micro ISCAS_Sogou_KB_CMN_5 ALL 0.0301 0.1397 0.0496 SF-ALL-Macro ISCAS_Sogou_KB_CMN_5 0 0.0943 0.1842 0.0873 SF-ALL-Macro ISCAS_Sogou_KB_CMN_5 1 0.0100 0.0394 0.0140 SF-ALL-Macro ISCAS_Sogou_KB_CMN_5 ALL 0.0482 0.1050 0.0472 LDC-MAX-ALL-Micro ISCAS_Sogou_KB_CMN_5 0 0.1678 0.2112 0.1870 LDC-MAX-ALL-Micro ISCAS_Sogou_KB_CMN_5 1 0.0227 0.1604 0.0397 LDC-MAX-ALL-Micro ISCAS_Sogou_KB_CMN_5 ALL 0.0633 0.1953 0.0957 LDC-MAX-ALL-Macro ISCAS_Sogou_KB_CMN_5 0 0.1123 0.2112 0.1147 LDC-MAX-ALL-Macro ISCAS_Sogou_KB_CMN_5 1 0.0085 0.0528 0.0146 LDC-MAX-ALL-Macro ISCAS_Sogou_KB_CMN_5 ALL 0.0649 0.1389 0.0690 LDC-MEAN-ALL-Macro ISCAS_Sogou_KB_CMN_5 0 0.0955 0.1758 0.0932 LDC-MEAN-ALL-Macro ISCAS_Sogou_KB_CMN_5 1 0.0102 0.0363 0.0136 LDC-MEAN-ALL-Macro ISCAS_Sogou_KB_CMN_5 ALL 0.0565 0.1121 0.0569 *ALL-Macro Prec, Recall and F1 refer to mean-precision, mean-recall and mean-F1.