===================================================================== TAC KBP 2017 ENGLISH KB CONSTRUCTION: COMPOSITE KB EVALUATION RESULTS ===================================================================== Team ID: ICTCAS_OKN Organization: Institute of Computing Technology, Chinese Academy of Sciences ************************************************************* Run ID: ICTCAS_OKN_KB_ENG_1 Did the run access the live Web during the evaluation window: No Did the run extract relations from the Cold Start source corpus: Yes Did the run generate meaningful confidence values: Yes Did the run attempt the full Entity Discovery and Linking task: No Did the run attempt the full Event Nugget Detection and Coreference task: Yes Did the run include SF relations: Yes Did the run attempt the Event Argument Extraction and Linking task: No Did the run include Sentiment relations: Yes --------------------- Composite KB Evaluation (All slot types): Scores based on confidence values and Average Precision (AP); k=3 justifications allowed: Metric RunID Hop AP SF-ALL-Macro ICTCAS_OKN_KB_ENG_1 0 0.0187 SF-ALL-Macro ICTCAS_OKN_KB_ENG_1 1 0.0001 SF-ALL-Macro ICTCAS_OKN_KB_ENG_1 ALL 0.0134 LDC-MEAN-ALL-Macro ICTCAS_OKN_KB_ENG_1 0 0.0225 LDC-MEAN-ALL-Macro ICTCAS_OKN_KB_ENG_1 1 0.0002 LDC-MEAN-ALL-Macro ICTCAS_OKN_KB_ENG_1 ALL 0.0164 (PRIMARY METRIC) *ALL-Macro AP refer to mean of corresponding AP values. Scores based on confidence values and Average Precision (AP); only k=1 justification allowed: Metric RunID Hop AP SF-ALL-Macro ICTCAS_OKN_KB_ENG_1 0 0.0316 SF-ALL-Macro ICTCAS_OKN_KB_ENG_1 1 0.0001 SF-ALL-Macro ICTCAS_OKN_KB_ENG_1 ALL 0.0237 LDC-MEAN-ALL-Macro ICTCAS_OKN_KB_ENG_1 0 0.0349 LDC-MEAN-ALL-Macro ICTCAS_OKN_KB_ENG_1 1 0.0002 LDC-MEAN-ALL-Macro ICTCAS_OKN_KB_ENG_1 ALL 0.0269 *ALL-Macro AP refer to mean of corresponding AP values. Scores based on P/R/F1; only k=1 justification allowed: Metric RunID Hop Prec Recall F1 SF-ALL-Micro ICTCAS_OKN_KB_ENG_1 0 0.1988 0.0436 0.0716 SF-ALL-Micro ICTCAS_OKN_KB_ENG_1 1 0.0007 0.0015 0.0009 SF-ALL-Micro ICTCAS_OKN_KB_ENG_1 ALL 0.0523 0.0340 0.0413 SF-ALL-Macro ICTCAS_OKN_KB_ENG_1 0 0.0390 0.0395 0.0316 SF-ALL-Macro ICTCAS_OKN_KB_ENG_1 1 0.0021 0.0021 0.0021 SF-ALL-Macro ICTCAS_OKN_KB_ENG_1 ALL 0.0263 0.0266 0.0215 LDC-MAX-ALL-Micro ICTCAS_OKN_KB_ENG_1 0 0.2480 0.0624 0.0998 LDC-MAX-ALL-Micro ICTCAS_OKN_KB_ENG_1 1 0.0000 0.0000 0.0000 LDC-MAX-ALL-Micro ICTCAS_OKN_KB_ENG_1 ALL 0.0630 0.0490 0.0551 LDC-MAX-ALL-Macro ICTCAS_OKN_KB_ENG_1 0 0.0488 0.0471 0.0386 LDC-MAX-ALL-Macro ICTCAS_OKN_KB_ENG_1 1 0.0000 0.0000 0.0000 LDC-MAX-ALL-Macro ICTCAS_OKN_KB_ENG_1 ALL 0.0321 0.0310 0.0254 LDC-MEAN-ALL-Macro ICTCAS_OKN_KB_ENG_1 0 0.0433 0.0422 0.0350 LDC-MEAN-ALL-Macro ICTCAS_OKN_KB_ENG_1 1 0.0027 0.0027 0.0027 LDC-MEAN-ALL-Macro ICTCAS_OKN_KB_ENG_1 ALL 0.0294 0.0287 0.0239 *ALL-Macro Prec, Recall and F1 refer to mean-precision, mean-recall and mean-F1. --------------------- Composite KB Evaluation (SF-SLOTS only): Scores based on confidence values and Average Precision (AP); k=3 justifications allowed: Metric RunID Hop AP SF-ALL-Macro ICTCAS_OKN_KB_ENG_1 0 0.0420 SF-ALL-Macro ICTCAS_OKN_KB_ENG_1 1 0.0004 SF-ALL-Macro ICTCAS_OKN_KB_ENG_1 ALL 0.0299 LDC-MEAN-ALL-Macro ICTCAS_OKN_KB_ENG_1 0 0.0472 LDC-MEAN-ALL-Macro ICTCAS_OKN_KB_ENG_1 1 0.0005 LDC-MEAN-ALL-Macro ICTCAS_OKN_KB_ENG_1 ALL 0.0343 (PRIMARY METRIC) *ALL-Macro AP refer to mean of corresponding AP values. Scores based on confidence values and Average Precision (AP); only k=1 justification allowed: Metric RunID Hop AP SF-ALL-Macro ICTCAS_OKN_KB_ENG_1 0 0.0718 SF-ALL-Macro ICTCAS_OKN_KB_ENG_1 1 0.0004 SF-ALL-Macro ICTCAS_OKN_KB_ENG_1 ALL 0.0538 LDC-MEAN-ALL-Macro ICTCAS_OKN_KB_ENG_1 0 0.0741 LDC-MEAN-ALL-Macro ICTCAS_OKN_KB_ENG_1 1 0.0005 LDC-MEAN-ALL-Macro ICTCAS_OKN_KB_ENG_1 ALL 0.0570 *ALL-Macro AP refer to mean of corresponding AP values. Scores based on P/R/F1; only k=1 justification allowed: Metric RunID Hop Prec Recall F1 SF-ALL-Micro ICTCAS_OKN_KB_ENG_1 0 0.2972 0.0829 0.1296 SF-ALL-Micro ICTCAS_OKN_KB_ENG_1 1 0.0244 0.0027 0.0049 SF-ALL-Micro ICTCAS_OKN_KB_ENG_1 ALL 0.2665 0.0636 0.1028 SF-ALL-Macro ICTCAS_OKN_KB_ENG_1 0 0.0830 0.0876 0.0708 SF-ALL-Macro ICTCAS_OKN_KB_ENG_1 1 0.0038 0.0038 0.0038 SF-ALL-Macro ICTCAS_OKN_KB_ENG_1 ALL 0.0516 0.0544 0.0443 LDC-MAX-ALL-Micro ICTCAS_OKN_KB_ENG_1 0 0.3550 0.1145 0.1732 LDC-MAX-ALL-Micro ICTCAS_OKN_KB_ENG_1 1 0.0000 0.0000 0.0000 LDC-MAX-ALL-Micro ICTCAS_OKN_KB_ENG_1 ALL 0.3061 0.0880 0.1367 LDC-MAX-ALL-Macro ICTCAS_OKN_KB_ENG_1 0 0.0971 0.0945 0.0805 LDC-MAX-ALL-Macro ICTCAS_OKN_KB_ENG_1 1 0.0000 0.0000 0.0000 LDC-MAX-ALL-Macro ICTCAS_OKN_KB_ENG_1 ALL 0.0594 0.0578 0.0493 LDC-MEAN-ALL-Macro ICTCAS_OKN_KB_ENG_1 0 0.0859 0.0870 0.0734 LDC-MEAN-ALL-Macro ICTCAS_OKN_KB_ENG_1 1 0.0047 0.0047 0.0047 LDC-MEAN-ALL-Macro ICTCAS_OKN_KB_ENG_1 ALL 0.0544 0.0551 0.0467 *ALL-Macro Prec, Recall and F1 refer to mean-precision, mean-recall and mean-F1. --------------------- Composite KB Evaluation (EVENT-SLOTS only): Scores based on confidence values and Average Precision (AP); k=3 justifications allowed: SUMMARY: This section provides summary of AP scores Metric RunID Hop AP SF-ALL-Macro ICTCAS_OKN_KB_ENG_1 0 0.0000 SF-ALL-Macro ICTCAS_OKN_KB_ENG_1 ALL 0.0000 LDC-MEAN-ALL-Macro ICTCAS_OKN_KB_ENG_1 0 0.0000 LDC-MEAN-ALL-Macro ICTCAS_OKN_KB_ENG_1 ALL 0.0000 (PRIMARY METRIC) *ALL-Macro AP refer to mean of corresponding AP values. Scores based on confidence values and Average Precision (AP); only k=1 justification allowed: SUMMARY: This section provides summary of AP scores Metric RunID Hop AP SF-ALL-Macro ICTCAS_OKN_KB_ENG_1 0 0.0000 SF-ALL-Macro ICTCAS_OKN_KB_ENG_1 ALL 0.0000 LDC-MEAN-ALL-Macro ICTCAS_OKN_KB_ENG_1 0 0.0000 LDC-MEAN-ALL-Macro ICTCAS_OKN_KB_ENG_1 ALL 0.0000 *ALL-Macro AP refer to mean of corresponding AP values. Scores based on P/R/F1; only k=1 justification allowed: Metric RunID Hop Prec Recall F1 SF-ALL-Micro ICTCAS_OKN_KB_ENG_1 0 0.0000 0.0000 0.0000 SF-ALL-Micro ICTCAS_OKN_KB_ENG_1 ALL 0.0000 0.0000 0.0000 SF-ALL-Macro ICTCAS_OKN_KB_ENG_1 0 0.0000 0.0000 0.0000 SF-ALL-Macro ICTCAS_OKN_KB_ENG_1 ALL 0.0000 0.0000 0.0000 LDC-MAX-ALL-Micro ICTCAS_OKN_KB_ENG_1 0 0.0000 0.0000 0.0000 LDC-MAX-ALL-Micro ICTCAS_OKN_KB_ENG_1 ALL 0.0000 0.0000 0.0000 LDC-MAX-ALL-Macro ICTCAS_OKN_KB_ENG_1 0 0.0000 0.0000 0.0000 LDC-MAX-ALL-Macro ICTCAS_OKN_KB_ENG_1 ALL 0.0000 0.0000 0.0000 LDC-MEAN-ALL-Macro ICTCAS_OKN_KB_ENG_1 0 0.0000 0.0000 0.0000 LDC-MEAN-ALL-Macro ICTCAS_OKN_KB_ENG_1 ALL 0.0000 0.0000 0.0000 *ALL-Macro Prec, Recall and F1 refer to mean-precision, mean-recall and mean-F1. --------------------- Composite KB Evaluation (SENTIMENT-SLOTS only): Scores based on confidence values and Average Precision (AP); k=3 justifications allowed: Metric RunID Hop AP SF-ALL-Macro ICTCAS_OKN_KB_ENG_1 0 0.0026 SF-ALL-Macro ICTCAS_OKN_KB_ENG_1 1 0.0000 SF-ALL-Macro ICTCAS_OKN_KB_ENG_1 ALL 0.0023 LDC-MEAN-ALL-Macro ICTCAS_OKN_KB_ENG_1 0 0.0032 LDC-MEAN-ALL-Macro ICTCAS_OKN_KB_ENG_1 1 0.0000 LDC-MEAN-ALL-Macro ICTCAS_OKN_KB_ENG_1 ALL 0.0028 (PRIMARY METRIC) *ALL-Macro AP refer to mean of corresponding AP values. Scores based on confidence values and Average Precision (AP); only k=1 justification allowed: Metric RunID Hop AP SF-ALL-Macro ICTCAS_OKN_KB_ENG_1 0 0.0026 SF-ALL-Macro ICTCAS_OKN_KB_ENG_1 1 0.0000 SF-ALL-Macro ICTCAS_OKN_KB_ENG_1 ALL 0.0023 LDC-MEAN-ALL-Macro ICTCAS_OKN_KB_ENG_1 0 0.0032 LDC-MEAN-ALL-Macro ICTCAS_OKN_KB_ENG_1 1 0.0000 LDC-MEAN-ALL-Macro ICTCAS_OKN_KB_ENG_1 ALL 0.0028 *ALL-Macro AP refer to mean of corresponding AP values. Scores based on P/R/F1; only k=1 justification allowed: Metric RunID Hop Prec Recall F1 SF-ALL-Micro ICTCAS_OKN_KB_ENG_1 0 0.0316 0.0189 0.0236 SF-ALL-Micro ICTCAS_OKN_KB_ENG_1 1 0.0000 0.0000 0.0000 SF-ALL-Micro ICTCAS_OKN_KB_ENG_1 ALL 0.0039 0.0151 0.0062 SF-ALL-Macro ICTCAS_OKN_KB_ENG_1 0 0.0162 0.0082 0.0050 SF-ALL-Macro ICTCAS_OKN_KB_ENG_1 1 0.0000 0.0000 0.0000 SF-ALL-Macro ICTCAS_OKN_KB_ENG_1 ALL 0.0115 0.0058 0.0035 LDC-MAX-ALL-Micro ICTCAS_OKN_KB_ENG_1 0 0.0288 0.0268 0.0278 LDC-MAX-ALL-Micro ICTCAS_OKN_KB_ENG_1 1 0.0000 0.0000 0.0000 LDC-MAX-ALL-Micro ICTCAS_OKN_KB_ENG_1 ALL 0.0037 0.0211 0.0064 LDC-MAX-ALL-Macro ICTCAS_OKN_KB_ENG_1 0 0.0210 0.0183 0.0071 LDC-MAX-ALL-Macro ICTCAS_OKN_KB_ENG_1 1 0.0000 0.0000 0.0000 LDC-MAX-ALL-Macro ICTCAS_OKN_KB_ENG_1 ALL 0.0147 0.0129 0.0050 LDC-MEAN-ALL-Macro ICTCAS_OKN_KB_ENG_1 0 0.0193 0.0102 0.0053 LDC-MEAN-ALL-Macro ICTCAS_OKN_KB_ENG_1 1 0.0000 0.0000 0.0000 LDC-MEAN-ALL-Macro ICTCAS_OKN_KB_ENG_1 ALL 0.0135 0.0072 0.0037 *ALL-Macro Prec, Recall and F1 refer to mean-precision, mean-recall and mean-F1. ************************************************************* Run ID: ICTCAS_OKN_KB_ENG_2 Did the run access the live Web during the evaluation window: No Did the run extract relations from the Cold Start source corpus: Yes Did the run generate meaningful confidence values: Yes Did the run attempt the full Entity Discovery and Linking task: No Did the run attempt the full Event Nugget Detection and Coreference task: Yes Did the run include SF relations: Yes Did the run attempt the Event Argument Extraction and Linking task: No Did the run include Sentiment relations: No --------------------- Composite KB Evaluation (All slot types): Scores based on confidence values and Average Precision (AP); k=3 justifications allowed: Metric RunID Hop AP SF-ALL-Macro ICTCAS_OKN_KB_ENG_2 0 0.0170 SF-ALL-Macro ICTCAS_OKN_KB_ENG_2 1 0.0000 SF-ALL-Macro ICTCAS_OKN_KB_ENG_2 ALL 0.0130 LDC-MEAN-ALL-Macro ICTCAS_OKN_KB_ENG_2 0 0.0203 LDC-MEAN-ALL-Macro ICTCAS_OKN_KB_ENG_2 1 0.0000 LDC-MEAN-ALL-Macro ICTCAS_OKN_KB_ENG_2 ALL 0.0155 (PRIMARY METRIC) *ALL-Macro AP refer to mean of corresponding AP values. Scores based on confidence values and Average Precision (AP); only k=1 justification allowed: Metric RunID Hop AP SF-ALL-Macro ICTCAS_OKN_KB_ENG_2 0 0.0262 SF-ALL-Macro ICTCAS_OKN_KB_ENG_2 1 0.0000 SF-ALL-Macro ICTCAS_OKN_KB_ENG_2 ALL 0.0206 LDC-MEAN-ALL-Macro ICTCAS_OKN_KB_ENG_2 0 0.0304 LDC-MEAN-ALL-Macro ICTCAS_OKN_KB_ENG_2 1 0.0000 LDC-MEAN-ALL-Macro ICTCAS_OKN_KB_ENG_2 ALL 0.0242 *ALL-Macro AP refer to mean of corresponding AP values. Scores based on P/R/F1; only k=1 justification allowed: Metric RunID Hop Prec Recall F1 SF-ALL-Micro ICTCAS_OKN_KB_ENG_2 0 0.3485 0.0359 0.0652 SF-ALL-Micro ICTCAS_OKN_KB_ENG_2 1 0.0000 0.0000 0.0000 SF-ALL-Micro ICTCAS_OKN_KB_ENG_2 ALL 0.3000 0.0278 0.0508 SF-ALL-Macro ICTCAS_OKN_KB_ENG_2 0 0.0345 0.0330 0.0275 SF-ALL-Macro ICTCAS_OKN_KB_ENG_2 1 0.0000 0.0000 0.0000 SF-ALL-Macro ICTCAS_OKN_KB_ENG_2 ALL 0.0226 0.0217 0.0181 LDC-MAX-ALL-Micro ICTCAS_OKN_KB_ENG_2 0 0.4603 0.0575 0.1022 LDC-MAX-ALL-Micro ICTCAS_OKN_KB_ENG_2 1 0.0000 0.0000 0.0000 LDC-MAX-ALL-Micro ICTCAS_OKN_KB_ENG_2 ALL 0.3816 0.0451 0.0806 LDC-MAX-ALL-Macro ICTCAS_OKN_KB_ENG_2 0 0.0507 0.0439 0.0394 LDC-MAX-ALL-Macro ICTCAS_OKN_KB_ENG_2 1 0.0000 0.0000 0.0000 LDC-MAX-ALL-Macro ICTCAS_OKN_KB_ENG_2 ALL 0.0334 0.0289 0.0259 LDC-MEAN-ALL-Macro ICTCAS_OKN_KB_ENG_2 0 0.0394 0.0364 0.0317 LDC-MEAN-ALL-Macro ICTCAS_OKN_KB_ENG_2 1 0.0000 0.0000 0.0000 LDC-MEAN-ALL-Macro ICTCAS_OKN_KB_ENG_2 ALL 0.0260 0.0240 0.0208 *ALL-Macro Prec, Recall and F1 refer to mean-precision, mean-recall and mean-F1. --------------------- Composite KB Evaluation (SF-SLOTS only): Scores based on confidence values and Average Precision (AP); k=3 justifications allowed: Metric RunID Hop AP SF-ALL-Macro ICTCAS_OKN_KB_ENG_2 0 0.0392 SF-ALL-Macro ICTCAS_OKN_KB_ENG_2 1 0.0000 SF-ALL-Macro ICTCAS_OKN_KB_ENG_2 ALL 0.0299 LDC-MEAN-ALL-Macro ICTCAS_OKN_KB_ENG_2 0 0.0437 LDC-MEAN-ALL-Macro ICTCAS_OKN_KB_ENG_2 1 0.0000 LDC-MEAN-ALL-Macro ICTCAS_OKN_KB_ENG_2 ALL 0.0335 (PRIMARY METRIC) *ALL-Macro AP refer to mean of corresponding AP values. Scores based on confidence values and Average Precision (AP); only k=1 justification allowed: Metric RunID Hop AP SF-ALL-Macro ICTCAS_OKN_KB_ENG_2 0 0.0606 SF-ALL-Macro ICTCAS_OKN_KB_ENG_2 1 0.0000 SF-ALL-Macro ICTCAS_OKN_KB_ENG_2 ALL 0.0476 LDC-MEAN-ALL-Macro ICTCAS_OKN_KB_ENG_2 0 0.0656 LDC-MEAN-ALL-Macro ICTCAS_OKN_KB_ENG_2 1 0.0000 LDC-MEAN-ALL-Macro ICTCAS_OKN_KB_ENG_2 ALL 0.0521 *ALL-Macro AP refer to mean of corresponding AP values. Scores based on P/R/F1; only k=1 justification allowed: Metric RunID Hop Prec Recall F1 SF-ALL-Micro ICTCAS_OKN_KB_ENG_2 0 0.3485 0.0725 0.1201 SF-ALL-Micro ICTCAS_OKN_KB_ENG_2 1 0.0000 0.0000 0.0000 SF-ALL-Micro ICTCAS_OKN_KB_ENG_2 ALL 0.3000 0.0551 0.0931 SF-ALL-Macro ICTCAS_OKN_KB_ENG_2 0 0.0796 0.0763 0.0636 SF-ALL-Macro ICTCAS_OKN_KB_ENG_2 1 0.0000 0.0000 0.0000 SF-ALL-Macro ICTCAS_OKN_KB_ENG_2 ALL 0.0480 0.0460 0.0384 LDC-MAX-ALL-Micro ICTCAS_OKN_KB_ENG_2 0 0.4603 0.1107 0.1785 LDC-MAX-ALL-Micro ICTCAS_OKN_KB_ENG_2 1 0.0000 0.0000 0.0000 LDC-MAX-ALL-Micro ICTCAS_OKN_KB_ENG_2 ALL 0.3816 0.0850 0.1391 LDC-MAX-ALL-Macro ICTCAS_OKN_KB_ENG_2 0 0.1093 0.0946 0.0849 LDC-MAX-ALL-Macro ICTCAS_OKN_KB_ENG_2 1 0.0000 0.0000 0.0000 LDC-MAX-ALL-Macro ICTCAS_OKN_KB_ENG_2 ALL 0.0669 0.0579 0.0520 LDC-MEAN-ALL-Macro ICTCAS_OKN_KB_ENG_2 0 0.0850 0.0785 0.0682 LDC-MEAN-ALL-Macro ICTCAS_OKN_KB_ENG_2 1 0.0000 0.0000 0.0000 LDC-MEAN-ALL-Macro ICTCAS_OKN_KB_ENG_2 ALL 0.0520 0.0480 0.0417 *ALL-Macro Prec, Recall and F1 refer to mean-precision, mean-recall and mean-F1. --------------------- Composite KB Evaluation (EVENT-SLOTS only): Scores based on confidence values and Average Precision (AP); k=3 justifications allowed: SUMMARY: This section provides summary of AP scores Metric RunID Hop AP SF-ALL-Macro ICTCAS_OKN_KB_ENG_2 0 0.0000 SF-ALL-Macro ICTCAS_OKN_KB_ENG_2 ALL 0.0000 LDC-MEAN-ALL-Macro ICTCAS_OKN_KB_ENG_2 0 0.0000 LDC-MEAN-ALL-Macro ICTCAS_OKN_KB_ENG_2 ALL 0.0000 (PRIMARY METRIC) *ALL-Macro AP refer to mean of corresponding AP values. Scores based on confidence values and Average Precision (AP); only k=1 justification allowed: SUMMARY: This section provides summary of AP scores Metric RunID Hop AP SF-ALL-Macro ICTCAS_OKN_KB_ENG_2 0 0.0000 SF-ALL-Macro ICTCAS_OKN_KB_ENG_2 ALL 0.0000 LDC-MEAN-ALL-Macro ICTCAS_OKN_KB_ENG_2 0 0.0000 LDC-MEAN-ALL-Macro ICTCAS_OKN_KB_ENG_2 ALL 0.0000 *ALL-Macro AP refer to mean of corresponding AP values. Scores based on P/R/F1; only k=1 justification allowed: Metric RunID Hop Prec Recall F1 SF-ALL-Micro ICTCAS_OKN_KB_ENG_2 0 0.0000 0.0000 0.0000 SF-ALL-Micro ICTCAS_OKN_KB_ENG_2 ALL 0.0000 0.0000 0.0000 SF-ALL-Macro ICTCAS_OKN_KB_ENG_2 0 0.0000 0.0000 0.0000 SF-ALL-Macro ICTCAS_OKN_KB_ENG_2 ALL 0.0000 0.0000 0.0000 LDC-MAX-ALL-Micro ICTCAS_OKN_KB_ENG_2 0 0.0000 0.0000 0.0000 LDC-MAX-ALL-Micro ICTCAS_OKN_KB_ENG_2 ALL 0.0000 0.0000 0.0000 LDC-MAX-ALL-Macro ICTCAS_OKN_KB_ENG_2 0 0.0000 0.0000 0.0000 LDC-MAX-ALL-Macro ICTCAS_OKN_KB_ENG_2 ALL 0.0000 0.0000 0.0000 LDC-MEAN-ALL-Macro ICTCAS_OKN_KB_ENG_2 0 0.0000 0.0000 0.0000 LDC-MEAN-ALL-Macro ICTCAS_OKN_KB_ENG_2 ALL 0.0000 0.0000 0.0000 *ALL-Macro Prec, Recall and F1 refer to mean-precision, mean-recall and mean-F1. --------------------- Composite KB Evaluation (SENTIMENT-SLOTS only): Scores based on confidence values and Average Precision (AP); k=3 justifications allowed: Metric RunID Hop AP SF-ALL-Macro ICTCAS_OKN_KB_ENG_2 0 0.0000 SF-ALL-Macro ICTCAS_OKN_KB_ENG_2 1 0.0000 SF-ALL-Macro ICTCAS_OKN_KB_ENG_2 ALL 0.0000 LDC-MEAN-ALL-Macro ICTCAS_OKN_KB_ENG_2 0 0.0000 LDC-MEAN-ALL-Macro ICTCAS_OKN_KB_ENG_2 1 0.0000 LDC-MEAN-ALL-Macro ICTCAS_OKN_KB_ENG_2 ALL 0.0000 (PRIMARY METRIC) *ALL-Macro AP refer to mean of corresponding AP values. Scores based on confidence values and Average Precision (AP); only k=1 justification allowed: Metric RunID Hop AP SF-ALL-Macro ICTCAS_OKN_KB_ENG_2 0 0.0000 SF-ALL-Macro ICTCAS_OKN_KB_ENG_2 1 0.0000 SF-ALL-Macro ICTCAS_OKN_KB_ENG_2 ALL 0.0000 LDC-MEAN-ALL-Macro ICTCAS_OKN_KB_ENG_2 0 0.0000 LDC-MEAN-ALL-Macro ICTCAS_OKN_KB_ENG_2 1 0.0000 LDC-MEAN-ALL-Macro ICTCAS_OKN_KB_ENG_2 ALL 0.0000 *ALL-Macro AP refer to mean of corresponding AP values. Scores based on P/R/F1; only k=1 justification allowed: Metric RunID Hop Prec Recall F1 SF-ALL-Micro ICTCAS_OKN_KB_ENG_2 0 0.0000 0.0000 0.0000 SF-ALL-Micro ICTCAS_OKN_KB_ENG_2 1 0.0000 0.0000 0.0000 SF-ALL-Micro ICTCAS_OKN_KB_ENG_2 ALL 0.0000 0.0000 0.0000 SF-ALL-Macro ICTCAS_OKN_KB_ENG_2 0 0.0000 0.0000 0.0000 SF-ALL-Macro ICTCAS_OKN_KB_ENG_2 1 0.0000 0.0000 0.0000 SF-ALL-Macro ICTCAS_OKN_KB_ENG_2 ALL 0.0000 0.0000 0.0000 LDC-MAX-ALL-Micro ICTCAS_OKN_KB_ENG_2 0 0.0000 0.0000 0.0000 LDC-MAX-ALL-Micro ICTCAS_OKN_KB_ENG_2 1 0.0000 0.0000 0.0000 LDC-MAX-ALL-Micro ICTCAS_OKN_KB_ENG_2 ALL 0.0000 0.0000 0.0000 LDC-MAX-ALL-Macro ICTCAS_OKN_KB_ENG_2 0 0.0000 0.0000 0.0000 LDC-MAX-ALL-Macro ICTCAS_OKN_KB_ENG_2 1 0.0000 0.0000 0.0000 LDC-MAX-ALL-Macro ICTCAS_OKN_KB_ENG_2 ALL 0.0000 0.0000 0.0000 LDC-MEAN-ALL-Macro ICTCAS_OKN_KB_ENG_2 0 0.0000 0.0000 0.0000 LDC-MEAN-ALL-Macro ICTCAS_OKN_KB_ENG_2 1 0.0000 0.0000 0.0000 LDC-MEAN-ALL-Macro ICTCAS_OKN_KB_ENG_2 ALL 0.0000 0.0000 0.0000 *ALL-Macro Prec, Recall and F1 refer to mean-precision, mean-recall and mean-F1. ************************************************************* Run ID: ICTCAS_OKN_KB_ENG_3 Did the run access the live Web during the evaluation window: No Did the run extract relations from the Cold Start source corpus: Yes Did the run generate meaningful confidence values: Yes Did the run attempt the full Entity Discovery and Linking task: No Did the run attempt the full Event Nugget Detection and Coreference task: Yes Did the run include SF relations: Yes Did the run attempt the Event Argument Extraction and Linking task: No Did the run include Sentiment relations: Yes --------------------- Composite KB Evaluation (All slot types): Scores based on confidence values and Average Precision (AP); k=3 justifications allowed: Metric RunID Hop AP SF-ALL-Macro ICTCAS_OKN_KB_ENG_3 0 0.0233 SF-ALL-Macro ICTCAS_OKN_KB_ENG_3 1 0.0001 SF-ALL-Macro ICTCAS_OKN_KB_ENG_3 ALL 0.0167 LDC-MEAN-ALL-Macro ICTCAS_OKN_KB_ENG_3 0 0.0294 LDC-MEAN-ALL-Macro ICTCAS_OKN_KB_ENG_3 1 0.0002 LDC-MEAN-ALL-Macro ICTCAS_OKN_KB_ENG_3 ALL 0.0221 (PRIMARY METRIC) *ALL-Macro AP refer to mean of corresponding AP values. Scores based on confidence values and Average Precision (AP); only k=1 justification allowed: Metric RunID Hop AP SF-ALL-Macro ICTCAS_OKN_KB_ENG_3 0 0.0323 SF-ALL-Macro ICTCAS_OKN_KB_ENG_3 1 0.0001 SF-ALL-Macro ICTCAS_OKN_KB_ENG_3 ALL 0.0260 LDC-MEAN-ALL-Macro ICTCAS_OKN_KB_ENG_3 0 0.0384 LDC-MEAN-ALL-Macro ICTCAS_OKN_KB_ENG_3 1 0.0002 LDC-MEAN-ALL-Macro ICTCAS_OKN_KB_ENG_3 ALL 0.0317 *ALL-Macro AP refer to mean of corresponding AP values. Scores based on P/R/F1; only k=1 justification allowed: Metric RunID Hop Prec Recall F1 SF-ALL-Micro ICTCAS_OKN_KB_ENG_3 0 0.2363 0.0424 0.0718 SF-ALL-Micro ICTCAS_OKN_KB_ENG_3 1 0.0009 0.0015 0.0011 SF-ALL-Micro ICTCAS_OKN_KB_ENG_3 ALL 0.0638 0.0331 0.0435 SF-ALL-Macro ICTCAS_OKN_KB_ENG_3 0 0.0437 0.0387 0.0346 SF-ALL-Macro ICTCAS_OKN_KB_ENG_3 1 0.0021 0.0021 0.0021 SF-ALL-Macro ICTCAS_OKN_KB_ENG_3 ALL 0.0294 0.0261 0.0234 LDC-MAX-ALL-Micro ICTCAS_OKN_KB_ENG_3 0 0.2878 0.0585 0.0972 LDC-MAX-ALL-Micro ICTCAS_OKN_KB_ENG_3 1 0.0018 0.0036 0.0024 LDC-MAX-ALL-Micro ICTCAS_OKN_KB_ENG_3 ALL 0.0775 0.0466 0.0582 LDC-MAX-ALL-Macro ICTCAS_OKN_KB_ENG_3 0 0.0582 0.0485 0.0456 LDC-MAX-ALL-Macro ICTCAS_OKN_KB_ENG_3 1 0.0053 0.0053 0.0053 LDC-MAX-ALL-Macro ICTCAS_OKN_KB_ENG_3 ALL 0.0401 0.0337 0.0318 LDC-MEAN-ALL-Macro ICTCAS_OKN_KB_ENG_3 0 0.0517 0.0444 0.0411 LDC-MEAN-ALL-Macro ICTCAS_OKN_KB_ENG_3 1 0.0027 0.0027 0.0027 LDC-MEAN-ALL-Macro ICTCAS_OKN_KB_ENG_3 ALL 0.0349 0.0301 0.0280 *ALL-Macro Prec, Recall and F1 refer to mean-precision, mean-recall and mean-F1. --------------------- Composite KB Evaluation (SF-SLOTS only): Scores based on confidence values and Average Precision (AP); k=3 justifications allowed: Metric RunID Hop AP SF-ALL-Macro ICTCAS_OKN_KB_ENG_3 0 0.0489 SF-ALL-Macro ICTCAS_OKN_KB_ENG_3 1 0.0004 SF-ALL-Macro ICTCAS_OKN_KB_ENG_3 ALL 0.0341 LDC-MEAN-ALL-Macro ICTCAS_OKN_KB_ENG_3 0 0.0561 LDC-MEAN-ALL-Macro ICTCAS_OKN_KB_ENG_3 1 0.0005 LDC-MEAN-ALL-Macro ICTCAS_OKN_KB_ENG_3 ALL 0.0409 (PRIMARY METRIC) *ALL-Macro AP refer to mean of corresponding AP values. Scores based on confidence values and Average Precision (AP); only k=1 justification allowed: Metric RunID Hop AP SF-ALL-Macro ICTCAS_OKN_KB_ENG_3 0 0.0708 SF-ALL-Macro ICTCAS_OKN_KB_ENG_3 1 0.0004 SF-ALL-Macro ICTCAS_OKN_KB_ENG_3 ALL 0.0566 LDC-MEAN-ALL-Macro ICTCAS_OKN_KB_ENG_3 0 0.0786 LDC-MEAN-ALL-Macro ICTCAS_OKN_KB_ENG_3 1 0.0005 LDC-MEAN-ALL-Macro ICTCAS_OKN_KB_ENG_3 ALL 0.0644 *ALL-Macro AP refer to mean of corresponding AP values. Scores based on P/R/F1; only k=1 justification allowed: Metric RunID Hop Prec Recall F1 SF-ALL-Micro ICTCAS_OKN_KB_ENG_3 0 0.3889 0.0786 0.1307 SF-ALL-Micro ICTCAS_OKN_KB_ENG_3 1 0.0294 0.0027 0.0050 SF-ALL-Micro ICTCAS_OKN_KB_ENG_3 ALL 0.3433 0.0604 0.1027 SF-ALL-Macro ICTCAS_OKN_KB_ENG_3 0 0.0938 0.0832 0.0737 SF-ALL-Macro ICTCAS_OKN_KB_ENG_3 1 0.0038 0.0038 0.0038 SF-ALL-Macro ICTCAS_OKN_KB_ENG_3 ALL 0.0581 0.0517 0.0460 LDC-MAX-ALL-Micro ICTCAS_OKN_KB_ENG_3 0 0.4583 0.1050 0.1708 LDC-MAX-ALL-Micro ICTCAS_OKN_KB_ENG_3 1 0.0909 0.0063 0.0118 LDC-MAX-ALL-Micro ICTCAS_OKN_KB_ENG_3 ALL 0.4275 0.0821 0.1378 LDC-MAX-ALL-Macro ICTCAS_OKN_KB_ENG_3 0 0.1148 0.0945 0.0884 LDC-MAX-ALL-Macro ICTCAS_OKN_KB_ENG_3 1 0.0094 0.0094 0.0094 LDC-MAX-ALL-Macro ICTCAS_OKN_KB_ENG_3 ALL 0.0739 0.0615 0.0577 LDC-MEAN-ALL-Macro ICTCAS_OKN_KB_ENG_3 0 0.1041 0.0888 0.0820 LDC-MEAN-ALL-Macro ICTCAS_OKN_KB_ENG_3 1 0.0047 0.0047 0.0047 LDC-MEAN-ALL-Macro ICTCAS_OKN_KB_ENG_3 ALL 0.0655 0.0561 0.0520 *ALL-Macro Prec, Recall and F1 refer to mean-precision, mean-recall and mean-F1. --------------------- Composite KB Evaluation (EVENT-SLOTS only): Scores based on confidence values and Average Precision (AP); k=3 justifications allowed: SUMMARY: This section provides summary of AP scores Metric RunID Hop AP SF-ALL-Macro ICTCAS_OKN_KB_ENG_3 0 0.0000 SF-ALL-Macro ICTCAS_OKN_KB_ENG_3 ALL 0.0000 LDC-MEAN-ALL-Macro ICTCAS_OKN_KB_ENG_3 0 0.0000 LDC-MEAN-ALL-Macro ICTCAS_OKN_KB_ENG_3 ALL 0.0000 (PRIMARY METRIC) *ALL-Macro AP refer to mean of corresponding AP values. Scores based on confidence values and Average Precision (AP); only k=1 justification allowed: SUMMARY: This section provides summary of AP scores Metric RunID Hop AP SF-ALL-Macro ICTCAS_OKN_KB_ENG_3 0 0.0000 SF-ALL-Macro ICTCAS_OKN_KB_ENG_3 ALL 0.0000 LDC-MEAN-ALL-Macro ICTCAS_OKN_KB_ENG_3 0 0.0000 LDC-MEAN-ALL-Macro ICTCAS_OKN_KB_ENG_3 ALL 0.0000 *ALL-Macro AP refer to mean of corresponding AP values. Scores based on P/R/F1; only k=1 justification allowed: Metric RunID Hop Prec Recall F1 SF-ALL-Micro ICTCAS_OKN_KB_ENG_3 0 0.0000 0.0000 0.0000 SF-ALL-Micro ICTCAS_OKN_KB_ENG_3 ALL 0.0000 0.0000 0.0000 SF-ALL-Macro ICTCAS_OKN_KB_ENG_3 0 0.0000 0.0000 0.0000 SF-ALL-Macro ICTCAS_OKN_KB_ENG_3 ALL 0.0000 0.0000 0.0000 LDC-MAX-ALL-Micro ICTCAS_OKN_KB_ENG_3 0 0.0000 0.0000 0.0000 LDC-MAX-ALL-Micro ICTCAS_OKN_KB_ENG_3 ALL 0.0000 0.0000 0.0000 LDC-MAX-ALL-Macro ICTCAS_OKN_KB_ENG_3 0 0.0000 0.0000 0.0000 LDC-MAX-ALL-Macro ICTCAS_OKN_KB_ENG_3 ALL 0.0000 0.0000 0.0000 LDC-MEAN-ALL-Macro ICTCAS_OKN_KB_ENG_3 0 0.0000 0.0000 0.0000 LDC-MEAN-ALL-Macro ICTCAS_OKN_KB_ENG_3 ALL 0.0000 0.0000 0.0000 *ALL-Macro Prec, Recall and F1 refer to mean-precision, mean-recall and mean-F1. --------------------- Composite KB Evaluation (SENTIMENT-SLOTS only): Scores based on confidence values and Average Precision (AP); k=3 justifications allowed: Metric RunID Hop AP SF-ALL-Macro ICTCAS_OKN_KB_ENG_3 0 0.0112 SF-ALL-Macro ICTCAS_OKN_KB_ENG_3 1 0.0000 SF-ALL-Macro ICTCAS_OKN_KB_ENG_3 ALL 0.0104 LDC-MEAN-ALL-Macro ICTCAS_OKN_KB_ENG_3 0 0.0189 LDC-MEAN-ALL-Macro ICTCAS_OKN_KB_ENG_3 1 0.0000 LDC-MEAN-ALL-Macro ICTCAS_OKN_KB_ENG_3 ALL 0.0179 (PRIMARY METRIC) *ALL-Macro AP refer to mean of corresponding AP values. Scores based on confidence values and Average Precision (AP); only k=1 justification allowed: Metric RunID Hop AP SF-ALL-Macro ICTCAS_OKN_KB_ENG_3 0 0.0083 SF-ALL-Macro ICTCAS_OKN_KB_ENG_3 1 0.0000 SF-ALL-Macro ICTCAS_OKN_KB_ENG_3 ALL 0.0076 LDC-MEAN-ALL-Macro ICTCAS_OKN_KB_ENG_3 0 0.0111 LDC-MEAN-ALL-Macro ICTCAS_OKN_KB_ENG_3 1 0.0000 LDC-MEAN-ALL-Macro ICTCAS_OKN_KB_ENG_3 ALL 0.0101 *ALL-Macro AP refer to mean of corresponding AP values. Scores based on P/R/F1; only k=1 justification allowed: Metric RunID Hop Prec Recall F1 SF-ALL-Micro ICTCAS_OKN_KB_ENG_3 0 0.0432 0.0252 0.0318 SF-ALL-Micro ICTCAS_OKN_KB_ENG_3 1 0.0000 0.0000 0.0000 SF-ALL-Micro ICTCAS_OKN_KB_ENG_3 ALL 0.0064 0.0201 0.0097 SF-ALL-Macro ICTCAS_OKN_KB_ENG_3 0 0.0161 0.0139 0.0139 SF-ALL-Macro ICTCAS_OKN_KB_ENG_3 1 0.0000 0.0000 0.0000 SF-ALL-Macro ICTCAS_OKN_KB_ENG_3 ALL 0.0114 0.0098 0.0098 LDC-MAX-ALL-Micro ICTCAS_OKN_KB_ENG_3 0 0.0460 0.0357 0.0402 LDC-MAX-ALL-Micro ICTCAS_OKN_KB_ENG_3 1 0.0000 0.0000 0.0000 LDC-MAX-ALL-Micro ICTCAS_OKN_KB_ENG_3 ALL 0.0064 0.0282 0.0104 LDC-MAX-ALL-Macro ICTCAS_OKN_KB_ENG_3 0 0.0279 0.0262 0.0258 LDC-MAX-ALL-Macro ICTCAS_OKN_KB_ENG_3 1 0.0000 0.0000 0.0000 LDC-MAX-ALL-Macro ICTCAS_OKN_KB_ENG_3 ALL 0.0196 0.0184 0.0181 LDC-MEAN-ALL-Macro ICTCAS_OKN_KB_ENG_3 0 0.0191 0.0180 0.0175 LDC-MEAN-ALL-Macro ICTCAS_OKN_KB_ENG_3 1 0.0000 0.0000 0.0000 LDC-MEAN-ALL-Macro ICTCAS_OKN_KB_ENG_3 ALL 0.0134 0.0127 0.0123 *ALL-Macro Prec, Recall and F1 refer to mean-precision, mean-recall and mean-F1. ************************************************************* Run ID: ICTCAS_OKN_KB_ENG_4 Did the run access the live Web during the evaluation window: No Did the run extract relations from the Cold Start source corpus: Yes Did the run generate meaningful confidence values: Yes Did the run attempt the full Entity Discovery and Linking task: No Did the run attempt the full Event Nugget Detection and Coreference task: No Did the run include SF relations: Yes Did the run attempt the Event Argument Extraction and Linking task: No Did the run include Sentiment relations: No --------------------- Composite KB Evaluation (All slot types): Scores based on confidence values and Average Precision (AP); k=3 justifications allowed: Metric RunID Hop AP SF-ALL-Macro ICTCAS_OKN_KB_ENG_4 0 0.0056 SF-ALL-Macro ICTCAS_OKN_KB_ENG_4 1 0.0000 SF-ALL-Macro ICTCAS_OKN_KB_ENG_4 ALL 0.0042 LDC-MEAN-ALL-Macro ICTCAS_OKN_KB_ENG_4 0 0.0049 LDC-MEAN-ALL-Macro ICTCAS_OKN_KB_ENG_4 1 0.0000 LDC-MEAN-ALL-Macro ICTCAS_OKN_KB_ENG_4 ALL 0.0038 (PRIMARY METRIC) *ALL-Macro AP refer to mean of corresponding AP values. Scores based on confidence values and Average Precision (AP); only k=1 justification allowed: Metric RunID Hop AP SF-ALL-Macro ICTCAS_OKN_KB_ENG_4 0 0.0142 SF-ALL-Macro ICTCAS_OKN_KB_ENG_4 1 0.0000 SF-ALL-Macro ICTCAS_OKN_KB_ENG_4 ALL 0.0103 LDC-MEAN-ALL-Macro ICTCAS_OKN_KB_ENG_4 0 0.0126 LDC-MEAN-ALL-Macro ICTCAS_OKN_KB_ENG_4 1 0.0000 LDC-MEAN-ALL-Macro ICTCAS_OKN_KB_ENG_4 ALL 0.0095 *ALL-Macro AP refer to mean of corresponding AP values. Scores based on P/R/F1; only k=1 justification allowed: Metric RunID Hop Prec Recall F1 SF-ALL-Micro ICTCAS_OKN_KB_ENG_4 0 0.2500 0.0086 0.0165 SF-ALL-Micro ICTCAS_OKN_KB_ENG_4 1 0.0000 0.0000 0.0000 SF-ALL-Micro ICTCAS_OKN_KB_ENG_4 ALL 0.2439 0.0066 0.0129 SF-ALL-Macro ICTCAS_OKN_KB_ENG_4 0 0.0155 0.0165 0.0151 SF-ALL-Macro ICTCAS_OKN_KB_ENG_4 1 0.0000 0.0000 0.0000 SF-ALL-Macro ICTCAS_OKN_KB_ENG_4 ALL 0.0101 0.0108 0.0099 LDC-MAX-ALL-Micro ICTCAS_OKN_KB_ENG_4 0 0.2778 0.0099 0.0191 LDC-MAX-ALL-Micro ICTCAS_OKN_KB_ENG_4 1 0.0000 0.0000 0.0000 LDC-MAX-ALL-Micro ICTCAS_OKN_KB_ENG_4 ALL 0.2703 0.0078 0.0151 LDC-MAX-ALL-Macro ICTCAS_OKN_KB_ENG_4 0 0.0168 0.0170 0.0153 LDC-MAX-ALL-Macro ICTCAS_OKN_KB_ENG_4 1 0.0000 0.0000 0.0000 LDC-MAX-ALL-Macro ICTCAS_OKN_KB_ENG_4 ALL 0.0110 0.0112 0.0101 LDC-MEAN-ALL-Macro ICTCAS_OKN_KB_ENG_4 0 0.0142 0.0155 0.0138 LDC-MEAN-ALL-Macro ICTCAS_OKN_KB_ENG_4 1 0.0000 0.0000 0.0000 LDC-MEAN-ALL-Macro ICTCAS_OKN_KB_ENG_4 ALL 0.0094 0.0102 0.0091 *ALL-Macro Prec, Recall and F1 refer to mean-precision, mean-recall and mean-F1. --------------------- Composite KB Evaluation (SF-SLOTS only): Scores based on confidence values and Average Precision (AP); k=3 justifications allowed: Metric RunID Hop AP SF-ALL-Macro ICTCAS_OKN_KB_ENG_4 0 0.0130 SF-ALL-Macro ICTCAS_OKN_KB_ENG_4 1 0.0000 SF-ALL-Macro ICTCAS_OKN_KB_ENG_4 ALL 0.0096 LDC-MEAN-ALL-Macro ICTCAS_OKN_KB_ENG_4 0 0.0106 LDC-MEAN-ALL-Macro ICTCAS_OKN_KB_ENG_4 1 0.0000 LDC-MEAN-ALL-Macro ICTCAS_OKN_KB_ENG_4 ALL 0.0081 (PRIMARY METRIC) *ALL-Macro AP refer to mean of corresponding AP values. Scores based on confidence values and Average Precision (AP); only k=1 justification allowed: Metric RunID Hop AP SF-ALL-Macro ICTCAS_OKN_KB_ENG_4 0 0.0328 SF-ALL-Macro ICTCAS_OKN_KB_ENG_4 1 0.0000 SF-ALL-Macro ICTCAS_OKN_KB_ENG_4 ALL 0.0238 LDC-MEAN-ALL-Macro ICTCAS_OKN_KB_ENG_4 0 0.0272 LDC-MEAN-ALL-Macro ICTCAS_OKN_KB_ENG_4 1 0.0000 LDC-MEAN-ALL-Macro ICTCAS_OKN_KB_ENG_4 ALL 0.0205 *ALL-Macro AP refer to mean of corresponding AP values. Scores based on P/R/F1; only k=1 justification allowed: Metric RunID Hop Prec Recall F1 SF-ALL-Micro ICTCAS_OKN_KB_ENG_4 0 0.2500 0.0173 0.0323 SF-ALL-Micro ICTCAS_OKN_KB_ENG_4 1 0.0000 0.0000 0.0000 SF-ALL-Micro ICTCAS_OKN_KB_ENG_4 ALL 0.2439 0.0131 0.0249 SF-ALL-Macro ICTCAS_OKN_KB_ENG_4 0 0.0357 0.0380 0.0350 SF-ALL-Macro ICTCAS_OKN_KB_ENG_4 1 0.0000 0.0000 0.0000 SF-ALL-Macro ICTCAS_OKN_KB_ENG_4 ALL 0.0215 0.0230 0.0211 LDC-MAX-ALL-Micro ICTCAS_OKN_KB_ENG_4 0 0.2703 0.0191 0.0357 LDC-MAX-ALL-Micro ICTCAS_OKN_KB_ENG_4 1 0.0000 0.0000 0.0000 LDC-MAX-ALL-Micro ICTCAS_OKN_KB_ENG_4 ALL 0.2632 0.0147 0.0278 LDC-MAX-ALL-Macro ICTCAS_OKN_KB_ENG_4 0 0.0361 0.0366 0.0330 LDC-MAX-ALL-Macro ICTCAS_OKN_KB_ENG_4 1 0.0000 0.0000 0.0000 LDC-MAX-ALL-Macro ICTCAS_OKN_KB_ENG_4 ALL 0.0221 0.0224 0.0202 LDC-MEAN-ALL-Macro ICTCAS_OKN_KB_ENG_4 0 0.0306 0.0335 0.0298 LDC-MEAN-ALL-Macro ICTCAS_OKN_KB_ENG_4 1 0.0000 0.0000 0.0000 LDC-MEAN-ALL-Macro ICTCAS_OKN_KB_ENG_4 ALL 0.0187 0.0205 0.0182 *ALL-Macro Prec, Recall and F1 refer to mean-precision, mean-recall and mean-F1. --------------------- Composite KB Evaluation (EVENT-SLOTS only): Scores based on confidence values and Average Precision (AP); k=3 justifications allowed: SUMMARY: This section provides summary of AP scores Metric RunID Hop AP SF-ALL-Macro ICTCAS_OKN_KB_ENG_4 0 0.0000 SF-ALL-Macro ICTCAS_OKN_KB_ENG_4 ALL 0.0000 LDC-MEAN-ALL-Macro ICTCAS_OKN_KB_ENG_4 0 0.0000 LDC-MEAN-ALL-Macro ICTCAS_OKN_KB_ENG_4 ALL 0.0000 (PRIMARY METRIC) *ALL-Macro AP refer to mean of corresponding AP values. Scores based on confidence values and Average Precision (AP); only k=1 justification allowed: SUMMARY: This section provides summary of AP scores Metric RunID Hop AP SF-ALL-Macro ICTCAS_OKN_KB_ENG_4 0 0.0000 SF-ALL-Macro ICTCAS_OKN_KB_ENG_4 ALL 0.0000 LDC-MEAN-ALL-Macro ICTCAS_OKN_KB_ENG_4 0 0.0000 LDC-MEAN-ALL-Macro ICTCAS_OKN_KB_ENG_4 ALL 0.0000 *ALL-Macro AP refer to mean of corresponding AP values. Scores based on P/R/F1; only k=1 justification allowed: Metric RunID Hop Prec Recall F1 SF-ALL-Micro ICTCAS_OKN_KB_ENG_4 0 0.0000 0.0000 0.0000 SF-ALL-Micro ICTCAS_OKN_KB_ENG_4 ALL 0.0000 0.0000 0.0000 SF-ALL-Macro ICTCAS_OKN_KB_ENG_4 0 0.0000 0.0000 0.0000 SF-ALL-Macro ICTCAS_OKN_KB_ENG_4 ALL 0.0000 0.0000 0.0000 LDC-MAX-ALL-Micro ICTCAS_OKN_KB_ENG_4 0 0.0000 0.0000 0.0000 LDC-MAX-ALL-Micro ICTCAS_OKN_KB_ENG_4 ALL 0.0000 0.0000 0.0000 LDC-MAX-ALL-Macro ICTCAS_OKN_KB_ENG_4 0 0.0000 0.0000 0.0000 LDC-MAX-ALL-Macro ICTCAS_OKN_KB_ENG_4 ALL 0.0000 0.0000 0.0000 LDC-MEAN-ALL-Macro ICTCAS_OKN_KB_ENG_4 0 0.0000 0.0000 0.0000 LDC-MEAN-ALL-Macro ICTCAS_OKN_KB_ENG_4 ALL 0.0000 0.0000 0.0000 *ALL-Macro Prec, Recall and F1 refer to mean-precision, mean-recall and mean-F1. --------------------- Composite KB Evaluation (SENTIMENT-SLOTS only): Scores based on confidence values and Average Precision (AP); k=3 justifications allowed: Metric RunID Hop AP SF-ALL-Macro ICTCAS_OKN_KB_ENG_4 0 0.0000 SF-ALL-Macro ICTCAS_OKN_KB_ENG_4 1 0.0000 SF-ALL-Macro ICTCAS_OKN_KB_ENG_4 ALL 0.0000 LDC-MEAN-ALL-Macro ICTCAS_OKN_KB_ENG_4 0 0.0000 LDC-MEAN-ALL-Macro ICTCAS_OKN_KB_ENG_4 1 0.0000 LDC-MEAN-ALL-Macro ICTCAS_OKN_KB_ENG_4 ALL 0.0000 (PRIMARY METRIC) *ALL-Macro AP refer to mean of corresponding AP values. Scores based on confidence values and Average Precision (AP); only k=1 justification allowed: Metric RunID Hop AP SF-ALL-Macro ICTCAS_OKN_KB_ENG_4 0 0.0000 SF-ALL-Macro ICTCAS_OKN_KB_ENG_4 1 0.0000 SF-ALL-Macro ICTCAS_OKN_KB_ENG_4 ALL 0.0000 LDC-MEAN-ALL-Macro ICTCAS_OKN_KB_ENG_4 0 0.0000 LDC-MEAN-ALL-Macro ICTCAS_OKN_KB_ENG_4 1 0.0000 LDC-MEAN-ALL-Macro ICTCAS_OKN_KB_ENG_4 ALL 0.0000 *ALL-Macro AP refer to mean of corresponding AP values. Scores based on P/R/F1; only k=1 justification allowed: Metric RunID Hop Prec Recall F1 SF-ALL-Micro ICTCAS_OKN_KB_ENG_4 0 0.0000 0.0000 0.0000 SF-ALL-Micro ICTCAS_OKN_KB_ENG_4 1 0.0000 0.0000 0.0000 SF-ALL-Micro ICTCAS_OKN_KB_ENG_4 ALL 0.0000 0.0000 0.0000 SF-ALL-Macro ICTCAS_OKN_KB_ENG_4 0 0.0000 0.0000 0.0000 SF-ALL-Macro ICTCAS_OKN_KB_ENG_4 1 0.0000 0.0000 0.0000 SF-ALL-Macro ICTCAS_OKN_KB_ENG_4 ALL 0.0000 0.0000 0.0000 LDC-MAX-ALL-Micro ICTCAS_OKN_KB_ENG_4 0 0.0000 0.0000 0.0000 LDC-MAX-ALL-Micro ICTCAS_OKN_KB_ENG_4 1 0.0000 0.0000 0.0000 LDC-MAX-ALL-Micro ICTCAS_OKN_KB_ENG_4 ALL 0.0000 0.0000 0.0000 LDC-MAX-ALL-Macro ICTCAS_OKN_KB_ENG_4 0 0.0000 0.0000 0.0000 LDC-MAX-ALL-Macro ICTCAS_OKN_KB_ENG_4 1 0.0000 0.0000 0.0000 LDC-MAX-ALL-Macro ICTCAS_OKN_KB_ENG_4 ALL 0.0000 0.0000 0.0000 LDC-MEAN-ALL-Macro ICTCAS_OKN_KB_ENG_4 0 0.0000 0.0000 0.0000 LDC-MEAN-ALL-Macro ICTCAS_OKN_KB_ENG_4 1 0.0000 0.0000 0.0000 LDC-MEAN-ALL-Macro ICTCAS_OKN_KB_ENG_4 ALL 0.0000 0.0000 0.0000 *ALL-Macro Prec, Recall and F1 refer to mean-precision, mean-recall and mean-F1. ************************************************************* Run ID: ICTCAS_OKN_KB_ENG_5 Did the run access the live Web during the evaluation window: No Did the run extract relations from the Cold Start source corpus: Yes Did the run generate meaningful confidence values: Yes Did the run attempt the full Entity Discovery and Linking task: No Did the run attempt the full Event Nugget Detection and Coreference task: No Did the run include SF relations: Yes Did the run attempt the Event Argument Extraction and Linking task: No Did the run include Sentiment relations: No --------------------- Composite KB Evaluation (All slot types): Scores based on confidence values and Average Precision (AP); k=3 justifications allowed: Metric RunID Hop AP SF-ALL-Macro ICTCAS_OKN_KB_ENG_5 0 0.0056 SF-ALL-Macro ICTCAS_OKN_KB_ENG_5 1 0.0000 SF-ALL-Macro ICTCAS_OKN_KB_ENG_5 ALL 0.0042 LDC-MEAN-ALL-Macro ICTCAS_OKN_KB_ENG_5 0 0.0049 LDC-MEAN-ALL-Macro ICTCAS_OKN_KB_ENG_5 1 0.0000 LDC-MEAN-ALL-Macro ICTCAS_OKN_KB_ENG_5 ALL 0.0038 (PRIMARY METRIC) *ALL-Macro AP refer to mean of corresponding AP values. Scores based on confidence values and Average Precision (AP); only k=1 justification allowed: Metric RunID Hop AP SF-ALL-Macro ICTCAS_OKN_KB_ENG_5 0 0.0142 SF-ALL-Macro ICTCAS_OKN_KB_ENG_5 1 0.0000 SF-ALL-Macro ICTCAS_OKN_KB_ENG_5 ALL 0.0103 LDC-MEAN-ALL-Macro ICTCAS_OKN_KB_ENG_5 0 0.0126 LDC-MEAN-ALL-Macro ICTCAS_OKN_KB_ENG_5 1 0.0000 LDC-MEAN-ALL-Macro ICTCAS_OKN_KB_ENG_5 ALL 0.0095 *ALL-Macro AP refer to mean of corresponding AP values. Scores based on P/R/F1; only k=1 justification allowed: Metric RunID Hop Prec Recall F1 SF-ALL-Micro ICTCAS_OKN_KB_ENG_5 0 0.2500 0.0086 0.0165 SF-ALL-Micro ICTCAS_OKN_KB_ENG_5 1 0.0000 0.0000 0.0000 SF-ALL-Micro ICTCAS_OKN_KB_ENG_5 ALL 0.2439 0.0066 0.0129 SF-ALL-Macro ICTCAS_OKN_KB_ENG_5 0 0.0155 0.0165 0.0151 SF-ALL-Macro ICTCAS_OKN_KB_ENG_5 1 0.0000 0.0000 0.0000 SF-ALL-Macro ICTCAS_OKN_KB_ENG_5 ALL 0.0101 0.0108 0.0099 LDC-MAX-ALL-Micro ICTCAS_OKN_KB_ENG_5 0 0.2778 0.0099 0.0191 LDC-MAX-ALL-Micro ICTCAS_OKN_KB_ENG_5 1 0.0000 0.0000 0.0000 LDC-MAX-ALL-Micro ICTCAS_OKN_KB_ENG_5 ALL 0.2703 0.0078 0.0151 LDC-MAX-ALL-Macro ICTCAS_OKN_KB_ENG_5 0 0.0168 0.0170 0.0153 LDC-MAX-ALL-Macro ICTCAS_OKN_KB_ENG_5 1 0.0000 0.0000 0.0000 LDC-MAX-ALL-Macro ICTCAS_OKN_KB_ENG_5 ALL 0.0110 0.0112 0.0101 LDC-MEAN-ALL-Macro ICTCAS_OKN_KB_ENG_5 0 0.0142 0.0155 0.0138 LDC-MEAN-ALL-Macro ICTCAS_OKN_KB_ENG_5 1 0.0000 0.0000 0.0000 LDC-MEAN-ALL-Macro ICTCAS_OKN_KB_ENG_5 ALL 0.0094 0.0102 0.0091 *ALL-Macro Prec, Recall and F1 refer to mean-precision, mean-recall and mean-F1. --------------------- Composite KB Evaluation (SF-SLOTS only): Scores based on confidence values and Average Precision (AP); k=3 justifications allowed: Metric RunID Hop AP SF-ALL-Macro ICTCAS_OKN_KB_ENG_5 0 0.0130 SF-ALL-Macro ICTCAS_OKN_KB_ENG_5 1 0.0000 SF-ALL-Macro ICTCAS_OKN_KB_ENG_5 ALL 0.0096 LDC-MEAN-ALL-Macro ICTCAS_OKN_KB_ENG_5 0 0.0106 LDC-MEAN-ALL-Macro ICTCAS_OKN_KB_ENG_5 1 0.0000 LDC-MEAN-ALL-Macro ICTCAS_OKN_KB_ENG_5 ALL 0.0081 (PRIMARY METRIC) *ALL-Macro AP refer to mean of corresponding AP values. Scores based on confidence values and Average Precision (AP); only k=1 justification allowed: Metric RunID Hop AP SF-ALL-Macro ICTCAS_OKN_KB_ENG_5 0 0.0328 SF-ALL-Macro ICTCAS_OKN_KB_ENG_5 1 0.0000 SF-ALL-Macro ICTCAS_OKN_KB_ENG_5 ALL 0.0238 LDC-MEAN-ALL-Macro ICTCAS_OKN_KB_ENG_5 0 0.0272 LDC-MEAN-ALL-Macro ICTCAS_OKN_KB_ENG_5 1 0.0000 LDC-MEAN-ALL-Macro ICTCAS_OKN_KB_ENG_5 ALL 0.0205 *ALL-Macro AP refer to mean of corresponding AP values. Scores based on P/R/F1; only k=1 justification allowed: Metric RunID Hop Prec Recall F1 SF-ALL-Micro ICTCAS_OKN_KB_ENG_5 0 0.2500 0.0173 0.0323 SF-ALL-Micro ICTCAS_OKN_KB_ENG_5 1 0.0000 0.0000 0.0000 SF-ALL-Micro ICTCAS_OKN_KB_ENG_5 ALL 0.2439 0.0131 0.0249 SF-ALL-Macro ICTCAS_OKN_KB_ENG_5 0 0.0357 0.0380 0.0350 SF-ALL-Macro ICTCAS_OKN_KB_ENG_5 1 0.0000 0.0000 0.0000 SF-ALL-Macro ICTCAS_OKN_KB_ENG_5 ALL 0.0215 0.0230 0.0211 LDC-MAX-ALL-Micro ICTCAS_OKN_KB_ENG_5 0 0.2857 0.0191 0.0358 LDC-MAX-ALL-Micro ICTCAS_OKN_KB_ENG_5 1 0.0000 0.0000 0.0000 LDC-MAX-ALL-Micro ICTCAS_OKN_KB_ENG_5 ALL 0.2778 0.0147 0.0279 LDC-MAX-ALL-Macro ICTCAS_OKN_KB_ENG_5 0 0.0361 0.0366 0.0330 LDC-MAX-ALL-Macro ICTCAS_OKN_KB_ENG_5 1 0.0000 0.0000 0.0000 LDC-MAX-ALL-Macro ICTCAS_OKN_KB_ENG_5 ALL 0.0221 0.0224 0.0202 LDC-MEAN-ALL-Macro ICTCAS_OKN_KB_ENG_5 0 0.0306 0.0335 0.0298 LDC-MEAN-ALL-Macro ICTCAS_OKN_KB_ENG_5 1 0.0000 0.0000 0.0000 LDC-MEAN-ALL-Macro ICTCAS_OKN_KB_ENG_5 ALL 0.0187 0.0205 0.0182 *ALL-Macro Prec, Recall and F1 refer to mean-precision, mean-recall and mean-F1. --------------------- Composite KB Evaluation (EVENT-SLOTS only): Scores based on confidence values and Average Precision (AP); k=3 justifications allowed: SUMMARY: This section provides summary of AP scores Metric RunID Hop AP SF-ALL-Macro ICTCAS_OKN_KB_ENG_5 0 0.0000 SF-ALL-Macro ICTCAS_OKN_KB_ENG_5 ALL 0.0000 LDC-MEAN-ALL-Macro ICTCAS_OKN_KB_ENG_5 0 0.0000 LDC-MEAN-ALL-Macro ICTCAS_OKN_KB_ENG_5 ALL 0.0000 (PRIMARY METRIC) *ALL-Macro AP refer to mean of corresponding AP values. Scores based on confidence values and Average Precision (AP); only k=1 justification allowed: SUMMARY: This section provides summary of AP scores Metric RunID Hop AP SF-ALL-Macro ICTCAS_OKN_KB_ENG_5 0 0.0000 SF-ALL-Macro ICTCAS_OKN_KB_ENG_5 ALL 0.0000 LDC-MEAN-ALL-Macro ICTCAS_OKN_KB_ENG_5 0 0.0000 LDC-MEAN-ALL-Macro ICTCAS_OKN_KB_ENG_5 ALL 0.0000 *ALL-Macro AP refer to mean of corresponding AP values. Scores based on P/R/F1; only k=1 justification allowed: Metric RunID Hop Prec Recall F1 SF-ALL-Micro ICTCAS_OKN_KB_ENG_5 0 0.0000 0.0000 0.0000 SF-ALL-Micro ICTCAS_OKN_KB_ENG_5 ALL 0.0000 0.0000 0.0000 SF-ALL-Macro ICTCAS_OKN_KB_ENG_5 0 0.0000 0.0000 0.0000 SF-ALL-Macro ICTCAS_OKN_KB_ENG_5 ALL 0.0000 0.0000 0.0000 LDC-MAX-ALL-Micro ICTCAS_OKN_KB_ENG_5 0 0.0000 0.0000 0.0000 LDC-MAX-ALL-Micro ICTCAS_OKN_KB_ENG_5 ALL 0.0000 0.0000 0.0000 LDC-MAX-ALL-Macro ICTCAS_OKN_KB_ENG_5 0 0.0000 0.0000 0.0000 LDC-MAX-ALL-Macro ICTCAS_OKN_KB_ENG_5 ALL 0.0000 0.0000 0.0000 LDC-MEAN-ALL-Macro ICTCAS_OKN_KB_ENG_5 0 0.0000 0.0000 0.0000 LDC-MEAN-ALL-Macro ICTCAS_OKN_KB_ENG_5 ALL 0.0000 0.0000 0.0000 *ALL-Macro Prec, Recall and F1 refer to mean-precision, mean-recall and mean-F1. --------------------- Composite KB Evaluation (SENTIMENT-SLOTS only): Scores based on confidence values and Average Precision (AP); k=3 justifications allowed: Metric RunID Hop AP SF-ALL-Macro ICTCAS_OKN_KB_ENG_5 0 0.0000 SF-ALL-Macro ICTCAS_OKN_KB_ENG_5 1 0.0000 SF-ALL-Macro ICTCAS_OKN_KB_ENG_5 ALL 0.0000 LDC-MEAN-ALL-Macro ICTCAS_OKN_KB_ENG_5 0 0.0000 LDC-MEAN-ALL-Macro ICTCAS_OKN_KB_ENG_5 1 0.0000 LDC-MEAN-ALL-Macro ICTCAS_OKN_KB_ENG_5 ALL 0.0000 (PRIMARY METRIC) *ALL-Macro AP refer to mean of corresponding AP values. Scores based on confidence values and Average Precision (AP); only k=1 justification allowed: Metric RunID Hop AP SF-ALL-Macro ICTCAS_OKN_KB_ENG_5 0 0.0000 SF-ALL-Macro ICTCAS_OKN_KB_ENG_5 1 0.0000 SF-ALL-Macro ICTCAS_OKN_KB_ENG_5 ALL 0.0000 LDC-MEAN-ALL-Macro ICTCAS_OKN_KB_ENG_5 0 0.0000 LDC-MEAN-ALL-Macro ICTCAS_OKN_KB_ENG_5 1 0.0000 LDC-MEAN-ALL-Macro ICTCAS_OKN_KB_ENG_5 ALL 0.0000 *ALL-Macro AP refer to mean of corresponding AP values. Scores based on P/R/F1; only k=1 justification allowed: Metric RunID Hop Prec Recall F1 SF-ALL-Micro ICTCAS_OKN_KB_ENG_5 0 0.0000 0.0000 0.0000 SF-ALL-Micro ICTCAS_OKN_KB_ENG_5 1 0.0000 0.0000 0.0000 SF-ALL-Micro ICTCAS_OKN_KB_ENG_5 ALL 0.0000 0.0000 0.0000 SF-ALL-Macro ICTCAS_OKN_KB_ENG_5 0 0.0000 0.0000 0.0000 SF-ALL-Macro ICTCAS_OKN_KB_ENG_5 1 0.0000 0.0000 0.0000 SF-ALL-Macro ICTCAS_OKN_KB_ENG_5 ALL 0.0000 0.0000 0.0000 LDC-MAX-ALL-Micro ICTCAS_OKN_KB_ENG_5 0 0.0000 0.0000 0.0000 LDC-MAX-ALL-Micro ICTCAS_OKN_KB_ENG_5 1 0.0000 0.0000 0.0000 LDC-MAX-ALL-Micro ICTCAS_OKN_KB_ENG_5 ALL 0.0000 0.0000 0.0000 LDC-MAX-ALL-Macro ICTCAS_OKN_KB_ENG_5 0 0.0000 0.0000 0.0000 LDC-MAX-ALL-Macro ICTCAS_OKN_KB_ENG_5 1 0.0000 0.0000 0.0000 LDC-MAX-ALL-Macro ICTCAS_OKN_KB_ENG_5 ALL 0.0000 0.0000 0.0000 LDC-MEAN-ALL-Macro ICTCAS_OKN_KB_ENG_5 0 0.0000 0.0000 0.0000 LDC-MEAN-ALL-Macro ICTCAS_OKN_KB_ENG_5 1 0.0000 0.0000 0.0000 LDC-MEAN-ALL-Macro ICTCAS_OKN_KB_ENG_5 ALL 0.0000 0.0000 0.0000 *ALL-Macro Prec, Recall and F1 refer to mean-precision, mean-recall and mean-F1.