======================================================= TAC KBP 2016 ENGLISH KB CONSTRUCTION EVALUATION RESULTS ======================================================= Team ID: ICTCAS_OKN Organization: Institute of Computing Technology, Chinese Academy of Sciences ************************************************************* Run ID: ICTCAS_OKN_KB_ENG_1 Did the run access the live Web during the evaluation window: No Did the run extract relations from the Cold Start source corpus: Yes Did the run generate meaningful confidence values: Yes Entity Discovery Evaluation: ONLY English documents: Prec Recall F1 Metric 0.921 0.652 0.763 strong_mention_match 0.874 0.619 0.725 strong_typed_mention_match 0.000 0.000 0.000 entity_match 0.854 0.443 0.584 b_cubed 0.782 0.554 0.648 mention_ceaf 0.749 0.531 0.621 typed_mention_ceaf Slot Filling Evaluation: Metric RunID Hop Prec Recall F1 SF-ALL-Micro ICTCAS_OKN_KB_ENG_1 0 0.2228 0.1149 0.1516 SF-ALL-Micro ICTCAS_OKN_KB_ENG_1 1 0.0524 0.0343 0.0415 SF-ALL-Micro ICTCAS_OKN_KB_ENG_1 ALL 0.1559 0.0877 0.1122 SF-ALL-Macro ICTCAS_OKN_KB_ENG_1 0 0.1325 0.1517 0.1325 SF-ALL-Macro ICTCAS_OKN_KB_ENG_1 1 0.0246 0.0430 0.0285 SF-ALL-Macro ICTCAS_OKN_KB_ENG_1 ALL 0.0915 0.1104 0.0930 LDC-MAX-ALL-Micro ICTCAS_OKN_KB_ENG_1 0 0.2215 0.1176 0.1537 LDC-MAX-ALL-Micro ICTCAS_OKN_KB_ENG_1 1 0.0534 0.0357 0.0428 LDC-MAX-ALL-Micro ICTCAS_OKN_KB_ENG_1 ALL 0.1563 0.0902 0.1144 LDC-MAX-ALL-Macro ICTCAS_OKN_KB_ENG_1 0 0.1408 0.1627 0.1398 LDC-MAX-ALL-Macro ICTCAS_OKN_KB_ENG_1 1 0.0261 0.0376 0.0282 LDC-MAX-ALL-Macro ICTCAS_OKN_KB_ENG_1 ALL 0.0957 0.1136 0.0960 LDC-MEAN-ALL-Macro ICTCAS_OKN_KB_ENG_1 0 0.1283 0.1462 0.1264 LDC-MEAN-ALL-Macro ICTCAS_OKN_KB_ENG_1 1 0.0261 0.0376 0.0282 LDC-MEAN-ALL-Macro ICTCAS_OKN_KB_ENG_1 ALL 0.0881 0.1035 0.0878 *ALL-Macro Prec, Recall and F1 refer to mean-precision, mean-recall and mean-F1. NIL-DETECTION P/R/F1: 0.2995 0.9350 0.4537 ************************************************************* Run ID: ICTCAS_OKN_KB_ENG_2 Did the run access the live Web during the evaluation window: No Did the run extract relations from the Cold Start source corpus: Yes Did the run generate meaningful confidence values: Yes Entity Discovery Evaluation: ONLY English documents: Prec Recall F1 Metric 0.915 0.680 0.780 strong_mention_match 0.870 0.647 0.742 strong_typed_mention_match 0.000 0.000 0.000 entity_match 0.851 0.452 0.590 b_cubed 0.752 0.559 0.641 mention_ceaf 0.721 0.536 0.615 typed_mention_ceaf Slot Filling Evaluation: Metric RunID Hop Prec Recall F1 SF-ALL-Micro ICTCAS_OKN_KB_ENG_2 0 0.2170 0.1149 0.1502 SF-ALL-Micro ICTCAS_OKN_KB_ENG_2 1 0.0448 0.0294 0.0355 SF-ALL-Micro ICTCAS_OKN_KB_ENG_2 ALL 0.1503 0.0860 0.1094 SF-ALL-Macro ICTCAS_OKN_KB_ENG_2 0 0.1332 0.1515 0.1326 SF-ALL-Macro ICTCAS_OKN_KB_ENG_2 1 0.0193 0.0330 0.0217 SF-ALL-Macro ICTCAS_OKN_KB_ENG_2 ALL 0.0899 0.1065 0.0905 LDC-MAX-ALL-Micro ICTCAS_OKN_KB_ENG_2 0 0.2195 0.1176 0.1532 LDC-MAX-ALL-Micro ICTCAS_OKN_KB_ENG_2 1 0.0483 0.0325 0.0388 LDC-MAX-ALL-Micro ICTCAS_OKN_KB_ENG_2 ALL 0.1533 0.0891 0.1127 LDC-MAX-ALL-Macro ICTCAS_OKN_KB_ENG_2 0 0.1418 0.1623 0.1401 LDC-MAX-ALL-Macro ICTCAS_OKN_KB_ENG_2 1 0.0223 0.0309 0.0236 LDC-MAX-ALL-Macro ICTCAS_OKN_KB_ENG_2 ALL 0.0949 0.1107 0.0943 LDC-MEAN-ALL-Macro ICTCAS_OKN_KB_ENG_2 0 0.1295 0.1458 0.1268 LDC-MEAN-ALL-Macro ICTCAS_OKN_KB_ENG_2 1 0.0223 0.0309 0.0236 LDC-MEAN-ALL-Macro ICTCAS_OKN_KB_ENG_2 ALL 0.0874 0.1007 0.0862 *ALL-Macro Prec, Recall and F1 refer to mean-precision, mean-recall and mean-F1. NIL-DETECTION P/R/F1: 0.2995 0.9350 0.4537 ************************************************************* Run ID: ICTCAS_OKN_KB_ENG_3 Did the run access the live Web during the evaluation window: No Did the run extract relations from the Cold Start source corpus: Yes Did the run generate meaningful confidence values: Yes Entity Discovery Evaluation: ONLY English documents: Prec Recall F1 Metric 0.923 0.610 0.735 strong_mention_match 0.880 0.582 0.700 strong_typed_mention_match 0.000 0.000 0.000 entity_match 0.877 0.343 0.493 b_cubed 0.697 0.461 0.555 mention_ceaf 0.664 0.439 0.528 typed_mention_ceaf Slot Filling Evaluation: Metric RunID Hop Prec Recall F1 SF-ALL-Micro ICTCAS_OKN_KB_ENG_3 0 0.2675 0.0811 0.1245 SF-ALL-Micro ICTCAS_OKN_KB_ENG_3 1 0.1016 0.0319 0.0485 SF-ALL-Micro ICTCAS_OKN_KB_ENG_3 ALL 0.2102 0.0645 0.0987 SF-ALL-Macro ICTCAS_OKN_KB_ENG_3 0 0.1107 0.1102 0.1019 SF-ALL-Macro ICTCAS_OKN_KB_ENG_3 1 0.0267 0.0343 0.0287 SF-ALL-Macro ICTCAS_OKN_KB_ENG_3 ALL 0.0788 0.0813 0.0741 LDC-MAX-ALL-Micro ICTCAS_OKN_KB_ENG_3 0 0.2837 0.0964 0.1439 LDC-MAX-ALL-Micro ICTCAS_OKN_KB_ENG_3 1 0.1092 0.0422 0.0609 LDC-MAX-ALL-Micro ICTCAS_OKN_KB_ENG_3 ALL 0.2202 0.0783 0.1155 LDC-MAX-ALL-Macro ICTCAS_OKN_KB_ENG_3 0 0.1378 0.1410 0.1283 LDC-MAX-ALL-Macro ICTCAS_OKN_KB_ENG_3 1 0.0358 0.0459 0.0384 LDC-MAX-ALL-Macro ICTCAS_OKN_KB_ENG_3 ALL 0.0978 0.1037 0.0930 LDC-MEAN-ALL-Macro ICTCAS_OKN_KB_ENG_3 0 0.1092 0.1074 0.0990 LDC-MEAN-ALL-Macro ICTCAS_OKN_KB_ENG_3 1 0.0275 0.0317 0.0283 LDC-MEAN-ALL-Macro ICTCAS_OKN_KB_ENG_3 ALL 0.0771 0.0777 0.0712 *ALL-Macro Prec, Recall and F1 refer to mean-precision, mean-recall and mean-F1. NIL-DETECTION P/R/F1: 0.3067 0.9675 0.4658 ************************************************************* Run ID: ICTCAS_OKN_KB_ENG_4 Did the run access the live Web during the evaluation window: No Did the run extract relations from the Cold Start source corpus: Yes Did the run generate meaningful confidence values: Yes Entity Discovery Evaluation: ONLY English documents: Prec Recall F1 Metric 0.916 0.680 0.781 strong_mention_match 0.833 0.619 0.710 strong_typed_mention_match 0.000 0.000 0.000 entity_match 0.852 0.452 0.591 b_cubed 0.753 0.559 0.642 mention_ceaf 0.714 0.531 0.609 typed_mention_ceaf Slot Filling Evaluation: Metric RunID Hop Prec Recall F1 SF-ALL-Micro ICTCAS_OKN_KB_ENG_4 0 0.2327 0.1049 0.1446 SF-ALL-Micro ICTCAS_OKN_KB_ENG_4 1 0.0514 0.0221 0.0309 SF-ALL-Micro ICTCAS_OKN_KB_ENG_4 ALL 0.1735 0.0769 0.1066 SF-ALL-Macro ICTCAS_OKN_KB_ENG_4 0 0.1235 0.1374 0.1204 SF-ALL-Macro ICTCAS_OKN_KB_ENG_4 1 0.0186 0.0373 0.0227 SF-ALL-Macro ICTCAS_OKN_KB_ENG_4 ALL 0.0836 0.0994 0.0833 LDC-MAX-ALL-Micro ICTCAS_OKN_KB_ENG_4 0 0.2410 0.1095 0.1506 LDC-MAX-ALL-Micro ICTCAS_OKN_KB_ENG_4 1 0.0420 0.0162 0.0234 LDC-MAX-ALL-Micro ICTCAS_OKN_KB_ENG_4 ALL 0.1814 0.0783 0.1093 LDC-MAX-ALL-Macro ICTCAS_OKN_KB_ENG_4 0 0.1358 0.1489 0.1299 LDC-MAX-ALL-Macro ICTCAS_OKN_KB_ENG_4 1 0.0158 0.0283 0.0186 LDC-MAX-ALL-Macro ICTCAS_OKN_KB_ENG_4 ALL 0.0887 0.1015 0.0862 LDC-MEAN-ALL-Macro ICTCAS_OKN_KB_ENG_4 0 0.1226 0.1316 0.1158 LDC-MEAN-ALL-Macro ICTCAS_OKN_KB_ENG_4 1 0.0158 0.0283 0.0186 LDC-MEAN-ALL-Macro ICTCAS_OKN_KB_ENG_4 ALL 0.0806 0.0911 0.0776 *ALL-Macro Prec, Recall and F1 refer to mean-precision, mean-recall and mean-F1. NIL-DETECTION P/R/F1: 0.3031 0.9512 0.4597