======================================================= TAC KBP 2016 CHINESE KB CONSTRUCTION EVALUATION RESULTS ======================================================= Team ID: hltcoe Organization: Human Language Technology Center of Excellence ************************************************************* Run ID: hltcoe_KB_CMN_1 Did the run access the live Web during the evaluation window: No Did the run extract relations from the Cold Start source corpus: Yes Did the run generate meaningful confidence values: No Entity Discovery Evaluation: ONLY Chinese documents: Prec Recall F1 Metric 0.723 0.614 0.664 strong_mention_match 0.682 0.579 0.626 strong_typed_mention_match 0.000 0.000 0.000 entity_match 0.638 0.416 0.504 b_cubed 0.637 0.541 0.585 mention_ceaf 0.617 0.524 0.567 typed_mention_ceaf Slot Filling Evaluation: Metric RunID Hop Prec Recall F1 SF-ALL-Micro hltcoe_KB_CMN_1 0 0.5301 0.1877 0.2773 SF-ALL-Micro hltcoe_KB_CMN_1 1 0.2984 0.2478 0.2708 SF-ALL-Micro hltcoe_KB_CMN_1 ALL 0.4333 0.2018 0.2754 SF-ALL-Macro hltcoe_KB_CMN_1 0 0.1664 0.1530 0.1496 SF-ALL-Macro hltcoe_KB_CMN_1 1 0.0911 0.1038 0.0938 SF-ALL-Macro hltcoe_KB_CMN_1 ALL 0.1315 0.1302 0.1237 LDC-MAX-ALL-Micro hltcoe_KB_CMN_1 0 0.5413 0.2449 0.3372 LDC-MAX-ALL-Micro hltcoe_KB_CMN_1 1 0.2984 0.3098 0.3040 LDC-MAX-ALL-Micro hltcoe_KB_CMN_1 ALL 0.4342 0.2615 0.3264 LDC-MAX-ALL-Macro hltcoe_KB_CMN_1 0 0.2069 0.1868 0.1840 LDC-MAX-ALL-Macro hltcoe_KB_CMN_1 1 0.1251 0.1425 0.1288 LDC-MAX-ALL-Macro hltcoe_KB_CMN_1 ALL 0.1700 0.1668 0.1591 LDC-MEAN-ALL-Macro hltcoe_KB_CMN_1 0 0.1937 0.1760 0.1736 LDC-MEAN-ALL-Macro hltcoe_KB_CMN_1 1 0.1194 0.1357 0.1226 LDC-MEAN-ALL-Macro hltcoe_KB_CMN_1 ALL 0.1602 0.1578 0.1506 *ALL-Macro Prec, Recall and F1 refer to mean-precision, mean-recall and mean-F1. NIL-DETECTION P/R/F1: 0.3085 0.9756 0.4688 ************************************************************* Run ID: hltcoe_KB_CMN_2 Did the run access the live Web during the evaluation window: No Did the run extract relations from the Cold Start source corpus: Yes Did the run generate meaningful confidence values: No Entity Discovery Evaluation: ONLY Chinese documents: Prec Recall F1 Metric 0.723 0.614 0.664 strong_mention_match 0.682 0.579 0.626 strong_typed_mention_match 0.000 0.000 0.000 entity_match 0.638 0.416 0.504 b_cubed 0.637 0.541 0.585 mention_ceaf 0.617 0.524 0.567 typed_mention_ceaf Slot Filling Evaluation: Metric RunID Hop Prec Recall F1 SF-ALL-Micro hltcoe_KB_CMN_2 0 0.5301 0.1877 0.2773 SF-ALL-Micro hltcoe_KB_CMN_2 1 0.2984 0.2478 0.2708 SF-ALL-Micro hltcoe_KB_CMN_2 ALL 0.4333 0.2018 0.2754 SF-ALL-Macro hltcoe_KB_CMN_2 0 0.1664 0.1530 0.1496 SF-ALL-Macro hltcoe_KB_CMN_2 1 0.0911 0.1038 0.0938 SF-ALL-Macro hltcoe_KB_CMN_2 ALL 0.1315 0.1302 0.1237 LDC-MAX-ALL-Micro hltcoe_KB_CMN_2 0 0.5413 0.2449 0.3372 LDC-MAX-ALL-Micro hltcoe_KB_CMN_2 1 0.2984 0.3098 0.3040 LDC-MAX-ALL-Micro hltcoe_KB_CMN_2 ALL 0.4342 0.2615 0.3264 LDC-MAX-ALL-Macro hltcoe_KB_CMN_2 0 0.2069 0.1868 0.1840 LDC-MAX-ALL-Macro hltcoe_KB_CMN_2 1 0.1251 0.1425 0.1288 LDC-MAX-ALL-Macro hltcoe_KB_CMN_2 ALL 0.1700 0.1668 0.1591 LDC-MEAN-ALL-Macro hltcoe_KB_CMN_2 0 0.1937 0.1760 0.1736 LDC-MEAN-ALL-Macro hltcoe_KB_CMN_2 1 0.1194 0.1357 0.1226 LDC-MEAN-ALL-Macro hltcoe_KB_CMN_2 ALL 0.1602 0.1578 0.1506 *ALL-Macro Prec, Recall and F1 refer to mean-precision, mean-recall and mean-F1. NIL-DETECTION P/R/F1: 0.3085 0.9756 0.4688 ************************************************************* Run ID: hltcoe_KB_CMN_3 Did the run access the live Web during the evaluation window: No Did the run extract relations from the Cold Start source corpus: Yes Did the run generate meaningful confidence values: No Entity Discovery Evaluation: ONLY Chinese documents: Prec Recall F1 Metric 0.723 0.614 0.664 strong_mention_match 0.682 0.579 0.626 strong_typed_mention_match 0.000 0.000 0.000 entity_match 0.638 0.416 0.504 b_cubed 0.637 0.541 0.585 mention_ceaf 0.617 0.524 0.567 typed_mention_ceaf Slot Filling Evaluation: Metric RunID Hop Prec Recall F1 SF-ALL-Micro hltcoe_KB_CMN_3 0 0.5301 0.1877 0.2773 SF-ALL-Micro hltcoe_KB_CMN_3 1 0.2984 0.2478 0.2708 SF-ALL-Micro hltcoe_KB_CMN_3 ALL 0.4333 0.2018 0.2754 SF-ALL-Macro hltcoe_KB_CMN_3 0 0.1664 0.1530 0.1496 SF-ALL-Macro hltcoe_KB_CMN_3 1 0.0911 0.1038 0.0938 SF-ALL-Macro hltcoe_KB_CMN_3 ALL 0.1315 0.1302 0.1237 LDC-MAX-ALL-Micro hltcoe_KB_CMN_3 0 0.5413 0.2449 0.3372 LDC-MAX-ALL-Micro hltcoe_KB_CMN_3 1 0.2984 0.3098 0.3040 LDC-MAX-ALL-Micro hltcoe_KB_CMN_3 ALL 0.4342 0.2615 0.3264 LDC-MAX-ALL-Macro hltcoe_KB_CMN_3 0 0.2069 0.1868 0.1840 LDC-MAX-ALL-Macro hltcoe_KB_CMN_3 1 0.1251 0.1425 0.1288 LDC-MAX-ALL-Macro hltcoe_KB_CMN_3 ALL 0.1700 0.1668 0.1591 LDC-MEAN-ALL-Macro hltcoe_KB_CMN_3 0 0.1937 0.1760 0.1736 LDC-MEAN-ALL-Macro hltcoe_KB_CMN_3 1 0.1194 0.1357 0.1226 LDC-MEAN-ALL-Macro hltcoe_KB_CMN_3 ALL 0.1602 0.1578 0.1506 *ALL-Macro Prec, Recall and F1 refer to mean-precision, mean-recall and mean-F1. NIL-DETECTION P/R/F1: 0.3085 0.9756 0.4688 ************************************************************* Run ID: hltcoe_KB_CMN_4 Did the run access the live Web during the evaluation window: No Did the run extract relations from the Cold Start source corpus: Yes Did the run generate meaningful confidence values: No Entity Discovery Evaluation: ONLY Chinese documents: Prec Recall F1 Metric 0.765 0.600 0.673 strong_mention_match 0.725 0.569 0.638 strong_typed_mention_match 0.000 0.000 0.000 entity_match 0.673 0.413 0.512 b_cubed 0.682 0.536 0.600 mention_ceaf 0.661 0.519 0.582 typed_mention_ceaf Slot Filling Evaluation: Metric RunID Hop Prec Recall F1 SF-ALL-Micro hltcoe_KB_CMN_4 0 0.5296 0.1904 0.2801 SF-ALL-Micro hltcoe_KB_CMN_4 1 0.2908 0.2478 0.2676 SF-ALL-Micro hltcoe_KB_CMN_4 ALL 0.4292 0.2039 0.2764 SF-ALL-Macro hltcoe_KB_CMN_4 0 0.1629 0.1561 0.1526 SF-ALL-Macro hltcoe_KB_CMN_4 1 0.0915 0.1038 0.0940 SF-ALL-Macro hltcoe_KB_CMN_4 ALL 0.1298 0.1319 0.1254 LDC-MAX-ALL-Micro hltcoe_KB_CMN_4 0 0.5519 0.2486 0.3428 LDC-MAX-ALL-Micro hltcoe_KB_CMN_4 1 0.2938 0.3098 0.3016 LDC-MAX-ALL-Micro hltcoe_KB_CMN_4 ALL 0.4368 0.2643 0.3293 LDC-MAX-ALL-Macro hltcoe_KB_CMN_4 0 0.2023 0.1908 0.1879 LDC-MAX-ALL-Macro hltcoe_KB_CMN_4 1 0.1256 0.1425 0.1290 LDC-MAX-ALL-Macro hltcoe_KB_CMN_4 ALL 0.1677 0.1690 0.1613 LDC-MEAN-ALL-Macro hltcoe_KB_CMN_4 0 0.1886 0.1801 0.1772 LDC-MEAN-ALL-Macro hltcoe_KB_CMN_4 1 0.1199 0.1357 0.1228 LDC-MEAN-ALL-Macro hltcoe_KB_CMN_4 ALL 0.1576 0.1601 0.1527 *ALL-Macro Prec, Recall and F1 refer to mean-precision, mean-recall and mean-F1. NIL-DETECTION P/R/F1: 0.3085 0.9756 0.4688