============================================================= TAC KBP 2016 CROSS-LINGUAL KB CONSTRUCTION EVALUATION RESULTS ============================================================= Team ID: hltcoe Organization: Human Language Technology Center of Excellence ************************************************************* Run ID: hltcoe_KB_XLING_1 Did the run access the live Web during the evaluation window: No Did the run extract relations from the Cold Start source corpus: Yes Did the run generate meaningful confidence values: No Run number of the English KB system that is most closely configured to the English component of this run: 3 Run number of the Spanish KB system that is most closely configured to the Spanish component of this run: 2 Run number of the Chinese KB system that is most closely configured to the Chinese component of this run: 2 Entity Discovery Evaluation: ALL English, Chinese, and Spanish documents: Prec Recall F1 Metric 0.747 0.583 0.655 strong_mention_match 0.703 0.549 0.617 strong_typed_mention_match 0.000 0.000 0.000 entity_match 0.603 0.354 0.446 b_cubed 0.605 0.473 0.531 mention_ceaf 0.589 0.460 0.516 typed_mention_ceaf ONLY English documents: Prec Recall F1 Metric 0.756 0.642 0.695 strong_mention_match 0.717 0.609 0.658 strong_typed_mention_match 0.000 0.000 0.000 entity_match 0.617 0.451 0.521 b_cubed 0.657 0.558 0.604 mention_ceaf 0.636 0.540 0.584 typed_mention_ceaf ONLY Chinese documents: Prec Recall F1 Metric 0.723 0.613 0.664 strong_mention_match 0.682 0.579 0.626 strong_typed_mention_match 0.000 0.000 0.000 entity_match 0.594 0.412 0.487 b_cubed 0.613 0.520 0.562 mention_ceaf 0.587 0.498 0.539 typed_mention_ceaf ONLY Spanish documents: Prec Recall F1 Metric 0.771 0.467 0.581 strong_mention_match 0.716 0.433 0.540 strong_typed_mention_match 0.000 0.000 0.000 entity_match 0.634 0.285 0.394 b_cubed 0.641 0.388 0.484 mention_ceaf 0.609 0.369 0.459 typed_mention_ceaf Slot Filling Evaluation: Metric RunID Hop Prec Recall F1 SF-ALL-Micro hltcoe_KB_XLING_1 0 0.4153 0.2042 0.2738 SF-ALL-Micro hltcoe_KB_XLING_1 1 0.1521 0.1786 0.1643 SF-ALL-Micro hltcoe_KB_XLING_1 ALL 0.2748 0.1959 0.2287 SF-ALL-Macro hltcoe_KB_XLING_1 0 0.1807 0.1685 0.1609 SF-ALL-Macro hltcoe_KB_XLING_1 1 0.1308 0.1562 0.1357 SF-ALL-Macro hltcoe_KB_XLING_1 ALL 0.1552 0.1622 0.1480 LDC-MAX-ALL-Micro hltcoe_KB_XLING_1 0 0.4138 0.2920 0.3424 LDC-MAX-ALL-Micro hltcoe_KB_XLING_1 1 0.1577 0.2246 0.1853 LDC-MAX-ALL-Micro hltcoe_KB_XLING_1 ALL 0.2846 0.2694 0.2768 LDC-MAX-ALL-Macro hltcoe_KB_XLING_1 0 0.2661 0.2452 0.2377 LDC-MAX-ALL-Macro hltcoe_KB_XLING_1 1 0.1368 0.1704 0.1424 LDC-MAX-ALL-Macro hltcoe_KB_XLING_1 ALL 0.2022 0.2082 0.1906 LDC-MEAN-ALL-Macro hltcoe_KB_XLING_1 0 0.1742 0.1674 0.1589 LDC-MEAN-ALL-Macro hltcoe_KB_XLING_1 1 0.1090 0.1352 0.1135 LDC-MEAN-ALL-Macro hltcoe_KB_XLING_1 ALL 0.1419 0.1515 0.1364 *ALL-Macro Prec, Recall and F1 refer to mean-precision, mean-recall and mean-F1. NIL-DETECTION P/R/F1: 0.2884 0.8862 0.4352 ************************************************************* Run ID: hltcoe_KB_XLING_2 Did the run access the live Web during the evaluation window: No Did the run extract relations from the Cold Start source corpus: Yes Did the run generate meaningful confidence values: No Run number of the English KB system that is most closely configured to the English component of this run: 3 Run number of the Spanish KB system that is most closely configured to the Spanish component of this run: 2 Run number of the Chinese KB system that is most closely configured to the Chinese component of this run: 2 Entity Discovery Evaluation: ALL English, Chinese, and Spanish documents: Prec Recall F1 Metric 0.747 0.583 0.655 strong_mention_match 0.703 0.549 0.617 strong_typed_mention_match 0.000 0.000 0.000 entity_match 0.635 0.351 0.452 b_cubed 0.610 0.476 0.535 mention_ceaf 0.595 0.465 0.522 typed_mention_ceaf ONLY English documents: Prec Recall F1 Metric 0.756 0.642 0.695 strong_mention_match 0.717 0.609 0.658 strong_typed_mention_match 0.000 0.000 0.000 entity_match 0.633 0.460 0.533 b_cubed 0.670 0.569 0.616 mention_ceaf 0.649 0.551 0.596 typed_mention_ceaf ONLY Chinese documents: Prec Recall F1 Metric 0.723 0.614 0.664 strong_mention_match 0.682 0.579 0.626 strong_typed_mention_match 0.000 0.000 0.000 entity_match 0.638 0.416 0.504 b_cubed 0.637 0.541 0.585 mention_ceaf 0.617 0.524 0.567 typed_mention_ceaf ONLY Spanish documents: Prec Recall F1 Metric 0.771 0.467 0.581 strong_mention_match 0.716 0.433 0.540 strong_typed_mention_match 0.000 0.000 0.000 entity_match 0.648 0.290 0.400 b_cubed 0.660 0.399 0.498 mention_ceaf 0.631 0.382 0.475 typed_mention_ceaf Slot Filling Evaluation: Metric RunID Hop Prec Recall F1 SF-ALL-Micro hltcoe_KB_XLING_2 0 0.3911 0.1737 0.2405 SF-ALL-Micro hltcoe_KB_XLING_2 1 0.1853 0.1384 0.1585 SF-ALL-Micro hltcoe_KB_XLING_2 ALL 0.2991 0.1622 0.2104 SF-ALL-Macro hltcoe_KB_XLING_2 0 0.1615 0.1513 0.1418 SF-ALL-Macro hltcoe_KB_XLING_2 1 0.0809 0.0984 0.0845 SF-ALL-Macro hltcoe_KB_XLING_2 ALL 0.1203 0.1242 0.1125 LDC-MAX-ALL-Micro hltcoe_KB_XLING_2 0 0.4218 0.2787 0.3357 LDC-MAX-ALL-Micro hltcoe_KB_XLING_2 1 0.1909 0.2000 0.1954 LDC-MAX-ALL-Micro hltcoe_KB_XLING_2 ALL 0.3192 0.2523 0.2819 LDC-MAX-ALL-Macro hltcoe_KB_XLING_2 0 0.2573 0.2334 0.2256 LDC-MAX-ALL-Macro hltcoe_KB_XLING_2 1 0.1268 0.1516 0.1303 LDC-MAX-ALL-Macro hltcoe_KB_XLING_2 ALL 0.1927 0.1929 0.1785 LDC-MEAN-ALL-Macro hltcoe_KB_XLING_2 0 0.1587 0.1530 0.1431 LDC-MEAN-ALL-Macro hltcoe_KB_XLING_2 1 0.0738 0.0918 0.0767 LDC-MEAN-ALL-Macro hltcoe_KB_XLING_2 ALL 0.1167 0.1227 0.1102 *ALL-Macro Prec, Recall and F1 refer to mean-precision, mean-recall and mean-F1. NIL-DETECTION P/R/F1: 0.2921 0.9024 0.4413 ************************************************************* Run ID: hltcoe_KB_XLING_3 Did the run access the live Web during the evaluation window: No Did the run extract relations from the Cold Start source corpus: Yes Did the run generate meaningful confidence values: No Run number of the English KB system that is most closely configured to the English component of this run: 2 Run number of the Spanish KB system that is most closely configured to the Spanish component of this run: 2 Run number of the Chinese KB system that is most closely configured to the Chinese component of this run: 2 Entity Discovery Evaluation: ALL English, Chinese, and Spanish documents: Prec Recall F1 Metric 0.747 0.583 0.655 strong_mention_match 0.703 0.549 0.617 strong_typed_mention_match 0.000 0.000 0.000 entity_match 0.634 0.350 0.451 b_cubed 0.609 0.476 0.534 mention_ceaf 0.594 0.464 0.521 typed_mention_ceaf ONLY English documents: Prec Recall F1 Metric 0.756 0.642 0.694 strong_mention_match 0.716 0.609 0.658 strong_typed_mention_match 0.000 0.000 0.000 entity_match 0.633 0.459 0.532 b_cubed 0.669 0.568 0.615 mention_ceaf 0.647 0.550 0.595 typed_mention_ceaf ONLY Chinese documents: Prec Recall F1 Metric 0.723 0.614 0.664 strong_mention_match 0.682 0.579 0.626 strong_typed_mention_match 0.000 0.000 0.000 entity_match 0.638 0.416 0.504 b_cubed 0.637 0.541 0.585 mention_ceaf 0.617 0.524 0.567 typed_mention_ceaf ONLY Spanish documents: Prec Recall F1 Metric 0.771 0.467 0.581 strong_mention_match 0.716 0.433 0.540 strong_typed_mention_match 0.000 0.000 0.000 entity_match 0.647 0.290 0.400 b_cubed 0.660 0.399 0.497 mention_ceaf 0.630 0.381 0.475 typed_mention_ceaf Slot Filling Evaluation: Metric RunID Hop Prec Recall F1 SF-ALL-Micro hltcoe_KB_XLING_3 0 0.4261 0.1698 0.2428 SF-ALL-Micro hltcoe_KB_XLING_3 1 0.2054 0.1344 0.1625 SF-ALL-Micro hltcoe_KB_XLING_3 ALL 0.3289 0.1583 0.2137 SF-ALL-Macro hltcoe_KB_XLING_3 0 0.1576 0.1405 0.1346 SF-ALL-Macro hltcoe_KB_XLING_3 1 0.0774 0.0936 0.0805 SF-ALL-Macro hltcoe_KB_XLING_3 ALL 0.1166 0.1165 0.1069 LDC-MAX-ALL-Micro hltcoe_KB_XLING_3 0 0.4435 0.2730 0.3379 LDC-MAX-ALL-Micro hltcoe_KB_XLING_3 1 0.2077 0.1934 0.2003 LDC-MAX-ALL-Micro hltcoe_KB_XLING_3 ALL 0.3415 0.2463 0.2862 LDC-MAX-ALL-Macro hltcoe_KB_XLING_3 0 0.2586 0.2266 0.2229 LDC-MAX-ALL-Macro hltcoe_KB_XLING_3 1 0.1193 0.1436 0.1229 LDC-MAX-ALL-Macro hltcoe_KB_XLING_3 ALL 0.1897 0.1855 0.1734 LDC-MEAN-ALL-Macro hltcoe_KB_XLING_3 0 0.1540 0.1422 0.1351 LDC-MEAN-ALL-Macro hltcoe_KB_XLING_3 1 0.0707 0.0882 0.0733 LDC-MEAN-ALL-Macro hltcoe_KB_XLING_3 ALL 0.1128 0.1155 0.1046 *ALL-Macro Prec, Recall and F1 refer to mean-precision, mean-recall and mean-F1. NIL-DETECTION P/R/F1: 0.3049 0.9593 0.4627 ************************************************************* Run ID: hltcoe_KB_XLING_4 Did the run access the live Web during the evaluation window: No Did the run extract relations from the Cold Start source corpus: Yes Did the run generate meaningful confidence values: No Run number of the English KB system that is most closely configured to the English component of this run: 1 Run number of the Spanish KB system that is most closely configured to the Spanish component of this run: 1 Run number of the Chinese KB system that is most closely configured to the Chinese component of this run: 1 Entity Discovery Evaluation: ALL English, Chinese, and Spanish documents: Prec Recall F1 Metric 0.745 0.583 0.655 strong_mention_match 0.702 0.549 0.616 strong_typed_mention_match 0.000 0.000 0.000 entity_match 0.645 0.267 0.378 b_cubed 0.482 0.377 0.423 mention_ceaf 0.467 0.366 0.410 typed_mention_ceaf ONLY English documents: Prec Recall F1 Metric 0.753 0.642 0.693 strong_mention_match 0.713 0.609 0.657 strong_typed_mention_match 0.000 0.000 0.000 entity_match 0.640 0.413 0.502 b_cubed 0.616 0.526 0.567 mention_ceaf 0.597 0.509 0.550 typed_mention_ceaf ONLY Chinese documents: Prec Recall F1 Metric 0.723 0.614 0.664 strong_mention_match 0.682 0.579 0.626 strong_typed_mention_match 0.000 0.000 0.000 entity_match 0.641 0.400 0.492 b_cubed 0.618 0.525 0.568 mention_ceaf 0.598 0.508 0.550 typed_mention_ceaf ONLY Spanish documents: Prec Recall F1 Metric 0.771 0.467 0.581 strong_mention_match 0.716 0.433 0.540 strong_typed_mention_match 0.000 0.000 0.000 entity_match 0.665 0.248 0.362 b_cubed 0.568 0.344 0.428 mention_ceaf 0.539 0.326 0.407 typed_mention_ceaf Slot Filling Evaluation: Metric RunID Hop Prec Recall F1 SF-ALL-Micro hltcoe_KB_XLING_4 0 0.4531 0.0779 0.1330 SF-ALL-Micro hltcoe_KB_XLING_4 1 0.1779 0.0539 0.0828 SF-ALL-Micro hltcoe_KB_XLING_4 ALL 0.3269 0.0701 0.1155 SF-ALL-Macro hltcoe_KB_XLING_4 0 0.1196 0.0921 0.0934 SF-ALL-Macro hltcoe_KB_XLING_4 1 0.0357 0.0388 0.0355 SF-ALL-Macro hltcoe_KB_XLING_4 ALL 0.0767 0.0648 0.0637 LDC-MAX-ALL-Micro hltcoe_KB_XLING_4 0 0.5083 0.1778 0.2635 LDC-MAX-ALL-Micro hltcoe_KB_XLING_4 1 0.3049 0.1230 0.1752 LDC-MAX-ALL-Micro hltcoe_KB_XLING_4 ALL 0.4335 0.1594 0.2331 LDC-MAX-ALL-Macro hltcoe_KB_XLING_4 0 0.2406 0.1953 0.1962 LDC-MAX-ALL-Macro hltcoe_KB_XLING_4 1 0.0742 0.0820 0.0732 LDC-MAX-ALL-Macro hltcoe_KB_XLING_4 ALL 0.1583 0.1393 0.1354 LDC-MEAN-ALL-Macro hltcoe_KB_XLING_4 0 0.1186 0.0984 0.0973 LDC-MEAN-ALL-Macro hltcoe_KB_XLING_4 1 0.0338 0.0378 0.0332 LDC-MEAN-ALL-Macro hltcoe_KB_XLING_4 ALL 0.0766 0.0684 0.0656 *ALL-Macro Prec, Recall and F1 refer to mean-precision, mean-recall and mean-F1. NIL-DETECTION P/R/F1: 0.3067 0.9675 0.4658