======================================================================== TAC KBP 2017 TRI-LINGUAL ENTITY DISCOVERY AND LINKING EVALUATION RESULTS ======================================================================== Team ID: TAI Organization: AI platform department of Tencent ************************************************************* Run ID: TAI1 Languages addressed: Chinese, English, Spanish Did the run access the live Web during the evaluation window: No Did the run use the Wikipedia links in the Reference KB: Yes Did the run use the relation/slot information encoded in the Reference KB: Yes Did the run use the text descriptions in the nodes of the Reference KB: Yes ALL English, Chinese, and Spanish documents: Prec Recall F1 Metric 0.860 0.769 0.812 strong_mention_match 0.820 0.734 0.774 strong_typed_mention_match 0.708 0.634 0.669 strong_typed_all_match 0.769 0.648 0.703 strong_typed_link_match 0.586 0.599 0.593 strong_typed_nil_match 0.791 0.730 0.759 entity_match 0.753 0.674 0.711 mention_ceaf 0.731 0.654 0.690 typed_mention_ceaf 0.694 0.621 0.655 typed_mention_ceaf_plus 0.757 0.586 0.661 b_cubed 0.665 0.538 0.595 b_cubed_plus ONLY English documents: Prec Recall F1 Metric 0.843 0.817 0.830 strong_mention_match 0.793 0.768 0.781 strong_typed_mention_match 0.680 0.659 0.669 strong_typed_all_match 0.713 0.656 0.683 strong_typed_link_match 0.625 0.664 0.644 strong_typed_nil_match 0.758 0.771 0.765 entity_match 0.733 0.710 0.722 mention_ceaf 0.703 0.681 0.692 typed_mention_ceaf 0.658 0.637 0.648 typed_mention_ceaf_plus 0.742 0.640 0.687 b_cubed 0.643 0.578 0.609 b_cubed_plus ONLY Chinese documents: Prec Recall F1 Metric 0.868 0.742 0.800 strong_mention_match 0.837 0.715 0.771 strong_typed_mention_match 0.760 0.650 0.701 strong_typed_all_match 0.819 0.707 0.759 strong_typed_link_match 0.577 0.479 0.523 strong_typed_nil_match 0.829 0.771 0.799 entity_match 0.805 0.689 0.742 mention_ceaf 0.784 0.670 0.723 typed_mention_ceaf 0.750 0.641 0.691 typed_mention_ceaf_plus 0.782 0.609 0.685 b_cubed 0.719 0.567 0.634 b_cubed_plus ONLY Spanish documents: Prec Recall F1 Metric 0.866 0.762 0.811 strong_mention_match 0.824 0.726 0.772 strong_typed_mention_match 0.666 0.587 0.624 strong_typed_all_match 0.745 0.549 0.632 strong_typed_link_match 0.559 0.670 0.609 strong_typed_nil_match 0.781 0.651 0.710 entity_match 0.729 0.641 0.682 mention_ceaf 0.709 0.624 0.664 typed_mention_ceaf 0.655 0.577 0.614 typed_mention_ceaf_plus 0.772 0.549 0.642 b_cubed 0.634 0.492 0.554 b_cubed_plus ************************************************************* Run ID: TAI2 Languages addressed: Chinese, English, Spanish Did the run access the live Web during the evaluation window: No Did the run use the Wikipedia links in the Reference KB: Yes Did the run use the relation/slot information encoded in the Reference KB: Yes Did the run use the text descriptions in the nodes of the Reference KB: Yes ALL English, Chinese, and Spanish documents: Prec Recall F1 Metric 0.869 0.770 0.816 strong_mention_match 0.832 0.738 0.782 strong_typed_mention_match 0.720 0.639 0.677 strong_typed_all_match 0.771 0.658 0.710 strong_typed_link_match 0.613 0.593 0.603 strong_typed_nil_match 0.788 0.737 0.761 entity_match 0.763 0.676 0.717 mention_ceaf 0.742 0.658 0.698 typed_mention_ceaf 0.706 0.625 0.663 typed_mention_ceaf_plus 0.763 0.593 0.667 b_cubed 0.676 0.543 0.602 b_cubed_plus ONLY English documents: Prec Recall F1 Metric 0.843 0.817 0.830 strong_mention_match 0.794 0.769 0.781 strong_typed_mention_match 0.679 0.657 0.668 strong_typed_all_match 0.707 0.659 0.682 strong_typed_link_match 0.630 0.654 0.642 strong_typed_nil_match 0.748 0.772 0.760 entity_match 0.734 0.711 0.722 mention_ceaf 0.704 0.681 0.692 typed_mention_ceaf 0.658 0.637 0.647 typed_mention_ceaf_plus 0.740 0.642 0.688 b_cubed 0.641 0.577 0.607 b_cubed_plus ONLY Chinese documents: Prec Recall F1 Metric 0.885 0.733 0.802 strong_mention_match 0.861 0.713 0.780 strong_typed_mention_match 0.786 0.651 0.713 strong_typed_all_match 0.833 0.711 0.767 strong_typed_link_match 0.629 0.475 0.541 strong_typed_nil_match 0.855 0.768 0.809 entity_match 0.824 0.682 0.746 mention_ceaf 0.807 0.668 0.731 typed_mention_ceaf 0.775 0.642 0.702 typed_mention_ceaf_plus 0.795 0.607 0.689 b_cubed 0.741 0.568 0.643 b_cubed_plus ONLY Spanish documents: Prec Recall F1 Metric 0.874 0.778 0.823 strong_mention_match 0.834 0.743 0.786 strong_typed_mention_match 0.677 0.602 0.637 strong_typed_all_match 0.736 0.575 0.645 strong_typed_link_match 0.586 0.664 0.622 strong_typed_nil_match 0.754 0.674 0.712 entity_match 0.736 0.656 0.694 mention_ceaf 0.717 0.639 0.676 typed_mention_ceaf 0.665 0.592 0.626 typed_mention_ceaf_plus 0.776 0.568 0.656 b_cubed 0.643 0.507 0.567 b_cubed_plus ************************************************************* Run ID: TAI3 Languages addressed: Chinese, English, Spanish Did the run access the live Web during the evaluation window: No Did the run use the Wikipedia links in the Reference KB: Yes Did the run use the relation/slot information encoded in the Reference KB: Yes Did the run use the text descriptions in the nodes of the Reference KB: Yes ALL English, Chinese, and Spanish documents: Prec Recall F1 Metric 0.853 0.790 0.820 strong_mention_match 0.813 0.753 0.782 strong_typed_mention_match 0.700 0.648 0.673 strong_typed_all_match 0.760 0.660 0.706 strong_typed_link_match 0.581 0.618 0.599 strong_typed_nil_match 0.787 0.737 0.761 entity_match 0.740 0.685 0.712 mention_ceaf 0.719 0.666 0.691 typed_mention_ceaf 0.683 0.632 0.656 typed_mention_ceaf_plus 0.743 0.604 0.666 b_cubed 0.653 0.551 0.598 b_cubed_plus ONLY English documents: Prec Recall F1 Metric 0.831 0.842 0.837 strong_mention_match 0.779 0.789 0.784 strong_typed_mention_match 0.664 0.672 0.668 strong_typed_all_match 0.699 0.662 0.680 strong_typed_link_match 0.606 0.692 0.646 strong_typed_nil_match 0.748 0.772 0.760 entity_match 0.713 0.722 0.717 mention_ceaf 0.683 0.692 0.688 typed_mention_ceaf 0.639 0.647 0.643 typed_mention_ceaf_plus 0.726 0.657 0.690 b_cubed 0.624 0.589 0.606 b_cubed_plus ONLY Chinese documents: Prec Recall F1 Metric 0.865 0.743 0.799 strong_mention_match 0.839 0.721 0.775 strong_typed_mention_match 0.763 0.655 0.705 strong_typed_all_match 0.820 0.713 0.763 strong_typed_link_match 0.582 0.482 0.527 strong_typed_nil_match 0.853 0.768 0.808 entity_match 0.800 0.687 0.739 mention_ceaf 0.782 0.672 0.723 typed_mention_ceaf 0.751 0.645 0.694 typed_mention_ceaf_plus 0.767 0.614 0.682 b_cubed 0.712 0.572 0.634 b_cubed_plus ONLY Spanish documents: Prec Recall F1 Metric 0.859 0.806 0.832 strong_mention_match 0.815 0.764 0.789 strong_typed_mention_match 0.655 0.615 0.634 strong_typed_all_match 0.724 0.578 0.643 strong_typed_link_match 0.558 0.698 0.620 strong_typed_nil_match 0.753 0.675 0.712 entity_match 0.714 0.670 0.691 mention_ceaf 0.692 0.649 0.670 typed_mention_ceaf 0.640 0.601 0.620 typed_mention_ceaf_plus 0.758 0.583 0.659 b_cubed 0.622 0.518 0.565 b_cubed_plus ************************************************************* Run ID: TAI4 Languages addressed: Chinese, English, Spanish Did the run access the live Web during the evaluation window: No Did the run use the Wikipedia links in the Reference KB: Yes Did the run use the relation/slot information encoded in the Reference KB: Yes Did the run use the text descriptions in the nodes of the Reference KB: Yes ALL English, Chinese, and Spanish documents: Prec Recall F1 Metric 0.885 0.714 0.790 strong_mention_match 0.850 0.686 0.759 strong_typed_mention_match 0.760 0.613 0.678 strong_typed_all_match 0.760 0.660 0.706 strong_typed_link_match 0.760 0.498 0.602 strong_typed_nil_match 0.787 0.737 0.761 entity_match 0.810 0.654 0.724 mention_ceaf 0.790 0.637 0.705 typed_mention_ceaf 0.754 0.609 0.674 typed_mention_ceaf_plus 0.767 0.570 0.654 b_cubed 0.705 0.526 0.603 b_cubed_plus ONLY English documents: Prec Recall F1 Metric 0.866 0.734 0.795 strong_mention_match 0.817 0.692 0.749 strong_typed_mention_match 0.721 0.611 0.662 strong_typed_all_match 0.699 0.662 0.680 strong_typed_link_match 0.783 0.513 0.620 strong_typed_nil_match 0.748 0.772 0.760 entity_match 0.793 0.672 0.727 mention_ceaf 0.761 0.645 0.699 typed_mention_ceaf 0.716 0.607 0.657 typed_mention_ceaf_plus 0.754 0.603 0.670 b_cubed 0.677 0.544 0.604 b_cubed_plus ONLY Chinese documents: Prec Recall F1 Metric 0.894 0.713 0.793 strong_mention_match 0.871 0.695 0.773 strong_typed_mention_match 0.800 0.638 0.710 strong_typed_all_match 0.820 0.713 0.763 strong_typed_link_match 0.708 0.416 0.524 strong_typed_nil_match 0.853 0.768 0.808 entity_match 0.840 0.671 0.746 mention_ceaf 0.824 0.658 0.732 typed_mention_ceaf 0.793 0.633 0.704 typed_mention_ceaf_plus 0.792 0.597 0.681 b_cubed 0.745 0.559 0.639 b_cubed_plus ONLY Spanish documents: Prec Recall F1 Metric 0.891 0.695 0.781 strong_mention_match 0.855 0.667 0.749 strong_typed_mention_match 0.742 0.578 0.650 strong_typed_all_match 0.724 0.578 0.643 strong_typed_link_match 0.785 0.578 0.666 strong_typed_nil_match 0.753 0.675 0.712 entity_match 0.808 0.630 0.708 mention_ceaf 0.787 0.613 0.689 typed_mention_ceaf 0.739 0.576 0.648 typed_mention_ceaf_plus 0.780 0.541 0.639 b_cubed 0.697 0.491 0.577 b_cubed_plus ************************************************************* Run ID: TAI5 Languages addressed: Chinese, English, Spanish Did the run access the live Web during the evaluation window: No Did the run use the Wikipedia links in the Reference KB: Yes Did the run use the relation/slot information encoded in the Reference KB: Yes Did the run use the text descriptions in the nodes of the Reference KB: Yes ALL English, Chinese, and Spanish documents: Prec Recall F1 Metric 0.854 0.794 0.823 strong_mention_match 0.817 0.759 0.787 strong_typed_mention_match 0.703 0.654 0.678 strong_typed_all_match 0.762 0.671 0.713 strong_typed_link_match 0.586 0.614 0.599 strong_typed_nil_match 0.787 0.747 0.767 entity_match 0.744 0.692 0.717 mention_ceaf 0.723 0.672 0.697 typed_mention_ceaf 0.687 0.639 0.662 typed_mention_ceaf_plus 0.746 0.612 0.672 b_cubed 0.655 0.559 0.604 b_cubed_plus ONLY English documents: Prec Recall F1 Metric 0.850 0.848 0.849 strong_mention_match 0.803 0.801 0.802 strong_typed_mention_match 0.685 0.683 0.684 strong_typed_all_match 0.718 0.683 0.700 strong_typed_link_match 0.629 0.684 0.655 strong_typed_nil_match 0.760 0.785 0.772 entity_match 0.736 0.734 0.735 mention_ceaf 0.707 0.705 0.706 typed_mention_ceaf 0.662 0.660 0.661 typed_mention_ceaf_plus 0.748 0.667 0.705 b_cubed 0.646 0.600 0.622 b_cubed_plus ONLY Chinese documents: Prec Recall F1 Metric 0.865 0.743 0.799 strong_mention_match 0.839 0.721 0.775 strong_typed_mention_match 0.762 0.654 0.704 strong_typed_all_match 0.819 0.712 0.762 strong_typed_link_match 0.583 0.483 0.528 strong_typed_nil_match 0.850 0.768 0.807 entity_match 0.800 0.687 0.739 mention_ceaf 0.782 0.672 0.723 typed_mention_ceaf 0.751 0.645 0.694 typed_mention_ceaf_plus 0.768 0.613 0.682 b_cubed 0.712 0.572 0.634 b_cubed_plus ONLY Spanish documents: Prec Recall F1 Metric 0.845 0.816 0.831 strong_mention_match 0.802 0.774 0.788 strong_typed_mention_match 0.648 0.625 0.636 strong_typed_all_match 0.715 0.596 0.650 strong_typed_link_match 0.548 0.692 0.612 strong_typed_nil_match 0.746 0.694 0.719 entity_match 0.703 0.679 0.691 mention_ceaf 0.682 0.658 0.670 typed_mention_ceaf 0.633 0.611 0.622 typed_mention_ceaf_plus 0.745 0.597 0.663 b_cubed 0.609 0.531 0.567 b_cubed_plus