======================================================================= TAC KBP 2015 TRILINGUAL ENTITY DISCOVERY AND LINKING EVALUATION RESULTS ======================================================================= Team ID: IBM Organization: IBM T.J. Watson Research Center ************************************************************* Run ID: IBM1 Did the run access the live Web during the evaluation window: No Did the run use the Wikipedia links in the Reference KB: Yes Did the run use the relation/slot information encoded in the Reference KB: No Did the run use the text descriptions in the nodes of the Reference KB: No ALL English, Chinese, and Spanish documents: Prec Recall F1 Metric 0.838 0.670 0.745 strong_mention_match 0.798 0.639 0.710 strong_typed_mention_match 0.640 0.512 0.569 strong_typed_all_match 0.705 0.456 0.554 strong_typed_link_match 0.544 0.672 0.601 strong_typed_nil_match 0.728 0.542 0.621 entity_match 0.693 0.554 0.616 mention_ceaf 0.669 0.535 0.594 typed_mention_ceaf 0.629 0.503 0.559 typed_mention_ceaf_plus ONLY English documents: Prec Recall F1 Metric 0.832 0.701 0.761 strong_mention_match 0.792 0.667 0.724 strong_typed_mention_match 0.660 0.556 0.603 strong_typed_all_match 0.678 0.549 0.607 strong_typed_link_match 0.619 0.572 0.595 strong_typed_nil_match 0.703 0.653 0.677 entity_match 0.718 0.605 0.657 mention_ceaf 0.691 0.582 0.632 typed_mention_ceaf 0.647 0.545 0.592 typed_mention_ceaf_plus ONLY Chinese documents: Prec Recall F1 Metric 0.851 0.578 0.688 strong_mention_match 0.819 0.557 0.663 strong_typed_mention_match 0.637 0.433 0.516 strong_typed_all_match 0.770 0.345 0.476 strong_typed_link_match 0.495 0.750 0.597 strong_typed_nil_match 0.750 0.390 0.513 entity_match 0.751 0.510 0.608 mention_ceaf 0.730 0.496 0.591 typed_mention_ceaf 0.634 0.431 0.513 typed_mention_ceaf_plus ONLY Spanish documents: Prec Recall F1 Metric 0.834 0.767 0.799 strong_mention_match 0.784 0.722 0.752 strong_typed_mention_match 0.597 0.549 0.572 strong_typed_all_match 0.696 0.444 0.542 strong_typed_link_match 0.498 0.819 0.620 strong_typed_nil_match 0.769 0.532 0.629 entity_match 0.769 0.707 0.737 mention_ceaf 0.729 0.671 0.699 typed_mention_ceaf 0.590 0.543 0.565 typed_mention_ceaf_plus ************************************************************* Run ID: IBM2 Did the run access the live Web during the evaluation window: No Did the run use the Wikipedia links in the Reference KB: Yes Did the run use the relation/slot information encoded in the Reference KB: No Did the run use the text descriptions in the nodes of the Reference KB: No ALL English, Chinese, and Spanish documents: Prec Recall F1 Metric 0.832 0.684 0.751 strong_mention_match 0.794 0.653 0.716 strong_typed_mention_match 0.571 0.469 0.515 strong_typed_all_match 0.614 0.397 0.482 strong_typed_link_match 0.511 0.673 0.581 strong_typed_nil_match 0.756 0.501 0.602 entity_match 0.620 0.510 0.560 mention_ceaf 0.597 0.491 0.539 typed_mention_ceaf 0.560 0.460 0.505 typed_mention_ceaf_plus ONLY English documents: Prec Recall F1 Metric 0.831 0.707 0.764 strong_mention_match 0.791 0.673 0.727 strong_typed_mention_match 0.629 0.535 0.578 strong_typed_all_match 0.654 0.518 0.578 strong_typed_link_match 0.579 0.577 0.578 strong_typed_nil_match 0.738 0.643 0.687 entity_match 0.693 0.589 0.637 mention_ceaf 0.664 0.565 0.611 typed_mention_ceaf 0.617 0.525 0.567 typed_mention_ceaf_plus ONLY Chinese documents: Prec Recall F1 Metric 0.833 0.610 0.704 strong_mention_match 0.804 0.588 0.680 strong_typed_mention_match 0.471 0.344 0.398 strong_typed_all_match 0.480 0.235 0.315 strong_typed_link_match 0.461 0.740 0.568 strong_typed_nil_match 0.769 0.301 0.432 entity_match 0.547 0.400 0.462 mention_ceaf 0.532 0.389 0.450 typed_mention_ceaf 0.466 0.341 0.394 typed_mention_ceaf_plus ONLY Spanish documents: Prec Recall F1 Metric 0.834 0.767 0.799 strong_mention_match 0.785 0.722 0.752 strong_typed_mention_match 0.582 0.535 0.558 strong_typed_all_match 0.704 0.420 0.526 strong_typed_link_match 0.475 0.832 0.605 strong_typed_nil_match 0.792 0.493 0.608 entity_match 0.762 0.701 0.730 mention_ceaf 0.722 0.664 0.692 typed_mention_ceaf 0.574 0.528 0.550 typed_mention_ceaf_plus ************************************************************* Run ID: IBM3 Did the run access the live Web during the evaluation window: No Did the run use the Wikipedia links in the Reference KB: Yes Did the run use the relation/slot information encoded in the Reference KB: No Did the run use the text descriptions in the nodes of the Reference KB: No ALL English, Chinese, and Spanish documents: Prec Recall F1 Metric 0.838 0.670 0.745 strong_mention_match 0.798 0.639 0.710 strong_typed_mention_match 0.566 0.453 0.504 strong_typed_all_match 0.538 0.389 0.452 strong_typed_link_match 0.622 0.635 0.629 strong_typed_nil_match 0.694 0.424 0.527 entity_match 0.607 0.486 0.540 mention_ceaf 0.589 0.472 0.524 typed_mention_ceaf 0.558 0.447 0.496 typed_mention_ceaf_plus ONLY English documents: Prec Recall F1 Metric 0.832 0.701 0.761 strong_mention_match 0.792 0.667 0.724 strong_typed_mention_match 0.660 0.556 0.603 strong_typed_all_match 0.678 0.549 0.607 strong_typed_link_match 0.619 0.572 0.595 strong_typed_nil_match 0.702 0.653 0.677 entity_match 0.718 0.605 0.657 mention_ceaf 0.691 0.582 0.632 typed_mention_ceaf 0.647 0.545 0.592 typed_mention_ceaf_plus ONLY Chinese documents: Prec Recall F1 Metric 0.851 0.578 0.688 strong_mention_match 0.819 0.557 0.663 strong_typed_mention_match 0.655 0.445 0.530 strong_typed_all_match 0.774 0.361 0.492 strong_typed_link_match 0.517 0.750 0.612 strong_typed_nil_match 0.751 0.397 0.520 entity_match 0.768 0.522 0.622 mention_ceaf 0.747 0.508 0.605 typed_mention_ceaf 0.652 0.443 0.528 typed_mention_ceaf_plus ONLY Spanish documents: Prec Recall F1 Metric 0.833 0.767 0.799 strong_mention_match 0.784 0.722 0.752 strong_typed_mention_match 0.218 0.201 0.209 strong_typed_all_match 0.032 0.033 0.032 strong_typed_link_match 0.985 0.632 0.770 strong_typed_nil_match 0.235 0.028 0.049 entity_match 0.266 0.245 0.255 mention_ceaf 0.265 0.244 0.254 typed_mention_ceaf 0.218 0.200 0.209 typed_mention_ceaf_plus ************************************************************* Run ID: IBM4 Did the run access the live Web during the evaluation window: No Did the run use the Wikipedia links in the Reference KB: Yes Did the run use the relation/slot information encoded in the Reference KB: No Did the run use the text descriptions in the nodes of the Reference KB: No ALL English, Chinese, and Spanish documents: Prec Recall F1 Metric 0.838 0.670 0.745 strong_mention_match 0.798 0.639 0.710 strong_typed_mention_match 0.571 0.457 0.507 strong_typed_all_match 0.544 0.393 0.457 strong_typed_link_match 0.623 0.636 0.630 strong_typed_nil_match 0.694 0.426 0.528 entity_match 0.610 0.488 0.542 mention_ceaf 0.592 0.474 0.526 typed_mention_ceaf 0.562 0.450 0.499 typed_mention_ceaf_plus ONLY English documents: Prec Recall F1 Metric 0.832 0.701 0.761 strong_mention_match 0.792 0.667 0.724 strong_typed_mention_match 0.668 0.562 0.611 strong_typed_all_match 0.689 0.558 0.616 strong_typed_link_match 0.621 0.574 0.596 strong_typed_nil_match 0.702 0.657 0.679 entity_match 0.723 0.609 0.661 mention_ceaf 0.695 0.586 0.636 typed_mention_ceaf 0.654 0.551 0.598 typed_mention_ceaf_plus ONLY Chinese documents: Prec Recall F1 Metric 0.851 0.578 0.688 strong_mention_match 0.819 0.557 0.663 strong_typed_mention_match 0.655 0.445 0.530 strong_typed_all_match 0.774 0.361 0.492 strong_typed_link_match 0.517 0.750 0.612 strong_typed_nil_match 0.751 0.397 0.520 entity_match 0.768 0.522 0.622 mention_ceaf 0.747 0.508 0.605 typed_mention_ceaf 0.652 0.443 0.528 typed_mention_ceaf_plus ONLY Spanish documents: Prec Recall F1 Metric 0.833 0.767 0.799 strong_mention_match 0.784 0.722 0.752 strong_typed_mention_match 0.220 0.203 0.211 strong_typed_all_match 0.034 0.035 0.035 strong_typed_link_match 0.985 0.632 0.770 strong_typed_nil_match 0.242 0.029 0.051 entity_match 0.268 0.247 0.257 mention_ceaf 0.267 0.246 0.256 typed_mention_ceaf 0.220 0.202 0.211 typed_mention_ceaf_plus