================================================================================ TAC KBP 2016 ENTITY DISCOVERY AND LINKING EVALUATION RESULTS (SECOND EDL WINDOW) ================================================================================ Team ID: UI_CCG Organization: University of Illinois at Urbana-Champaign ************************************************************* Run ID: UI_CCG1 Did the run access the live Web during the evaluation window: Yes Did the run use the Wikipedia links in the Reference KB: Yes Did the run use the relation/slot information encoded in the Reference KB: No Did the run use the text descriptions in the nodes of the Reference KB: No Did the run use any of the distributed EDL1 runs (from the first evaluation window): No ALL English, Chinese, and Spanish documents: Prec Recall F1 Metric 0.685 0.421 0.521 strong_typed_mention_match 0.569 0.350 0.433 strong_typed_all_match 0.640 0.339 0.443 strong_typed_link_match 0.425 0.387 0.405 strong_typed_nil_match 0.835 0.385 0.527 entity_match 0.609 0.374 0.464 mention_ceaf 0.587 0.360 0.446 typed_mention_ceaf 0.560 0.344 0.426 typed_mention_ceaf_plus 0.520 0.258 0.345 b_cubed 0.448 0.232 0.306 b_cubed_plus ONLY English documents: Prec Recall F1 Metric 0.000 0.000 0.000 strong_mention_match 0.000 0.000 0.000 strong_typed_mention_match 0.000 0.000 0.000 strong_typed_all_match 0.000 0.000 0.000 strong_typed_link_match 0.000 0.000 0.000 strong_typed_nil_match 0.000 0.000 0.000 entity_match 0.000 0.000 0.000 mention_ceaf 0.000 0.000 0.000 typed_mention_ceaf 0.000 0.000 0.000 typed_mention_ceaf_plus 0.000 0.000 0.000 b_cubed 0.000 0.000 0.000 b_cubed_plus ONLY Chinese documents: Prec Recall F1 Metric 0.719 0.686 0.702 strong_mention_match 0.674 0.643 0.658 strong_typed_mention_match 0.562 0.536 0.549 strong_typed_all_match 0.643 0.531 0.582 strong_typed_link_match 0.376 0.556 0.449 strong_typed_nil_match 0.810 0.593 0.685 entity_match 0.609 0.581 0.595 mention_ceaf 0.582 0.555 0.568 typed_mention_ceaf 0.555 0.529 0.542 typed_mention_ceaf_plus 0.548 0.488 0.516 b_cubed 0.461 0.441 0.451 b_cubed_plus ONLY Spanish documents: Prec Recall F1 Metric 0.729 0.727 0.728 strong_mention_match 0.698 0.696 0.697 strong_typed_mention_match 0.577 0.576 0.576 strong_typed_all_match 0.635 0.547 0.588 strong_typed_link_match 0.475 0.656 0.551 strong_typed_nil_match 0.859 0.637 0.731 entity_match 0.618 0.616 0.617 mention_ceaf 0.600 0.598 0.599 typed_mention_ceaf 0.567 0.566 0.566 typed_mention_ceaf_plus 0.499 0.531 0.514 b_cubed 0.440 0.478 0.458 b_cubed_plus ************************************************************* Run ID: UI_CCG2 Did the run access the live Web during the evaluation window: Yes Did the run use the Wikipedia links in the Reference KB: Yes Did the run use the relation/slot information encoded in the Reference KB: No Did the run use the text descriptions in the nodes of the Reference KB: No Did the run use any of the distributed EDL1 runs (from the first evaluation window): No ALL English, Chinese, and Spanish documents: Prec Recall F1 Metric 0.659 0.430 0.520 strong_typed_mention_match 0.540 0.353 0.427 strong_typed_all_match 0.652 0.338 0.445 strong_typed_link_match 0.359 0.402 0.380 strong_typed_nil_match 0.835 0.385 0.527 entity_match 0.583 0.380 0.460 mention_ceaf 0.560 0.366 0.442 typed_mention_ceaf 0.532 0.347 0.420 typed_mention_ceaf_plus 0.512 0.262 0.347 b_cubed 0.431 0.235 0.304 b_cubed_plus ONLY English documents: Prec Recall F1 Metric 0.000 0.000 0.000 strong_mention_match 0.000 0.000 0.000 strong_typed_mention_match 0.000 0.000 0.000 strong_typed_all_match 0.000 0.000 0.000 strong_typed_link_match 0.000 0.000 0.000 strong_typed_nil_match 0.000 0.000 0.000 entity_match 0.000 0.000 0.000 mention_ceaf 0.000 0.000 0.000 typed_mention_ceaf 0.000 0.000 0.000 typed_mention_ceaf_plus 0.000 0.000 0.000 b_cubed 0.000 0.000 0.000 b_cubed_plus ONLY Chinese documents: Prec Recall F1 Metric 0.695 0.700 0.698 strong_mention_match 0.650 0.655 0.652 strong_typed_mention_match 0.534 0.538 0.536 strong_typed_all_match 0.658 0.531 0.587 strong_typed_link_match 0.310 0.569 0.401 strong_typed_nil_match 0.810 0.593 0.685 entity_match 0.586 0.591 0.588 mention_ceaf 0.559 0.563 0.561 typed_mention_ceaf 0.528 0.532 0.530 typed_mention_ceaf_plus 0.537 0.494 0.515 b_cubed 0.444 0.442 0.443 b_cubed_plus ONLY Spanish documents: Prec Recall F1 Metric 0.704 0.750 0.726 strong_mention_match 0.670 0.715 0.692 strong_typed_mention_match 0.548 0.584 0.565 strong_typed_all_match 0.645 0.546 0.591 strong_typed_link_match 0.411 0.691 0.515 strong_typed_nil_match 0.859 0.637 0.731 entity_match 0.589 0.628 0.608 mention_ceaf 0.570 0.608 0.588 typed_mention_ceaf 0.536 0.572 0.554 typed_mention_ceaf_plus 0.492 0.541 0.515 b_cubed 0.424 0.485 0.452 b_cubed_plus ************************************************************* Run ID: UI_CCG3 Did the run access the live Web during the evaluation window: Yes Did the run use the Wikipedia links in the Reference KB: Yes Did the run use the relation/slot information encoded in the Reference KB: No Did the run use the text descriptions in the nodes of the Reference KB: No Did the run use any of the distributed EDL1 runs (from the first evaluation window): No ALL English, Chinese, and Spanish documents: Prec Recall F1 Metric 0.757 0.404 0.527 strong_typed_mention_match 0.648 0.346 0.451 strong_typed_all_match 0.706 0.338 0.457 strong_typed_link_match 0.514 0.373 0.432 strong_typed_nil_match 0.835 0.385 0.527 entity_match 0.695 0.371 0.484 mention_ceaf 0.670 0.357 0.466 typed_mention_ceaf 0.641 0.342 0.446 typed_mention_ceaf_plus 0.632 0.249 0.357 b_cubed 0.549 0.228 0.322 b_cubed_plus ONLY English documents: Prec Recall F1 Metric 0.000 0.000 0.000 strong_mention_match 0.000 0.000 0.000 strong_typed_mention_match 0.000 0.000 0.000 strong_typed_all_match 0.000 0.000 0.000 strong_typed_link_match 0.000 0.000 0.000 strong_typed_nil_match 0.000 0.000 0.000 entity_match 0.000 0.000 0.000 mention_ceaf 0.000 0.000 0.000 typed_mention_ceaf 0.000 0.000 0.000 typed_mention_ceaf_plus 0.000 0.000 0.000 b_cubed 0.000 0.000 0.000 b_cubed_plus ONLY Chinese documents: Prec Recall F1 Metric 0.809 0.655 0.724 strong_mention_match 0.763 0.617 0.682 strong_typed_mention_match 0.652 0.528 0.583 strong_typed_all_match 0.721 0.528 0.609 strong_typed_link_match 0.468 0.528 0.496 strong_typed_nil_match 0.810 0.593 0.685 entity_match 0.708 0.573 0.634 mention_ceaf 0.676 0.547 0.605 typed_mention_ceaf 0.647 0.524 0.579 typed_mention_ceaf_plus 0.681 0.469 0.556 b_cubed 0.578 0.431 0.494 b_cubed_plus ONLY Spanish documents: Prec Recall F1 Metric 0.783 0.696 0.737 strong_mention_match 0.751 0.668 0.707 strong_typed_mention_match 0.643 0.572 0.606 strong_typed_all_match 0.688 0.548 0.610 strong_typed_link_match 0.556 0.638 0.594 strong_typed_nil_match 0.859 0.637 0.731 entity_match 0.690 0.614 0.650 mention_ceaf 0.670 0.595 0.630 typed_mention_ceaf 0.634 0.564 0.597 typed_mention_ceaf_plus 0.586 0.516 0.549 b_cubed 0.524 0.473 0.497 b_cubed_plus ************************************************************* Run ID: UI_CCG4 Did the run access the live Web during the evaluation window: Yes Did the run use the Wikipedia links in the Reference KB: Yes Did the run use the relation/slot information encoded in the Reference KB: No Did the run use the text descriptions in the nodes of the Reference KB: No Did the run use any of the distributed EDL1 runs (from the first evaluation window): No ALL English, Chinese, and Spanish documents: Prec Recall F1 Metric 0.797 0.388 0.522 strong_typed_mention_match 0.698 0.340 0.457 strong_typed_all_match 0.751 0.334 0.462 strong_typed_link_match 0.568 0.361 0.441 strong_typed_nil_match 0.835 0.385 0.527 entity_match 0.751 0.366 0.492 mention_ceaf 0.723 0.352 0.473 typed_mention_ceaf 0.693 0.338 0.454 typed_mention_ceaf_plus 0.718 0.242 0.362 b_cubed 0.627 0.223 0.329 b_cubed_plus ONLY English documents: Prec Recall F1 Metric 0.000 0.000 0.000 strong_mention_match 0.000 0.000 0.000 strong_typed_mention_match 0.000 0.000 0.000 strong_typed_all_match 0.000 0.000 0.000 strong_typed_link_match 0.000 0.000 0.000 strong_typed_nil_match 0.000 0.000 0.000 entity_match 0.000 0.000 0.000 mention_ceaf 0.000 0.000 0.000 typed_mention_ceaf 0.000 0.000 0.000 typed_mention_ceaf_plus 0.000 0.000 0.000 b_cubed 0.000 0.000 0.000 b_cubed_plus ONLY Chinese documents: Prec Recall F1 Metric 0.840 0.640 0.726 strong_mention_match 0.793 0.604 0.685 strong_typed_mention_match 0.688 0.524 0.595 strong_typed_all_match 0.760 0.524 0.620 strong_typed_link_match 0.495 0.524 0.509 strong_typed_nil_match 0.810 0.593 0.685 entity_match 0.747 0.569 0.646 mention_ceaf 0.713 0.543 0.616 typed_mention_ceaf 0.684 0.521 0.591 typed_mention_ceaf_plus 0.746 0.462 0.571 b_cubed 0.635 0.427 0.510 b_cubed_plus ONLY Spanish documents: Prec Recall F1 Metric 0.836 0.655 0.734 strong_mention_match 0.803 0.629 0.705 strong_typed_mention_match 0.710 0.556 0.624 strong_typed_all_match 0.738 0.539 0.623 strong_typed_link_match 0.646 0.605 0.625 strong_typed_nil_match 0.859 0.637 0.731 entity_match 0.768 0.602 0.675 mention_ceaf 0.744 0.583 0.654 typed_mention_ceaf 0.705 0.553 0.619 typed_mention_ceaf_plus 0.694 0.494 0.577 b_cubed 0.625 0.458 0.529 b_cubed_plus ************************************************************* Run ID: UI_CCG5 Did the run access the live Web during the evaluation window: Yes Did the run use the Wikipedia links in the Reference KB: Yes Did the run use the relation/slot information encoded in the Reference KB: No Did the run use the text descriptions in the nodes of the Reference KB: No Did the run use any of the distributed EDL1 runs (from the first evaluation window): No ALL English, Chinese, and Spanish documents: Prec Recall F1 Metric 0.679 0.425 0.523 strong_typed_mention_match 0.564 0.353 0.435 strong_typed_all_match 0.643 0.338 0.443 strong_typed_link_match 0.416 0.406 0.411 strong_typed_nil_match 0.835 0.385 0.527 entity_match 0.601 0.376 0.463 mention_ceaf 0.579 0.362 0.446 typed_mention_ceaf 0.553 0.347 0.426 typed_mention_ceaf_plus 0.517 0.260 0.346 b_cubed 0.444 0.235 0.307 b_cubed_plus ONLY English documents: Prec Recall F1 Metric 0.000 0.000 0.000 strong_mention_match 0.000 0.000 0.000 strong_typed_mention_match 0.000 0.000 0.000 strong_typed_all_match 0.000 0.000 0.000 strong_typed_link_match 0.000 0.000 0.000 strong_typed_nil_match 0.000 0.000 0.000 entity_match 0.000 0.000 0.000 mention_ceaf 0.000 0.000 0.000 typed_mention_ceaf 0.000 0.000 0.000 typed_mention_ceaf_plus 0.000 0.000 0.000 b_cubed 0.000 0.000 0.000 b_cubed_plus ONLY Chinese documents: Prec Recall F1 Metric 0.715 0.690 0.702 strong_mention_match 0.670 0.647 0.658 strong_typed_mention_match 0.559 0.540 0.549 strong_typed_all_match 0.646 0.531 0.583 strong_typed_link_match 0.370 0.577 0.451 strong_typed_nil_match 0.810 0.593 0.685 entity_match 0.603 0.582 0.592 mention_ceaf 0.575 0.556 0.565 typed_mention_ceaf 0.550 0.531 0.540 typed_mention_ceaf_plus 0.546 0.488 0.516 b_cubed 0.460 0.442 0.450 b_cubed_plus ONLY Spanish documents: Prec Recall F1 Metric 0.722 0.741 0.732 strong_mention_match 0.691 0.708 0.699 strong_typed_mention_match 0.571 0.585 0.578 strong_typed_all_match 0.638 0.546 0.589 strong_typed_link_match 0.463 0.696 0.556 strong_typed_nil_match 0.859 0.637 0.731 entity_match 0.608 0.623 0.615 mention_ceaf 0.590 0.605 0.597 typed_mention_ceaf 0.558 0.572 0.565 typed_mention_ceaf_plus 0.494 0.539 0.515 b_cubed 0.434 0.485 0.458 b_cubed_plus