================================================================================ TAC KBP 2016 ENTITY DISCOVERY AND LINKING EVALUATION RESULTS (SECOND EDL WINDOW) ================================================================================ Team ID: UTAustin Organization: University of Texas at Austin ************************************************************* Run ID: UTAustin1 Did the run access the live Web during the evaluation window: No Did the run use the Wikipedia links in the Reference KB: No Did the run use the relation/slot information encoded in the Reference KB: No Did the run use the text descriptions in the nodes of the Reference KB: No Did the run use any of the distributed EDL1 runs (from the first evaluation window): Yes ALL English, Chinese, and Spanish documents: Prec Recall F1 Metric 0.688 0.390 0.498 strong_typed_mention_match 0.540 0.306 0.391 strong_typed_all_match 0.681 0.260 0.377 strong_typed_link_match 0.385 0.467 0.422 strong_typed_nil_match 0.610 0.393 0.478 entity_match 0.597 0.339 0.432 mention_ceaf 0.569 0.323 0.412 typed_mention_ceaf 0.529 0.300 0.383 typed_mention_ceaf_plus 0.709 0.205 0.319 b_cubed 0.538 0.184 0.274 b_cubed_plus ONLY English documents: Prec Recall F1 Metric 0.628 0.286 0.393 strong_mention_match 0.540 0.246 0.338 strong_typed_mention_match 0.261 0.119 0.164 strong_typed_all_match 0.363 0.114 0.174 strong_typed_link_match 0.141 0.135 0.138 strong_typed_nil_match 0.383 0.292 0.331 entity_match 0.364 0.166 0.228 mention_ceaf 0.328 0.150 0.206 typed_mention_ceaf 0.258 0.118 0.162 typed_mention_ceaf_plus 0.580 0.075 0.133 b_cubed 0.255 0.046 0.078 b_cubed_plus ONLY Chinese documents: Prec Recall F1 Metric 0.774 0.502 0.609 strong_mention_match 0.743 0.482 0.585 strong_typed_mention_match 0.669 0.435 0.527 strong_typed_all_match 0.856 0.379 0.525 strong_typed_link_match 0.443 0.665 0.532 strong_typed_nil_match 0.836 0.461 0.594 entity_match 0.706 0.460 0.557 mention_ceaf 0.683 0.445 0.539 typed_mention_ceaf 0.650 0.422 0.512 typed_mention_ceaf_plus 0.748 0.330 0.458 b_cubed 0.665 0.308 0.421 b_cubed_plus ONLY Spanish documents: Prec Recall F1 Metric 0.812 0.494 0.614 strong_mention_match 0.759 0.462 0.575 strong_typed_mention_match 0.643 0.392 0.487 strong_typed_all_match 0.766 0.301 0.432 strong_typed_link_match 0.531 0.647 0.584 strong_typed_nil_match 0.746 0.446 0.558 entity_match 0.721 0.439 0.545 mention_ceaf 0.686 0.418 0.519 typed_mention_ceaf 0.634 0.386 0.480 typed_mention_ceaf_plus 0.798 0.313 0.449 b_cubed 0.655 0.281 0.393 b_cubed_plus ************************************************************* Run ID: UTAustin2 Did the run access the live Web during the evaluation window: No Did the run use the Wikipedia links in the Reference KB: No Did the run use the relation/slot information encoded in the Reference KB: No Did the run use the text descriptions in the nodes of the Reference KB: No Did the run use any of the distributed EDL1 runs (from the first evaluation window): Yes ALL English, Chinese, and Spanish documents: Prec Recall F1 Metric 0.653 0.506 0.570 strong_typed_mention_match 0.499 0.386 0.436 strong_typed_all_match 0.627 0.349 0.448 strong_typed_link_match 0.337 0.517 0.408 strong_typed_nil_match 0.545 0.494 0.518 entity_match 0.551 0.428 0.482 mention_ceaf 0.524 0.406 0.458 typed_mention_ceaf 0.489 0.379 0.427 typed_mention_ceaf_plus 0.666 0.279 0.394 b_cubed 0.491 0.247 0.329 b_cubed_plus ONLY English documents: Prec Recall F1 Metric 0.607 0.528 0.565 strong_mention_match 0.523 0.455 0.486 strong_typed_mention_match 0.279 0.243 0.260 strong_typed_all_match 0.378 0.236 0.291 strong_typed_link_match 0.153 0.267 0.195 strong_typed_nil_match 0.356 0.455 0.399 entity_match 0.358 0.311 0.333 mention_ceaf 0.325 0.283 0.302 typed_mention_ceaf 0.276 0.240 0.257 typed_mention_ceaf_plus 0.550 0.177 0.268 b_cubed 0.273 0.120 0.167 b_cubed_plus ONLY Chinese documents: Prec Recall F1 Metric 0.777 0.587 0.669 strong_mention_match 0.748 0.566 0.644 strong_typed_mention_match 0.676 0.512 0.582 strong_typed_all_match 0.847 0.472 0.606 strong_typed_link_match 0.427 0.675 0.523 strong_typed_nil_match 0.824 0.545 0.656 entity_match 0.706 0.538 0.611 mention_ceaf 0.686 0.521 0.592 typed_mention_ceaf 0.658 0.498 0.567 typed_mention_ceaf_plus 0.741 0.411 0.529 b_cubed 0.662 0.388 0.489 b_cubed_plus ONLY Spanish documents: Prec Recall F1 Metric 0.797 0.534 0.639 strong_mention_match 0.742 0.497 0.595 strong_typed_mention_match 0.623 0.417 0.500 strong_typed_all_match 0.731 0.336 0.461 strong_typed_link_match 0.512 0.645 0.571 strong_typed_nil_match 0.693 0.489 0.574 entity_match 0.702 0.470 0.563 mention_ceaf 0.668 0.447 0.536 typed_mention_ceaf 0.616 0.412 0.494 typed_mention_ceaf_plus 0.779 0.339 0.472 b_cubed 0.634 0.303 0.410 b_cubed_plus ************************************************************* Run ID: UTAustin3 Did the run access the live Web during the evaluation window: No Did the run use the Wikipedia links in the Reference KB: No Did the run use the relation/slot information encoded in the Reference KB: No Did the run use the text descriptions in the nodes of the Reference KB: No Did the run use any of the distributed EDL1 runs (from the first evaluation window): Yes ALL English, Chinese, and Spanish documents: Prec Recall F1 Metric 0.840 0.512 0.636 strong_typed_mention_match 0.746 0.454 0.565 strong_typed_all_match 0.789 0.409 0.539 strong_typed_link_match 0.660 0.610 0.634 strong_typed_nil_match 0.776 0.519 0.622 entity_match 0.789 0.481 0.597 mention_ceaf 0.760 0.463 0.576 typed_mention_ceaf 0.735 0.448 0.557 typed_mention_ceaf_plus 0.834 0.342 0.485 b_cubed 0.741 0.323 0.450 b_cubed_plus ONLY English documents: Prec Recall F1 Metric 0.916 0.572 0.704 strong_mention_match 0.872 0.544 0.670 strong_typed_mention_match 0.732 0.457 0.563 strong_typed_all_match 0.751 0.412 0.533 strong_typed_link_match 0.689 0.616 0.650 strong_typed_nil_match 0.725 0.547 0.624 entity_match 0.792 0.495 0.609 mention_ceaf 0.757 0.472 0.582 typed_mention_ceaf 0.717 0.448 0.551 typed_mention_ceaf_plus 0.867 0.372 0.521 b_cubed 0.739 0.340 0.466 b_cubed_plus ONLY Chinese documents: Prec Recall F1 Metric 0.843 0.532 0.652 strong_mention_match 0.818 0.516 0.633 strong_typed_mention_match 0.771 0.486 0.596 strong_typed_all_match 0.829 0.463 0.594 strong_typed_link_match 0.626 0.581 0.603 strong_typed_nil_match 0.825 0.523 0.640 entity_match 0.805 0.508 0.623 mention_ceaf 0.783 0.494 0.606 typed_mention_ceaf 0.764 0.482 0.591 typed_mention_ceaf_plus 0.807 0.380 0.517 b_cubed 0.755 0.368 0.495 b_cubed_plus ONLY Spanish documents: Prec Recall F1 Metric 0.866 0.487 0.623 strong_mention_match 0.824 0.463 0.592 strong_typed_mention_match 0.730 0.410 0.525 strong_typed_all_match 0.786 0.331 0.466 strong_typed_link_match 0.662 0.633 0.647 strong_typed_nil_match 0.801 0.480 0.601 entity_match 0.790 0.443 0.568 mention_ceaf 0.759 0.426 0.546 typed_mention_ceaf 0.723 0.406 0.520 typed_mention_ceaf_plus 0.839 0.319 0.463 b_cubed 0.734 0.298 0.424 b_cubed_plus ************************************************************* Run ID: UTAustin4 Did the run access the live Web during the evaluation window: No Did the run use the Wikipedia links in the Reference KB: No Did the run use the relation/slot information encoded in the Reference KB: No Did the run use the text descriptions in the nodes of the Reference KB: No Did the run use any of the distributed EDL1 runs (from the first evaluation window): Yes ALL English, Chinese, and Spanish documents: Prec Recall F1 Metric 0.775 0.467 0.582 strong_typed_mention_match 0.678 0.408 0.510 strong_typed_all_match 0.768 0.349 0.480 strong_typed_link_match 0.550 0.615 0.581 strong_typed_nil_match 0.771 0.470 0.584 entity_match 0.727 0.438 0.546 mention_ceaf 0.693 0.418 0.521 typed_mention_ceaf 0.666 0.401 0.501 typed_mention_ceaf_plus 0.797 0.299 0.435 b_cubed 0.679 0.280 0.396 b_cubed_plus ONLY English documents: Prec Recall F1 Metric 0.808 0.517 0.631 strong_mention_match 0.720 0.461 0.562 strong_typed_mention_match 0.573 0.367 0.447 strong_typed_all_match 0.748 0.287 0.415 strong_typed_link_match 0.419 0.649 0.509 strong_typed_nil_match 0.745 0.423 0.540 entity_match 0.639 0.409 0.499 mention_ceaf 0.592 0.379 0.462 typed_mention_ceaf 0.554 0.354 0.432 typed_mention_ceaf_plus 0.782 0.304 0.438 b_cubed 0.593 0.271 0.372 b_cubed_plus ONLY Chinese documents: Prec Recall F1 Metric 0.842 0.472 0.605 strong_mention_match 0.812 0.456 0.584 strong_typed_mention_match 0.764 0.429 0.549 strong_typed_all_match 0.786 0.397 0.527 strong_typed_link_match 0.708 0.560 0.626 strong_typed_nil_match 0.801 0.492 0.610 entity_match 0.808 0.453 0.581 mention_ceaf 0.783 0.439 0.563 typed_mention_ceaf 0.758 0.425 0.545 typed_mention_ceaf_plus 0.803 0.318 0.456 b_cubed 0.750 0.306 0.435 b_cubed_plus ONLY Spanish documents: Prec Recall F1 Metric 0.854 0.517 0.644 strong_mention_match 0.807 0.489 0.609 strong_typed_mention_match 0.723 0.438 0.545 strong_typed_all_match 0.766 0.370 0.499 strong_typed_link_match 0.662 0.629 0.645 strong_typed_nil_match 0.770 0.505 0.610 entity_match 0.785 0.475 0.592 mention_ceaf 0.754 0.456 0.569 typed_mention_ceaf 0.717 0.434 0.541 typed_mention_ceaf_plus 0.820 0.346 0.487 b_cubed 0.722 0.323 0.446 b_cubed_plus ************************************************************* Run ID: UTAustin5 Did the run access the live Web during the evaluation window: No Did the run use the Wikipedia links in the Reference KB: No Did the run use the relation/slot information encoded in the Reference KB: No Did the run use the text descriptions in the nodes of the Reference KB: No Did the run use any of the distributed EDL1 runs (from the first evaluation window): Yes ALL English, Chinese, and Spanish documents: Prec Recall F1 Metric 0.862 0.064 0.118 strong_typed_mention_match 0.821 0.061 0.113 strong_typed_all_match 0.491 0.001 0.003 strong_typed_link_match 0.832 0.267 0.404 strong_typed_nil_match 0.614 0.006 0.011 entity_match 0.848 0.063 0.116 mention_ceaf 0.841 0.062 0.115 typed_mention_ceaf 0.809 0.060 0.111 typed_mention_ceaf_plus 0.857 0.055 0.103 b_cubed 0.811 0.054 0.102 b_cubed_plus ONLY English documents: Prec Recall F1 Metric 0.660 0.007 0.015 strong_mention_match 0.583 0.007 0.013 strong_typed_mention_match 0.252 0.003 0.006 strong_typed_all_match 0.404 0.003 0.005 strong_typed_link_match 0.125 0.003 0.007 strong_typed_nil_match 0.553 0.011 0.022 entity_match 0.563 0.006 0.012 mention_ceaf 0.505 0.006 0.011 typed_mention_ceaf 0.252 0.003 0.006 typed_mention_ceaf_plus 0.655 0.002 0.003 b_cubed 0.257 0.001 0.002 b_cubed_plus ONLY Chinese documents: Prec Recall F1 Metric 0.815 0.086 0.156 strong_mention_match 0.814 0.086 0.156 strong_typed_mention_match 0.810 0.086 0.155 strong_typed_all_match 1.000 0.000 0.001 strong_typed_link_match 0.809 0.436 0.566 strong_typed_nil_match 1.000 0.002 0.003 entity_match 0.798 0.084 0.153 mention_ceaf 0.797 0.084 0.152 typed_mention_ceaf 0.793 0.084 0.152 typed_mention_ceaf_plus 0.792 0.080 0.146 b_cubed 0.788 0.080 0.146 b_cubed_plus ONLY Spanish documents: Prec Recall F1 Metric 0.963 0.112 0.200 strong_mention_match 0.953 0.111 0.198 strong_typed_mention_match 0.907 0.105 0.189 strong_typed_all_match 0.857 0.001 0.002 strong_typed_link_match 0.908 0.399 0.554 strong_typed_nil_match 0.857 0.003 0.006 entity_match 0.947 0.110 0.197 mention_ceaf 0.938 0.109 0.195 typed_mention_ceaf 0.900 0.104 0.187 typed_mention_ceaf_plus 0.958 0.095 0.173 b_cubed 0.907 0.093 0.168 b_cubed_plus