================================================================================ TAC KBP 2016 ENTITY DISCOVERY AND LINKING EVALUATION RESULTS (SECOND EDL WINDOW) ================================================================================ Team ID: hltcoe Organization: Human Language Technology Center of Excellence ************************************************************* Run ID: hltcoe1 Did the run access the live Web during the evaluation window: No Did the run use the Wikipedia links in the Reference KB: No Did the run use the relation/slot information encoded in the Reference KB: No Did the run use the text descriptions in the nodes of the Reference KB: No Did the run use any of the distributed EDL1 runs (from the first evaluation window): No ALL English, Chinese, and Spanish documents: Prec Recall F1 Metric 0.656 0.573 0.612 strong_typed_mention_match 0.476 0.416 0.444 strong_typed_all_match 0.562 0.363 0.441 strong_typed_link_match 0.360 0.600 0.450 strong_typed_nil_match 0.678 0.419 0.518 entity_match 0.562 0.491 0.525 mention_ceaf 0.544 0.475 0.507 typed_mention_ceaf 0.458 0.400 0.427 typed_mention_ceaf_plus 0.528 0.379 0.441 b_cubed 0.391 0.305 0.343 b_cubed_plus ONLY English documents: Prec Recall F1 Metric 0.680 0.668 0.674 strong_mention_match 0.641 0.629 0.635 strong_typed_mention_match 0.483 0.474 0.479 strong_typed_all_match 0.556 0.442 0.493 strong_typed_link_match 0.357 0.588 0.445 strong_typed_nil_match 0.736 0.505 0.599 entity_match 0.584 0.573 0.579 mention_ceaf 0.564 0.554 0.559 typed_mention_ceaf 0.466 0.458 0.462 typed_mention_ceaf_plus 0.502 0.472 0.487 b_cubed 0.382 0.376 0.379 b_cubed_plus ONLY Chinese documents: Prec Recall F1 Metric 0.693 0.636 0.663 strong_mention_match 0.649 0.596 0.622 strong_typed_mention_match 0.446 0.409 0.427 strong_typed_all_match 0.596 0.362 0.450 strong_typed_link_match 0.275 0.605 0.378 strong_typed_nil_match 0.642 0.394 0.489 entity_match 0.580 0.532 0.555 mention_ceaf 0.554 0.509 0.530 typed_mention_ceaf 0.437 0.402 0.419 typed_mention_ceaf_plus 0.555 0.426 0.482 b_cubed 0.390 0.320 0.351 b_cubed_plus ONLY Spanish documents: Prec Recall F1 Metric 0.754 0.509 0.608 strong_mention_match 0.698 0.471 0.562 strong_typed_mention_match 0.515 0.348 0.415 strong_typed_all_match 0.517 0.255 0.341 strong_typed_link_match 0.514 0.609 0.557 strong_typed_nil_match 0.631 0.340 0.442 entity_match 0.607 0.409 0.489 mention_ceaf 0.574 0.387 0.462 typed_mention_ceaf 0.478 0.322 0.385 typed_mention_ceaf_plus 0.579 0.319 0.411 b_cubed 0.433 0.255 0.321 b_cubed_plus ************************************************************* Run ID: hltcoe2 Did the run access the live Web during the evaluation window: No Did the run use the Wikipedia links in the Reference KB: No Did the run use the relation/slot information encoded in the Reference KB: No Did the run use the text descriptions in the nodes of the Reference KB: No Did the run use any of the distributed EDL1 runs (from the first evaluation window): No ALL English, Chinese, and Spanish documents: Prec Recall F1 Metric 0.656 0.573 0.612 strong_typed_mention_match 0.480 0.419 0.447 strong_typed_all_match 0.617 0.365 0.458 strong_typed_link_match 0.327 0.609 0.426 strong_typed_nil_match 0.736 0.409 0.526 entity_match 0.569 0.497 0.531 mention_ceaf 0.553 0.483 0.516 typed_mention_ceaf 0.461 0.403 0.430 typed_mention_ceaf_plus 0.566 0.376 0.451 b_cubed 0.402 0.304 0.346 b_cubed_plus ONLY English documents: Prec Recall F1 Metric 0.680 0.668 0.674 strong_mention_match 0.641 0.629 0.635 strong_typed_mention_match 0.491 0.482 0.486 strong_typed_all_match 0.575 0.451 0.506 strong_typed_link_match 0.350 0.589 0.439 strong_typed_nil_match 0.760 0.514 0.613 entity_match 0.599 0.588 0.593 mention_ceaf 0.579 0.569 0.574 typed_mention_ceaf 0.474 0.465 0.469 typed_mention_ceaf_plus 0.514 0.485 0.499 b_cubed 0.392 0.388 0.390 b_cubed_plus ONLY Chinese documents: Prec Recall F1 Metric 0.693 0.636 0.663 strong_mention_match 0.649 0.596 0.622 strong_typed_mention_match 0.433 0.397 0.414 strong_typed_all_match 0.710 0.340 0.460 strong_typed_link_match 0.232 0.631 0.339 strong_typed_nil_match 0.758 0.339 0.468 entity_match 0.607 0.557 0.581 mention_ceaf 0.585 0.537 0.560 typed_mention_ceaf 0.423 0.388 0.405 typed_mention_ceaf_plus 0.605 0.435 0.506 b_cubed 0.385 0.313 0.346 b_cubed_plus ONLY Spanish documents: Prec Recall F1 Metric 0.754 0.509 0.608 strong_mention_match 0.698 0.471 0.562 strong_typed_mention_match 0.540 0.364 0.435 strong_typed_all_match 0.585 0.276 0.375 strong_typed_link_match 0.491 0.610 0.544 strong_typed_nil_match 0.680 0.352 0.464 entity_match 0.640 0.431 0.515 mention_ceaf 0.609 0.410 0.490 typed_mention_ceaf 0.502 0.339 0.404 typed_mention_ceaf_plus 0.621 0.323 0.425 b_cubed 0.465 0.262 0.335 b_cubed_plus ************************************************************* Run ID: hltcoe3 Did the run access the live Web during the evaluation window: No Did the run use the Wikipedia links in the Reference KB: No Did the run use the relation/slot information encoded in the Reference KB: No Did the run use the text descriptions in the nodes of the Reference KB: No Did the run use any of the distributed EDL1 runs (from the first evaluation window): No ALL English, Chinese, and Spanish documents: Prec Recall F1 Metric 0.656 0.573 0.612 strong_typed_mention_match 0.464 0.405 0.433 strong_typed_all_match 0.567 0.349 0.432 strong_typed_link_match 0.339 0.601 0.433 strong_typed_nil_match 0.684 0.409 0.512 entity_match 0.552 0.482 0.514 mention_ceaf 0.533 0.466 0.497 typed_mention_ceaf 0.445 0.389 0.415 typed_mention_ceaf_plus 0.528 0.370 0.435 b_cubed 0.379 0.293 0.330 b_cubed_plus ONLY English documents: Prec Recall F1 Metric 0.680 0.668 0.674 strong_mention_match 0.641 0.629 0.635 strong_typed_mention_match 0.497 0.488 0.492 strong_typed_all_match 0.587 0.459 0.516 strong_typed_link_match 0.348 0.589 0.437 strong_typed_nil_match 0.754 0.509 0.608 entity_match 0.577 0.567 0.572 mention_ceaf 0.557 0.547 0.552 typed_mention_ceaf 0.480 0.471 0.475 typed_mention_ceaf_plus 0.501 0.465 0.483 b_cubed 0.390 0.381 0.385 b_cubed_plus ONLY Chinese documents: Prec Recall F1 Metric 0.693 0.636 0.663 strong_mention_match 0.649 0.596 0.622 strong_typed_mention_match 0.404 0.371 0.387 strong_typed_all_match 0.572 0.313 0.405 strong_typed_link_match 0.249 0.608 0.353 strong_typed_nil_match 0.635 0.367 0.465 entity_match 0.576 0.529 0.551 mention_ceaf 0.550 0.505 0.527 typed_mention_ceaf 0.395 0.363 0.378 typed_mention_ceaf_plus 0.556 0.421 0.479 b_cubed 0.354 0.292 0.320 b_cubed_plus ONLY Spanish documents: Prec Recall F1 Metric 0.754 0.509 0.608 strong_mention_match 0.698 0.471 0.562 strong_typed_mention_match 0.504 0.340 0.406 strong_typed_all_match 0.512 0.244 0.331 strong_typed_link_match 0.494 0.609 0.546 strong_typed_nil_match 0.632 0.332 0.435 entity_match 0.595 0.401 0.479 mention_ceaf 0.561 0.378 0.452 typed_mention_ceaf 0.466 0.314 0.376 typed_mention_ceaf_plus 0.579 0.313 0.406 b_cubed 0.422 0.247 0.311 b_cubed_plus ************************************************************* Run ID: hltcoe4 Did the run access the live Web during the evaluation window: No Did the run use the Wikipedia links in the Reference KB: No Did the run use the relation/slot information encoded in the Reference KB: No Did the run use the text descriptions in the nodes of the Reference KB: No Did the run use any of the distributed EDL1 runs (from the first evaluation window): No ALL English, Chinese, and Spanish documents: Prec Recall F1 Metric 0.656 0.573 0.612 strong_typed_mention_match 0.480 0.420 0.448 strong_typed_all_match 0.619 0.365 0.459 strong_typed_link_match 0.327 0.609 0.426 strong_typed_nil_match 0.739 0.409 0.527 entity_match 0.569 0.497 0.531 mention_ceaf 0.553 0.483 0.516 typed_mention_ceaf 0.462 0.403 0.430 typed_mention_ceaf_plus 0.565 0.376 0.451 b_cubed 0.402 0.305 0.347 b_cubed_plus ONLY English documents: Prec Recall F1 Metric 0.680 0.668 0.674 strong_mention_match 0.641 0.629 0.635 strong_typed_mention_match 0.492 0.483 0.488 strong_typed_all_match 0.577 0.453 0.507 strong_typed_link_match 0.351 0.591 0.441 strong_typed_nil_match 0.763 0.516 0.615 entity_match 0.598 0.588 0.593 mention_ceaf 0.579 0.568 0.574 typed_mention_ceaf 0.475 0.466 0.471 typed_mention_ceaf_plus 0.514 0.485 0.499 b_cubed 0.393 0.388 0.391 b_cubed_plus ONLY Chinese documents: Prec Recall F1 Metric 0.693 0.636 0.663 strong_mention_match 0.649 0.596 0.622 strong_typed_mention_match 0.433 0.397 0.414 strong_typed_all_match 0.710 0.340 0.460 strong_typed_link_match 0.232 0.631 0.339 strong_typed_nil_match 0.758 0.339 0.468 entity_match 0.607 0.557 0.581 mention_ceaf 0.585 0.537 0.560 typed_mention_ceaf 0.423 0.389 0.405 typed_mention_ceaf_plus 0.605 0.435 0.506 b_cubed 0.385 0.314 0.346 b_cubed_plus ONLY Spanish documents: Prec Recall F1 Metric 0.754 0.509 0.608 strong_mention_match 0.698 0.471 0.562 strong_typed_mention_match 0.540 0.364 0.435 strong_typed_all_match 0.590 0.277 0.377 strong_typed_link_match 0.486 0.610 0.541 strong_typed_nil_match 0.684 0.353 0.465 entity_match 0.638 0.430 0.514 mention_ceaf 0.608 0.410 0.490 typed_mention_ceaf 0.502 0.339 0.405 typed_mention_ceaf_plus 0.621 0.322 0.424 b_cubed 0.465 0.262 0.335 b_cubed_plus ************************************************************* Run ID: hltcoe5 Did the run access the live Web during the evaluation window: No Did the run use the Wikipedia links in the Reference KB: No Did the run use the relation/slot information encoded in the Reference KB: No Did the run use the text descriptions in the nodes of the Reference KB: No Did the run use any of the distributed EDL1 runs (from the first evaluation window): No ALL English, Chinese, and Spanish documents: Prec Recall F1 Metric 0.661 0.563 0.608 strong_typed_mention_match 0.481 0.409 0.442 strong_typed_all_match 0.567 0.355 0.436 strong_typed_link_match 0.366 0.599 0.455 strong_typed_nil_match 0.681 0.417 0.517 entity_match 0.567 0.482 0.521 mention_ceaf 0.548 0.466 0.504 typed_mention_ceaf 0.462 0.393 0.425 typed_mention_ceaf_plus 0.534 0.369 0.436 b_cubed 0.396 0.298 0.340 b_cubed_plus ONLY English documents: Prec Recall F1 Metric 0.682 0.658 0.670 strong_mention_match 0.642 0.619 0.630 strong_typed_mention_match 0.481 0.464 0.472 strong_typed_all_match 0.553 0.429 0.483 strong_typed_link_match 0.359 0.588 0.446 strong_typed_nil_match 0.732 0.501 0.595 entity_match 0.582 0.562 0.572 mention_ceaf 0.562 0.543 0.552 typed_mention_ceaf 0.464 0.447 0.455 typed_mention_ceaf_plus 0.500 0.462 0.480 b_cubed 0.379 0.365 0.372 b_cubed_plus ONLY Chinese documents: Prec Recall F1 Metric 0.696 0.636 0.665 strong_mention_match 0.652 0.596 0.623 strong_typed_mention_match 0.448 0.409 0.428 strong_typed_all_match 0.600 0.362 0.452 strong_typed_link_match 0.276 0.604 0.379 strong_typed_nil_match 0.644 0.396 0.490 entity_match 0.583 0.533 0.557 mention_ceaf 0.557 0.509 0.532 typed_mention_ceaf 0.439 0.401 0.420 typed_mention_ceaf_plus 0.561 0.427 0.485 b_cubed 0.395 0.320 0.353 b_cubed_plus ONLY Spanish documents: Prec Recall F1 Metric 0.773 0.480 0.592 strong_mention_match 0.718 0.446 0.550 strong_typed_mention_match 0.541 0.336 0.415 strong_typed_all_match 0.538 0.240 0.332 strong_typed_link_match 0.546 0.607 0.575 strong_typed_nil_match 0.643 0.337 0.442 entity_match 0.622 0.387 0.477 mention_ceaf 0.590 0.367 0.452 typed_mention_ceaf 0.501 0.311 0.384 typed_mention_ceaf_plus 0.598 0.299 0.399 b_cubed 0.456 0.245 0.319 b_cubed_plus