================================================================================ TAC KBP 2016 ENTITY DISCOVERY AND LINKING EVALUATION RESULTS (SECOND EDL WINDOW) ================================================================================ Team ID: OSU_DEFT Organization: Oregon State University ************************************************************* Run ID: OSU_DEFT1 Did the run access the live Web during the evaluation window: No Did the run use the Wikipedia links in the Reference KB: Yes Did the run use the relation/slot information encoded in the Reference KB: No Did the run use the text descriptions in the nodes of the Reference KB: No Did the run use any of the distributed EDL1 runs (from the first evaluation window): No ALL English, Chinese, and Spanish documents: Prec Recall F1 Metric 0.629 0.450 0.525 strong_typed_mention_match 0.505 0.361 0.421 strong_typed_all_match 0.563 0.451 0.501 strong_typed_link_match 0.112 0.046 0.066 strong_typed_nil_match 0.506 0.582 0.541 entity_match 0.586 0.419 0.488 mention_ceaf 0.548 0.392 0.457 typed_mention_ceaf 0.499 0.357 0.416 typed_mention_ceaf_plus 0.564 0.298 0.390 b_cubed 0.460 0.252 0.325 b_cubed_plus ONLY English documents: Prec Recall F1 Metric 0.689 0.592 0.637 strong_mention_match 0.631 0.542 0.583 strong_typed_mention_match 0.474 0.407 0.438 strong_typed_all_match 0.513 0.500 0.507 strong_typed_link_match 0.172 0.077 0.106 strong_typed_nil_match 0.453 0.638 0.530 entity_match 0.564 0.485 0.522 mention_ceaf 0.536 0.461 0.495 typed_mention_ceaf 0.468 0.402 0.433 typed_mention_ceaf_plus 0.548 0.375 0.446 b_cubed 0.427 0.301 0.353 b_cubed_plus ONLY Chinese documents: Prec Recall F1 Metric 0.715 0.455 0.556 strong_mention_match 0.645 0.410 0.502 strong_typed_mention_match 0.582 0.370 0.452 strong_typed_all_match 0.626 0.454 0.526 strong_typed_link_match 0.094 0.025 0.040 strong_typed_nil_match 0.560 0.560 0.560 entity_match 0.665 0.423 0.517 mention_ceaf 0.612 0.389 0.476 typed_mention_ceaf 0.575 0.366 0.447 typed_mention_ceaf_plus 0.588 0.324 0.418 b_cubed 0.522 0.285 0.368 b_cubed_plus ONLY Spanish documents: Prec Recall F1 Metric 0.715 0.446 0.549 strong_mention_match 0.605 0.377 0.465 strong_typed_mention_match 0.460 0.287 0.354 strong_typed_all_match 0.569 0.377 0.454 strong_typed_link_match 0.063 0.032 0.043 strong_typed_nil_match 0.541 0.536 0.539 entity_match 0.612 0.382 0.470 mention_ceaf 0.538 0.335 0.413 typed_mention_ceaf 0.460 0.287 0.353 typed_mention_ceaf_plus 0.632 0.259 0.368 b_cubed 0.468 0.201 0.281 b_cubed_plus ************************************************************* Run ID: OSU_DEFT2 Did the run access the live Web during the evaluation window: No Did the run use the Wikipedia links in the Reference KB: Yes Did the run use the relation/slot information encoded in the Reference KB: No Did the run use the text descriptions in the nodes of the Reference KB: No Did the run use any of the distributed EDL1 runs (from the first evaluation window): No ALL English, Chinese, and Spanish documents: Prec Recall F1 Metric 0.629 0.450 0.525 strong_typed_mention_match 0.514 0.367 0.429 strong_typed_all_match 0.569 0.460 0.508 strong_typed_link_match 0.117 0.046 0.066 strong_typed_nil_match 0.504 0.585 0.541 entity_match 0.596 0.426 0.497 mention_ceaf 0.558 0.399 0.466 typed_mention_ceaf 0.509 0.364 0.424 typed_mention_ceaf_plus 0.568 0.302 0.395 b_cubed 0.469 0.258 0.333 b_cubed_plus ONLY English documents: Prec Recall F1 Metric 0.689 0.592 0.637 strong_mention_match 0.631 0.542 0.583 strong_typed_mention_match 0.492 0.423 0.455 strong_typed_all_match 0.533 0.520 0.526 strong_typed_link_match 0.173 0.077 0.107 strong_typed_nil_match 0.455 0.639 0.532 entity_match 0.572 0.492 0.529 mention_ceaf 0.544 0.467 0.502 typed_mention_ceaf 0.485 0.417 0.449 typed_mention_ceaf_plus 0.551 0.382 0.452 b_cubed 0.442 0.316 0.368 b_cubed_plus ONLY Chinese documents: Prec Recall F1 Metric 0.715 0.455 0.556 strong_mention_match 0.645 0.410 0.502 strong_typed_mention_match 0.580 0.369 0.451 strong_typed_all_match 0.625 0.453 0.525 strong_typed_link_match 0.094 0.025 0.040 strong_typed_nil_match 0.556 0.558 0.557 entity_match 0.663 0.422 0.516 mention_ceaf 0.611 0.389 0.475 typed_mention_ceaf 0.574 0.365 0.446 typed_mention_ceaf_plus 0.589 0.324 0.418 b_cubed 0.521 0.284 0.368 b_cubed_plus ONLY Spanish documents: Prec Recall F1 Metric 0.715 0.446 0.549 strong_mention_match 0.605 0.377 0.465 strong_typed_mention_match 0.468 0.292 0.360 strong_typed_all_match 0.560 0.385 0.456 strong_typed_link_match 0.069 0.031 0.043 strong_typed_nil_match 0.533 0.546 0.539 entity_match 0.621 0.387 0.477 mention_ceaf 0.547 0.341 0.420 typed_mention_ceaf 0.468 0.292 0.359 typed_mention_ceaf_plus 0.637 0.262 0.372 b_cubed 0.478 0.206 0.287 b_cubed_plus ************************************************************* Run ID: OSU_DEFT3 Did the run access the live Web during the evaluation window: No Did the run use the Wikipedia links in the Reference KB: Yes Did the run use the relation/slot information encoded in the Reference KB: No Did the run use the text descriptions in the nodes of the Reference KB: No Did the run use any of the distributed EDL1 runs (from the first evaluation window): Yes ALL English, Chinese, and Spanish documents: Prec Recall F1 Metric 0.627 0.450 0.524 strong_typed_mention_match 0.503 0.361 0.420 strong_typed_all_match 0.561 0.452 0.500 strong_typed_link_match 0.109 0.045 0.064 strong_typed_nil_match 0.503 0.582 0.539 entity_match 0.585 0.420 0.489 mention_ceaf 0.547 0.392 0.457 typed_mention_ceaf 0.498 0.357 0.416 typed_mention_ceaf_plus 0.564 0.299 0.391 b_cubed 0.459 0.252 0.325 b_cubed_plus ONLY English documents: Prec Recall F1 Metric 0.688 0.595 0.638 strong_mention_match 0.626 0.542 0.581 strong_typed_mention_match 0.471 0.408 0.437 strong_typed_all_match 0.509 0.502 0.506 strong_typed_link_match 0.166 0.073 0.102 strong_typed_nil_match 0.447 0.639 0.526 entity_match 0.562 0.487 0.522 mention_ceaf 0.533 0.462 0.495 typed_mention_ceaf 0.465 0.403 0.432 typed_mention_ceaf_plus 0.548 0.377 0.447 b_cubed 0.425 0.301 0.353 b_cubed_plus ONLY Chinese documents: Prec Recall F1 Metric 0.715 0.455 0.556 strong_mention_match 0.645 0.410 0.502 strong_typed_mention_match 0.582 0.370 0.452 strong_typed_all_match 0.626 0.454 0.526 strong_typed_link_match 0.094 0.025 0.040 strong_typed_nil_match 0.560 0.560 0.560 entity_match 0.665 0.423 0.517 mention_ceaf 0.612 0.389 0.476 typed_mention_ceaf 0.575 0.366 0.447 typed_mention_ceaf_plus 0.588 0.324 0.418 b_cubed 0.522 0.285 0.368 b_cubed_plus ONLY Spanish documents: Prec Recall F1 Metric 0.715 0.446 0.549 strong_mention_match 0.605 0.377 0.465 strong_typed_mention_match 0.460 0.287 0.354 strong_typed_all_match 0.569 0.377 0.454 strong_typed_link_match 0.063 0.032 0.043 strong_typed_nil_match 0.541 0.536 0.539 entity_match 0.612 0.382 0.470 mention_ceaf 0.538 0.335 0.413 typed_mention_ceaf 0.460 0.287 0.353 typed_mention_ceaf_plus 0.632 0.259 0.368 b_cubed 0.468 0.201 0.281 b_cubed_plus ************************************************************* Run ID: OSU_DEFT4 Did the run access the live Web during the evaluation window: No Did the run use the Wikipedia links in the Reference KB: Yes Did the run use the relation/slot information encoded in the Reference KB: No Did the run use the text descriptions in the nodes of the Reference KB: No Did the run use any of the distributed EDL1 runs (from the first evaluation window): Yes ALL English, Chinese, and Spanish documents: Prec Recall F1 Metric 0.627 0.450 0.524 strong_typed_mention_match 0.513 0.368 0.428 strong_typed_all_match 0.567 0.460 0.508 strong_typed_link_match 0.114 0.044 0.064 strong_typed_nil_match 0.501 0.585 0.540 entity_match 0.595 0.427 0.497 mention_ceaf 0.557 0.399 0.465 typed_mention_ceaf 0.507 0.364 0.424 typed_mention_ceaf_plus 0.568 0.303 0.395 b_cubed 0.468 0.258 0.333 b_cubed_plus ONLY English documents: Prec Recall F1 Metric 0.688 0.595 0.638 strong_mention_match 0.626 0.542 0.581 strong_typed_mention_match 0.489 0.423 0.454 strong_typed_all_match 0.529 0.522 0.525 strong_typed_link_match 0.166 0.073 0.101 strong_typed_nil_match 0.449 0.640 0.528 entity_match 0.570 0.494 0.529 mention_ceaf 0.540 0.468 0.501 typed_mention_ceaf 0.483 0.418 0.448 typed_mention_ceaf_plus 0.551 0.384 0.453 b_cubed 0.441 0.317 0.368 b_cubed_plus ONLY Chinese documents: Prec Recall F1 Metric 0.715 0.455 0.556 strong_mention_match 0.645 0.410 0.502 strong_typed_mention_match 0.580 0.369 0.451 strong_typed_all_match 0.625 0.453 0.525 strong_typed_link_match 0.094 0.025 0.040 strong_typed_nil_match 0.556 0.558 0.557 entity_match 0.663 0.422 0.516 mention_ceaf 0.611 0.389 0.475 typed_mention_ceaf 0.574 0.365 0.446 typed_mention_ceaf_plus 0.589 0.324 0.418 b_cubed 0.521 0.284 0.368 b_cubed_plus ONLY Spanish documents: Prec Recall F1 Metric 0.715 0.446 0.549 strong_mention_match 0.605 0.377 0.465 strong_typed_mention_match 0.468 0.292 0.360 strong_typed_all_match 0.560 0.385 0.456 strong_typed_link_match 0.069 0.031 0.043 strong_typed_nil_match 0.533 0.546 0.539 entity_match 0.621 0.387 0.477 mention_ceaf 0.547 0.341 0.420 typed_mention_ceaf 0.468 0.292 0.359 typed_mention_ceaf_plus 0.637 0.262 0.372 b_cubed 0.478 0.206 0.287 b_cubed_plus