=============================================================================== TAC KBP 2016 ENTITY DISCOVERY AND LINKING EVALUATION RESULTS (FIRST EDL WINDOW) =============================================================================== Team ID: OSU_DEFT Organization: Oregon State University ************************************************************* Run ID: OSU_DEFT1 Did the run access the live Web during the evaluation window: No Did the run use the Wikipedia links in the Reference KB: Yes Did the run use the relation/slot information encoded in the Reference KB: No Did the run use the text descriptions in the nodes of the Reference KB: No ALL English, Chinese, and Spanish documents: Prec Recall F1 Metric 0.552 0.358 0.434 strong_typed_mention_match 0.462 0.299 0.363 strong_typed_all_match 0.510 0.381 0.436 strong_typed_link_match 0.045 0.013 0.021 strong_typed_nil_match 0.470 0.530 0.498 entity_match 0.555 0.360 0.437 mention_ceaf 0.503 0.326 0.396 typed_mention_ceaf 0.460 0.298 0.362 typed_mention_ceaf_plus 0.509 0.239 0.325 b_cubed 0.434 0.205 0.279 b_cubed_plus ONLY English documents: Prec Recall F1 Metric 0.553 0.370 0.443 strong_mention_match 0.445 0.298 0.357 strong_typed_mention_match 0.382 0.255 0.306 strong_typed_all_match 0.431 0.324 0.370 strong_typed_link_match 0.029 0.011 0.016 strong_typed_nil_match 0.420 0.493 0.453 entity_match 0.504 0.337 0.404 mention_ceaf 0.420 0.281 0.337 typed_mention_ceaf 0.380 0.254 0.305 typed_mention_ceaf_plus 0.465 0.232 0.310 b_cubed 0.407 0.202 0.270 b_cubed_plus ONLY Chinese documents: Prec Recall F1 Metric 0.715 0.455 0.556 strong_mention_match 0.645 0.410 0.502 strong_typed_mention_match 0.577 0.367 0.449 strong_typed_all_match 0.599 0.456 0.518 strong_typed_link_match 0.023 0.003 0.005 strong_typed_nil_match 0.527 0.568 0.547 entity_match 0.675 0.430 0.525 mention_ceaf 0.622 0.396 0.484 typed_mention_ceaf 0.577 0.367 0.449 typed_mention_ceaf_plus 0.595 0.324 0.420 b_cubed 0.515 0.282 0.365 b_cubed_plus ONLY Spanish documents: Prec Recall F1 Metric 0.672 0.428 0.523 strong_mention_match 0.584 0.372 0.454 strong_typed_mention_match 0.427 0.272 0.332 strong_typed_all_match 0.496 0.359 0.416 strong_typed_link_match 0.067 0.026 0.038 strong_typed_nil_match 0.477 0.537 0.505 entity_match 0.550 0.350 0.428 mention_ceaf 0.500 0.319 0.389 typed_mention_ceaf 0.423 0.270 0.329 typed_mention_ceaf_plus 0.520 0.232 0.321 b_cubed 0.396 0.177 0.244 b_cubed_plus ************************************************************* Run ID: OSU_DEFT2 Did the run access the live Web during the evaluation window: No Did the run use the Wikipedia links in the Reference KB: Yes Did the run use the relation/slot information encoded in the Reference KB: No Did the run use the text descriptions in the nodes of the Reference KB: No ALL English, Chinese, and Spanish documents: Prec Recall F1 Metric 0.552 0.358 0.434 strong_typed_mention_match 0.451 0.292 0.355 strong_typed_all_match 0.509 0.372 0.430 strong_typed_link_match 0.043 0.016 0.023 strong_typed_nil_match 0.456 0.506 0.480 entity_match 0.543 0.352 0.427 mention_ceaf 0.494 0.320 0.389 typed_mention_ceaf 0.448 0.291 0.353 typed_mention_ceaf_plus 0.508 0.233 0.319 b_cubed 0.424 0.198 0.270 b_cubed_plus ONLY English documents: Prec Recall F1 Metric 0.553 0.370 0.443 strong_mention_match 0.445 0.298 0.357 strong_typed_mention_match 0.358 0.239 0.287 strong_typed_all_match 0.420 0.303 0.352 strong_typed_link_match 0.026 0.012 0.017 strong_typed_nil_match 0.394 0.450 0.420 entity_match 0.481 0.322 0.386 mention_ceaf 0.403 0.269 0.323 typed_mention_ceaf 0.356 0.238 0.286 typed_mention_ceaf_plus 0.456 0.219 0.296 b_cubed 0.381 0.185 0.249 b_cubed_plus ONLY Chinese documents: Prec Recall F1 Metric 0.715 0.455 0.556 strong_mention_match 0.645 0.410 0.502 strong_typed_mention_match 0.573 0.364 0.445 strong_typed_all_match 0.596 0.452 0.514 strong_typed_link_match 0.034 0.005 0.008 strong_typed_nil_match 0.517 0.557 0.536 entity_match 0.675 0.430 0.525 mention_ceaf 0.622 0.396 0.484 typed_mention_ceaf 0.572 0.364 0.445 typed_mention_ceaf_plus 0.599 0.323 0.420 b_cubed 0.512 0.280 0.362 b_cubed_plus ONLY Spanish documents: Prec Recall F1 Metric 0.672 0.428 0.523 strong_mention_match 0.584 0.372 0.454 strong_typed_mention_match 0.425 0.271 0.331 strong_typed_all_match 0.505 0.357 0.418 strong_typed_link_match 0.067 0.030 0.041 strong_typed_nil_match 0.474 0.521 0.496 entity_match 0.541 0.345 0.421 mention_ceaf 0.493 0.314 0.384 typed_mention_ceaf 0.421 0.268 0.327 typed_mention_ceaf_plus 0.521 0.228 0.317 b_cubed 0.397 0.174 0.242 b_cubed_plus ************************************************************* Run ID: OSU_DEFT3 Did the run access the live Web during the evaluation window: No Did the run use the Wikipedia links in the Reference KB: Yes Did the run use the relation/slot information encoded in the Reference KB: No Did the run use the text descriptions in the nodes of the Reference KB: No ALL English, Chinese, and Spanish documents: Prec Recall F1 Metric 0.552 0.358 0.434 strong_typed_mention_match 0.462 0.300 0.364 strong_typed_all_match 0.510 0.382 0.437 strong_typed_link_match 0.045 0.013 0.021 strong_typed_nil_match 0.468 0.528 0.496 entity_match 0.557 0.361 0.438 mention_ceaf 0.504 0.327 0.397 typed_mention_ceaf 0.460 0.298 0.362 typed_mention_ceaf_plus 0.511 0.240 0.326 b_cubed 0.437 0.206 0.280 b_cubed_plus ONLY English documents: Prec Recall F1 Metric 0.553 0.370 0.443 strong_mention_match 0.445 0.298 0.357 strong_typed_mention_match 0.379 0.254 0.304 strong_typed_all_match 0.428 0.322 0.367 strong_typed_link_match 0.029 0.011 0.016 strong_typed_nil_match 0.410 0.481 0.442 entity_match 0.504 0.337 0.404 mention_ceaf 0.420 0.281 0.337 typed_mention_ceaf 0.378 0.253 0.303 typed_mention_ceaf_plus 0.466 0.232 0.310 b_cubed 0.405 0.200 0.267 b_cubed_plus ONLY Chinese documents: Prec Recall F1 Metric 0.715 0.455 0.556 strong_mention_match 0.645 0.410 0.502 strong_typed_mention_match 0.578 0.368 0.450 strong_typed_all_match 0.600 0.457 0.519 strong_typed_link_match 0.023 0.003 0.005 strong_typed_nil_match 0.524 0.567 0.544 entity_match 0.676 0.431 0.526 mention_ceaf 0.623 0.397 0.485 typed_mention_ceaf 0.578 0.368 0.450 typed_mention_ceaf_plus 0.599 0.325 0.421 b_cubed 0.519 0.283 0.366 b_cubed_plus ONLY Spanish documents: Prec Recall F1 Metric 0.672 0.428 0.523 strong_mention_match 0.584 0.372 0.454 strong_typed_mention_match 0.430 0.274 0.335 strong_typed_all_match 0.500 0.362 0.420 strong_typed_link_match 0.067 0.026 0.038 strong_typed_nil_match 0.488 0.545 0.515 entity_match 0.547 0.349 0.426 mention_ceaf 0.499 0.318 0.388 typed_mention_ceaf 0.427 0.272 0.332 typed_mention_ceaf_plus 0.523 0.230 0.319 b_cubed 0.404 0.178 0.247 b_cubed_plus ************************************************************* Run ID: OSU_DEFT4 Did the run access the live Web during the evaluation window: No Did the run use the Wikipedia links in the Reference KB: Yes Did the run use the relation/slot information encoded in the Reference KB: No Did the run use the text descriptions in the nodes of the Reference KB: No ALL English, Chinese, and Spanish documents: Prec Recall F1 Metric 0.522 0.375 0.436 strong_typed_mention_match 0.416 0.298 0.348 strong_typed_all_match 0.507 0.372 0.429 strong_typed_link_match 0.064 0.043 0.051 strong_typed_nil_match 0.456 0.506 0.480 entity_match 0.493 0.354 0.412 mention_ceaf 0.448 0.321 0.374 typed_mention_ceaf 0.408 0.292 0.341 typed_mention_ceaf_plus 0.459 0.238 0.313 b_cubed 0.383 0.202 0.265 b_cubed_plus ONLY English documents: Prec Recall F1 Metric 0.490 0.420 0.452 strong_mention_match 0.400 0.342 0.369 strong_typed_mention_match 0.299 0.256 0.276 strong_typed_all_match 0.417 0.303 0.351 strong_typed_link_match 0.066 0.087 0.075 strong_typed_nil_match 0.393 0.451 0.420 entity_match 0.383 0.328 0.354 mention_ceaf 0.322 0.276 0.297 typed_mention_ceaf 0.286 0.245 0.264 typed_mention_ceaf_plus 0.357 0.234 0.282 b_cubed 0.298 0.196 0.236 b_cubed_plus ONLY Chinese documents: Prec Recall F1 Metric 0.715 0.455 0.556 strong_mention_match 0.645 0.410 0.502 strong_typed_mention_match 0.573 0.364 0.445 strong_typed_all_match 0.596 0.452 0.514 strong_typed_link_match 0.034 0.005 0.008 strong_typed_nil_match 0.517 0.557 0.536 entity_match 0.675 0.430 0.525 mention_ceaf 0.622 0.396 0.484 typed_mention_ceaf 0.572 0.364 0.445 typed_mention_ceaf_plus 0.599 0.323 0.420 b_cubed 0.512 0.280 0.362 b_cubed_plus ONLY Spanish documents: Prec Recall F1 Metric 0.672 0.428 0.523 strong_mention_match 0.584 0.372 0.454 strong_typed_mention_match 0.425 0.271 0.331 strong_typed_all_match 0.505 0.357 0.418 strong_typed_link_match 0.067 0.030 0.041 strong_typed_nil_match 0.474 0.521 0.496 entity_match 0.541 0.345 0.421 mention_ceaf 0.493 0.314 0.384 typed_mention_ceaf 0.421 0.268 0.327 typed_mention_ceaf_plus 0.521 0.228 0.317 b_cubed 0.397 0.174 0.242 b_cubed_plus ************************************************************* Run ID: OSU_DEFT5 Did the run access the live Web during the evaluation window: No Did the run use the Wikipedia links in the Reference KB: Yes Did the run use the relation/slot information encoded in the Reference KB: No Did the run use the text descriptions in the nodes of the Reference KB: No ALL English, Chinese, and Spanish documents: Prec Recall F1 Metric 0.522 0.375 0.436 strong_typed_mention_match 0.416 0.298 0.348 strong_typed_all_match 0.507 0.372 0.429 strong_typed_link_match 0.064 0.043 0.051 strong_typed_nil_match 0.456 0.506 0.480 entity_match 0.493 0.354 0.412 mention_ceaf 0.448 0.321 0.374 typed_mention_ceaf 0.408 0.292 0.341 typed_mention_ceaf_plus 0.459 0.238 0.313 b_cubed 0.383 0.202 0.265 b_cubed_plus ONLY English documents: Prec Recall F1 Metric 0.490 0.420 0.452 strong_mention_match 0.400 0.342 0.369 strong_typed_mention_match 0.299 0.256 0.276 strong_typed_all_match 0.417 0.303 0.351 strong_typed_link_match 0.066 0.087 0.075 strong_typed_nil_match 0.393 0.451 0.420 entity_match 0.383 0.328 0.354 mention_ceaf 0.322 0.276 0.297 typed_mention_ceaf 0.286 0.245 0.264 typed_mention_ceaf_plus 0.357 0.234 0.282 b_cubed 0.298 0.196 0.236 b_cubed_plus ONLY Chinese documents: Prec Recall F1 Metric 0.715 0.455 0.556 strong_mention_match 0.645 0.410 0.502 strong_typed_mention_match 0.573 0.364 0.445 strong_typed_all_match 0.596 0.452 0.514 strong_typed_link_match 0.034 0.005 0.008 strong_typed_nil_match 0.517 0.557 0.536 entity_match 0.675 0.430 0.525 mention_ceaf 0.622 0.396 0.484 typed_mention_ceaf 0.572 0.364 0.445 typed_mention_ceaf_plus 0.599 0.323 0.420 b_cubed 0.512 0.280 0.362 b_cubed_plus ONLY Spanish documents: Prec Recall F1 Metric 0.672 0.428 0.523 strong_mention_match 0.584 0.372 0.454 strong_typed_mention_match 0.425 0.271 0.331 strong_typed_all_match 0.505 0.357 0.418 strong_typed_link_match 0.067 0.030 0.041 strong_typed_nil_match 0.474 0.521 0.496 entity_match 0.541 0.345 0.421 mention_ceaf 0.493 0.314 0.384 typed_mention_ceaf 0.421 0.268 0.327 typed_mention_ceaf_plus 0.521 0.228 0.317 b_cubed 0.397 0.174 0.242 b_cubed_plus