=================================================================================== TAC KBP 2017 LOW RESOURCE LANGUAGES ENTITY DISCOVERY AND LINKING EVALUATION RESULTS =================================================================================== Team ID: IBM Organization: IBM Research Team ID: IBM_Research Organization: IBM Research ************************************************************* Run ID: IBM1 Languages addressed: Polish, Chechen, Albanian, Swahili, Kannada, Yoruba, Northern Sotho, Nepali, Kikuyu, Somali Did the run access the live Web during the evaluation window: No Did the run use the relation/slot information encoded in the Reference KB: No Did the run use the text descriptions in the nodes of the Reference KB: No ALL languge documents: Prec Recall F1 Metric 0.540 0.490 0.514 strong_mention_match 0.499 0.453 0.475 strong_typed_mention_match 0.410 0.372 0.390 strong_typed_all_match 0.507 0.426 0.463 strong_typed_link_match 0.041 0.054 0.046 strong_typed_nil_match 0.518 0.484 0.500 entity_match ONLY Albanian documents: Prec Recall F1 Metric 0.789 0.565 0.659 strong_mention_match 0.657 0.471 0.549 strong_typed_mention_match 0.377 0.270 0.315 strong_typed_all_match 0.550 0.276 0.367 strong_typed_link_match 0.000 0.000 0.000 strong_typed_nil_match 0.563 0.407 0.472 entity_match ONLY Chechen documents: Prec Recall F1 Metric 0.002 0.002 0.002 strong_mention_match 0.000 0.000 0.000 strong_typed_mention_match 0.000 0.000 0.000 strong_typed_all_match 0.000 0.000 0.000 strong_typed_link_match 0.000 0.000 0.000 strong_typed_nil_match 0.000 0.000 0.000 entity_match ONLY Kannada documents: Prec Recall F1 Metric 0.576 0.353 0.438 strong_mention_match 0.424 0.260 0.323 strong_typed_mention_match 0.235 0.144 0.179 strong_typed_all_match 0.413 0.158 0.229 strong_typed_link_match 0.000 0.000 0.000 strong_typed_nil_match 0.613 0.244 0.349 entity_match ONLY Kikuyu documents: Prec Recall F1 Metric 0.727 0.901 0.805 strong_mention_match 0.726 0.899 0.803 strong_typed_mention_match 0.720 0.892 0.797 strong_typed_all_match 0.744 0.892 0.811 strong_typed_link_match 0.000 0.000 0.000 strong_typed_nil_match 0.246 0.568 0.343 entity_match ONLY Nepali documents: Prec Recall F1 Metric 0.552 0.259 0.352 strong_mention_match 0.500 0.234 0.319 strong_typed_mention_match 0.490 0.229 0.312 strong_typed_all_match 0.667 0.250 0.364 strong_typed_link_match 0.037 0.048 0.042 strong_typed_nil_match 0.844 0.429 0.568 entity_match ONLY Northern Sotho documents: Prec Recall F1 Metric 0.464 0.447 0.455 strong_mention_match 0.429 0.413 0.421 strong_typed_mention_match 0.387 0.373 0.380 strong_typed_all_match 0.397 0.373 0.385 strong_typed_link_match 0.000 0.000 0.000 strong_typed_nil_match 0.832 0.706 0.764 entity_match ONLY Polish documents: Prec Recall F1 Metric 0.719 0.704 0.711 strong_mention_match 0.641 0.627 0.634 strong_typed_mention_match 0.395 0.387 0.391 strong_typed_all_match 0.522 0.383 0.442 strong_typed_link_match 0.064 0.500 0.114 strong_typed_nil_match 0.532 0.584 0.557 entity_match ONLY Somali documents: Prec Recall F1 Metric 0.501 0.351 0.413 strong_mention_match 0.464 0.325 0.382 strong_typed_mention_match 0.328 0.230 0.271 strong_typed_all_match 0.441 0.246 0.316 strong_typed_link_match 0.139 0.171 0.153 strong_typed_nil_match 0.670 0.410 0.508 entity_match ONLY Swahili documents: Prec Recall F1 Metric 0.728 0.704 0.716 strong_mention_match 0.675 0.653 0.664 strong_typed_mention_match 0.518 0.502 0.510 strong_typed_all_match 0.568 0.515 0.540 strong_typed_link_match 0.130 0.265 0.175 strong_typed_nil_match 0.558 0.570 0.564 entity_match ONLY Yoruba documents: Prec Recall F1 Metric 0.531 0.437 0.480 strong_mention_match 0.475 0.391 0.429 strong_typed_mention_match 0.320 0.264 0.289 strong_typed_all_match 0.422 0.317 0.362 strong_typed_link_match 0.070 0.076 0.073 strong_typed_nil_match 0.443 0.428 0.435 entity_match ************************************************************* Run ID: IBM2 Languages addressed: Polish, Chechen, Albanian, Swahili, Kannada, Yoruba, Northern Sotho, Nepali, Kikuyu, Somali Did the run access the live Web during the evaluation window: No Did the run use the relation/slot information encoded in the Reference KB: No Did the run use the text descriptions in the nodes of the Reference KB: No ALL languge documents: Prec Recall F1 Metric 0.540 0.490 0.514 strong_mention_match 0.499 0.453 0.475 strong_typed_mention_match 0.384 0.348 0.365 strong_typed_all_match 0.481 0.398 0.436 strong_typed_link_match 0.039 0.054 0.045 strong_typed_nil_match 0.479 0.446 0.462 entity_match ONLY Albanian documents: Prec Recall F1 Metric 0.789 0.565 0.659 strong_mention_match 0.657 0.471 0.549 strong_typed_mention_match 0.359 0.257 0.300 strong_typed_all_match 0.528 0.262 0.350 strong_typed_link_match 0.000 0.000 0.000 strong_typed_nil_match 0.532 0.394 0.453 entity_match ONLY Chechen documents: Prec Recall F1 Metric 0.002 0.002 0.002 strong_mention_match 0.000 0.000 0.000 strong_typed_mention_match 0.000 0.000 0.000 strong_typed_all_match 0.000 0.000 0.000 strong_typed_link_match 0.000 0.000 0.000 strong_typed_nil_match 0.000 0.000 0.000 entity_match ONLY Kannada documents: Prec Recall F1 Metric 0.576 0.353 0.438 strong_mention_match 0.424 0.260 0.323 strong_typed_mention_match 0.189 0.116 0.144 strong_typed_all_match 0.410 0.128 0.195 strong_typed_link_match 0.000 0.000 0.000 strong_typed_nil_match 0.533 0.205 0.296 entity_match ONLY Kikuyu documents: Prec Recall F1 Metric 0.727 0.901 0.805 strong_mention_match 0.726 0.899 0.803 strong_typed_mention_match 0.715 0.886 0.792 strong_typed_all_match 0.742 0.886 0.808 strong_typed_link_match 0.000 0.000 0.000 strong_typed_nil_match 0.237 0.528 0.327 entity_match ONLY Nepali documents: Prec Recall F1 Metric 0.552 0.259 0.352 strong_mention_match 0.500 0.234 0.319 strong_typed_mention_match 0.125 0.059 0.080 strong_typed_all_match 0.344 0.060 0.102 strong_typed_link_match 0.016 0.048 0.024 strong_typed_nil_match 0.524 0.175 0.262 entity_match ONLY Northern Sotho documents: Prec Recall F1 Metric 0.464 0.447 0.455 strong_mention_match 0.429 0.413 0.421 strong_typed_mention_match 0.386 0.372 0.379 strong_typed_all_match 0.396 0.372 0.384 strong_typed_link_match 0.000 0.000 0.000 strong_typed_nil_match 0.819 0.686 0.747 entity_match ONLY Polish documents: Prec Recall F1 Metric 0.719 0.704 0.711 strong_mention_match 0.641 0.627 0.634 strong_typed_mention_match 0.302 0.296 0.299 strong_typed_all_match 0.400 0.289 0.335 strong_typed_link_match 0.062 0.500 0.110 strong_typed_nil_match 0.420 0.487 0.451 entity_match ONLY Somali documents: Prec Recall F1 Metric 0.501 0.351 0.413 strong_mention_match 0.464 0.325 0.382 strong_typed_mention_match 0.317 0.222 0.261 strong_typed_all_match 0.428 0.236 0.305 strong_typed_link_match 0.137 0.171 0.152 strong_typed_nil_match 0.660 0.404 0.501 entity_match ONLY Swahili documents: Prec Recall F1 Metric 0.728 0.704 0.716 strong_mention_match 0.675 0.653 0.664 strong_typed_mention_match 0.454 0.439 0.446 strong_typed_all_match 0.507 0.449 0.476 strong_typed_link_match 0.111 0.265 0.157 strong_typed_nil_match 0.508 0.515 0.512 entity_match ONLY Yoruba documents: Prec Recall F1 Metric 0.531 0.437 0.480 strong_mention_match 0.475 0.391 0.429 strong_typed_mention_match 0.257 0.212 0.232 strong_typed_all_match 0.334 0.251 0.286 strong_typed_link_match 0.070 0.076 0.073 strong_typed_nil_match 0.406 0.401 0.403 entity_match ************************************************************* Run ID: IBM3 Languages addressed: Polish, Chechen, Albanian, Swahili, Kannada, Yoruba, Northern Sotho, Nepali, Kikuyu, Somali Did the run access the live Web during the evaluation window: No Did the run use the relation/slot information encoded in the Reference KB: No Did the run use the text descriptions in the nodes of the Reference KB: No ALL languge documents: Prec Recall F1 Metric 0.540 0.490 0.514 strong_mention_match 0.499 0.453 0.475 strong_typed_mention_match 0.379 0.344 0.361 strong_typed_all_match 0.478 0.394 0.432 strong_typed_link_match 0.036 0.051 0.042 strong_typed_nil_match 0.486 0.447 0.465 entity_match ONLY Albanian documents: Prec Recall F1 Metric 0.789 0.565 0.659 strong_mention_match 0.657 0.471 0.549 strong_typed_mention_match 0.359 0.257 0.300 strong_typed_all_match 0.518 0.262 0.348 strong_typed_link_match 0.000 0.000 0.000 strong_typed_nil_match 0.534 0.407 0.462 entity_match ONLY Chechen documents: Prec Recall F1 Metric 0.002 0.002 0.002 strong_mention_match 0.000 0.000 0.000 strong_typed_mention_match 0.000 0.000 0.000 strong_typed_all_match 0.000 0.000 0.000 strong_typed_link_match 0.000 0.000 0.000 strong_typed_nil_match 0.000 0.000 0.000 entity_match ONLY Kannada documents: Prec Recall F1 Metric 0.576 0.353 0.438 strong_mention_match 0.424 0.260 0.323 strong_typed_mention_match 0.227 0.140 0.173 strong_typed_all_match 0.492 0.153 0.233 strong_typed_link_match 0.000 0.000 0.000 strong_typed_nil_match 0.667 0.256 0.370 entity_match ONLY Kikuyu documents: Prec Recall F1 Metric 0.727 0.901 0.805 strong_mention_match 0.726 0.899 0.803 strong_typed_mention_match 0.689 0.854 0.763 strong_typed_all_match 0.779 0.854 0.814 strong_typed_link_match 0.000 0.000 0.000 strong_typed_nil_match 0.238 0.408 0.301 entity_match ONLY Nepali documents: Prec Recall F1 Metric 0.552 0.259 0.352 strong_mention_match 0.500 0.234 0.319 strong_typed_mention_match 0.125 0.059 0.080 strong_typed_all_match 0.393 0.060 0.104 strong_typed_link_match 0.015 0.048 0.022 strong_typed_nil_match 0.600 0.190 0.289 entity_match ONLY Northern Sotho documents: Prec Recall F1 Metric 0.464 0.447 0.455 strong_mention_match 0.429 0.413 0.421 strong_typed_mention_match 0.387 0.373 0.380 strong_typed_all_match 0.393 0.373 0.383 strong_typed_link_match 0.000 0.000 0.000 strong_typed_nil_match 0.819 0.686 0.747 entity_match ONLY Polish documents: Prec Recall F1 Metric 0.719 0.704 0.711 strong_mention_match 0.641 0.627 0.634 strong_typed_mention_match 0.310 0.303 0.306 strong_typed_all_match 0.404 0.296 0.342 strong_typed_link_match 0.064 0.500 0.114 strong_typed_nil_match 0.423 0.487 0.453 entity_match ONLY Somali documents: Prec Recall F1 Metric 0.502 0.351 0.413 strong_mention_match 0.465 0.325 0.383 strong_typed_mention_match 0.309 0.216 0.254 strong_typed_all_match 0.360 0.230 0.281 strong_typed_link_match 0.178 0.165 0.171 strong_typed_nil_match 0.579 0.422 0.488 entity_match ONLY Swahili documents: Prec Recall F1 Metric 0.728 0.704 0.716 strong_mention_match 0.675 0.653 0.664 strong_typed_mention_match 0.452 0.438 0.445 strong_typed_all_match 0.504 0.448 0.474 strong_typed_link_match 0.113 0.265 0.158 strong_typed_nil_match 0.504 0.511 0.507 entity_match ONLY Yoruba documents: Prec Recall F1 Metric 0.531 0.437 0.480 strong_mention_match 0.475 0.391 0.429 strong_typed_mention_match 0.254 0.209 0.230 strong_typed_all_match 0.338 0.249 0.287 strong_typed_link_match 0.061 0.069 0.065 strong_typed_nil_match 0.411 0.411 0.411 entity_match ************************************************************* Run ID: IBM4 Languages addressed: Polish, Chechen, Albanian, Swahili, Kannada, Yoruba, Northern Sotho, Nepali, Kikuyu, Somali Did the run access the live Web during the evaluation window: No Did the run use the relation/slot information encoded in the Reference KB: No Did the run use the text descriptions in the nodes of the Reference KB: No ALL languge documents: Prec Recall F1 Metric 0.575 0.490 0.529 strong_mention_match 0.531 0.453 0.489 strong_typed_mention_match 0.436 0.372 0.401 strong_typed_all_match 0.508 0.426 0.463 strong_typed_link_match 0.057 0.054 0.055 strong_typed_nil_match 0.521 0.484 0.502 entity_match ONLY Albanian documents: Prec Recall F1 Metric 0.789 0.565 0.659 strong_mention_match 0.657 0.471 0.549 strong_typed_mention_match 0.377 0.270 0.315 strong_typed_all_match 0.550 0.276 0.367 strong_typed_link_match 0.000 0.000 0.000 strong_typed_nil_match 0.563 0.407 0.472 entity_match ONLY Chechen documents: Prec Recall F1 Metric 0.000 0.000 0.000 strong_mention_match 0.000 0.000 0.000 strong_typed_mention_match 0.000 0.000 0.000 strong_typed_all_match 0.000 0.000 0.000 strong_typed_link_match 0.000 0.000 0.000 strong_typed_nil_match 0.000 0.000 0.000 entity_match ONLY Kannada documents: Prec Recall F1 Metric 0.576 0.353 0.438 strong_mention_match 0.424 0.260 0.323 strong_typed_mention_match 0.235 0.144 0.179 strong_typed_all_match 0.413 0.158 0.229 strong_typed_link_match 0.000 0.000 0.000 strong_typed_nil_match 0.613 0.244 0.349 entity_match ONLY Kikuyu documents: Prec Recall F1 Metric 0.727 0.901 0.805 strong_mention_match 0.726 0.899 0.803 strong_typed_mention_match 0.720 0.892 0.797 strong_typed_all_match 0.744 0.892 0.811 strong_typed_link_match 0.000 0.000 0.000 strong_typed_nil_match 0.246 0.568 0.343 entity_match ONLY Nepali documents: Prec Recall F1 Metric 0.552 0.259 0.352 strong_mention_match 0.500 0.234 0.319 strong_typed_mention_match 0.490 0.229 0.312 strong_typed_all_match 0.667 0.250 0.364 strong_typed_link_match 0.037 0.048 0.042 strong_typed_nil_match 0.844 0.429 0.568 entity_match ONLY Northern Sotho documents: Prec Recall F1 Metric 0.464 0.447 0.455 strong_mention_match 0.429 0.413 0.421 strong_typed_mention_match 0.387 0.373 0.380 strong_typed_all_match 0.397 0.373 0.385 strong_typed_link_match 0.000 0.000 0.000 strong_typed_nil_match 0.832 0.706 0.764 entity_match ONLY Polish documents: Prec Recall F1 Metric 0.719 0.704 0.711 strong_mention_match 0.641 0.627 0.634 strong_typed_mention_match 0.395 0.387 0.391 strong_typed_all_match 0.522 0.383 0.442 strong_typed_link_match 0.064 0.500 0.114 strong_typed_nil_match 0.532 0.584 0.557 entity_match ONLY Somali documents: Prec Recall F1 Metric 0.501 0.351 0.413 strong_mention_match 0.464 0.325 0.382 strong_typed_mention_match 0.328 0.230 0.271 strong_typed_all_match 0.441 0.246 0.316 strong_typed_link_match 0.139 0.171 0.153 strong_typed_nil_match 0.670 0.410 0.508 entity_match ONLY Swahili documents: Prec Recall F1 Metric 0.728 0.704 0.716 strong_mention_match 0.675 0.653 0.664 strong_typed_mention_match 0.518 0.502 0.510 strong_typed_all_match 0.568 0.515 0.540 strong_typed_link_match 0.130 0.265 0.175 strong_typed_nil_match 0.558 0.570 0.564 entity_match ONLY Yoruba documents: Prec Recall F1 Metric 0.531 0.437 0.480 strong_mention_match 0.475 0.391 0.429 strong_typed_mention_match 0.320 0.264 0.289 strong_typed_all_match 0.422 0.317 0.362 strong_typed_link_match 0.070 0.076 0.073 strong_typed_nil_match 0.443 0.428 0.435 entity_match ************************************************************* Run ID: IBM5 Languages addressed: Polish, Chechen, Albanian, Swahili, Kannada, Yoruba, Northern Sotho, Nepali, Kikuyu, Somali Did the run access the live Web during the evaluation window: No Did the run use the relation/slot information encoded in the Reference KB: No Did the run use the text descriptions in the nodes of the Reference KB: No ALL languge documents: Prec Recall F1 Metric 0.574 0.490 0.529 strong_mention_match 0.530 0.453 0.488 strong_typed_mention_match 0.435 0.372 0.401 strong_typed_all_match 0.507 0.426 0.463 strong_typed_link_match 0.057 0.054 0.055 strong_typed_nil_match 0.520 0.484 0.501 entity_match ONLY Albanian documents: Prec Recall F1 Metric 0.789 0.565 0.659 strong_mention_match 0.657 0.471 0.549 strong_typed_mention_match 0.377 0.270 0.315 strong_typed_all_match 0.550 0.276 0.367 strong_typed_link_match 0.000 0.000 0.000 strong_typed_nil_match 0.563 0.407 0.472 entity_match ONLY Chechen documents: Prec Recall F1 Metric 0.000 0.000 0.000 strong_mention_match 0.000 0.000 0.000 strong_typed_mention_match 0.000 0.000 0.000 strong_typed_all_match 0.000 0.000 0.000 strong_typed_link_match 0.000 0.000 0.000 strong_typed_nil_match 0.000 0.000 0.000 entity_match ONLY Kannada documents: Prec Recall F1 Metric 0.576 0.353 0.438 strong_mention_match 0.424 0.260 0.323 strong_typed_mention_match 0.235 0.144 0.179 strong_typed_all_match 0.413 0.158 0.229 strong_typed_link_match 0.000 0.000 0.000 strong_typed_nil_match 0.613 0.244 0.349 entity_match ONLY Kikuyu documents: Prec Recall F1 Metric 0.727 0.901 0.805 strong_mention_match 0.726 0.899 0.803 strong_typed_mention_match 0.720 0.892 0.797 strong_typed_all_match 0.744 0.892 0.811 strong_typed_link_match 0.000 0.000 0.000 strong_typed_nil_match 0.246 0.568 0.343 entity_match ONLY Nepali documents: Prec Recall F1 Metric 0.552 0.259 0.352 strong_mention_match 0.500 0.234 0.319 strong_typed_mention_match 0.490 0.229 0.312 strong_typed_all_match 0.667 0.250 0.364 strong_typed_link_match 0.037 0.048 0.042 strong_typed_nil_match 0.844 0.429 0.568 entity_match ONLY Northern Sotho documents: Prec Recall F1 Metric 0.464 0.447 0.455 strong_mention_match 0.429 0.413 0.421 strong_typed_mention_match 0.387 0.373 0.380 strong_typed_all_match 0.397 0.373 0.385 strong_typed_link_match 0.000 0.000 0.000 strong_typed_nil_match 0.832 0.706 0.764 entity_match ONLY Polish documents: Prec Recall F1 Metric 0.719 0.704 0.711 strong_mention_match 0.641 0.627 0.634 strong_typed_mention_match 0.395 0.387 0.391 strong_typed_all_match 0.522 0.383 0.442 strong_typed_link_match 0.064 0.500 0.114 strong_typed_nil_match 0.532 0.584 0.557 entity_match ONLY Somali documents: Prec Recall F1 Metric 0.501 0.351 0.413 strong_mention_match 0.464 0.325 0.382 strong_typed_mention_match 0.328 0.230 0.271 strong_typed_all_match 0.441 0.246 0.316 strong_typed_link_match 0.139 0.171 0.153 strong_typed_nil_match 0.670 0.410 0.508 entity_match ONLY Swahili documents: Prec Recall F1 Metric 0.728 0.704 0.716 strong_mention_match 0.675 0.653 0.664 strong_typed_mention_match 0.518 0.502 0.510 strong_typed_all_match 0.568 0.515 0.540 strong_typed_link_match 0.130 0.265 0.175 strong_typed_nil_match 0.558 0.570 0.564 entity_match ONLY Yoruba documents: Prec Recall F1 Metric 0.531 0.437 0.480 strong_mention_match 0.475 0.391 0.429 strong_typed_mention_match 0.320 0.264 0.289 strong_typed_all_match 0.422 0.317 0.362 strong_typed_link_match 0.070 0.076 0.073 strong_typed_nil_match 0.443 0.428 0.435 entity_match