============================================================= TAC KBP 2016 CROSS-LINGUAL KB CONSTRUCTION EVALUATION RESULTS ============================================================= Team ID: UMass_IESL Organization: College of Information and Computer Sciences, University of Massachusetts Amherst ************************************************************* Run ID: UMass_IESL_KB_XLING_1 Did the run access the live Web during the evaluation window: No Did the run extract relations from the Cold Start source corpus: Yes Did the run generate meaningful confidence values: Yes Run number of the English KB system that is most closely configured to the English component of this run: 1 Run number of the Spanish KB system that is most closely configured to the Spanish component of this run: 1 Run number of the Chinese KB system that is most closely configured to the Chinese component of this run: NA Entity Discovery Evaluation: ALL English, Chinese, and Spanish documents: Prec Recall F1 Metric 0.860 0.250 0.388 strong_mention_match 0.755 0.219 0.340 strong_typed_mention_match 0.000 0.000 0.000 entity_match 0.824 0.098 0.175 b_cubed 0.714 0.208 0.322 mention_ceaf 0.671 0.195 0.302 typed_mention_ceaf ONLY English documents: Prec Recall F1 Metric 0.874 0.409 0.557 strong_mention_match 0.818 0.382 0.521 strong_typed_mention_match 0.000 0.000 0.000 entity_match 0.845 0.233 0.366 b_cubed 0.787 0.368 0.501 mention_ceaf 0.752 0.352 0.479 typed_mention_ceaf ONLY Chinese documents: Prec Recall F1 Metric 0.000 0.000 0.000 strong_mention_match 0.000 0.000 0.000 strong_typed_mention_match 0.000 0.000 0.000 entity_match 0.000 0.000 0.000 b_cubed 0.000 0.000 0.000 mention_ceaf 0.000 0.000 0.000 typed_mention_ceaf ONLY Spanish documents: Prec Recall F1 Metric 0.840 0.358 0.502 strong_mention_match 0.662 0.282 0.396 strong_typed_mention_match 0.000 0.000 0.000 entity_match 0.801 0.176 0.289 b_cubed 0.701 0.298 0.418 mention_ceaf 0.613 0.261 0.366 typed_mention_ceaf Slot Filling Evaluation: Metric RunID Hop Prec Recall F1 SF-ALL-Micro UMass_IESL_KB_XLING_1 0 0.2620 0.0481 0.0813 SF-ALL-Micro UMass_IESL_KB_XLING_1 1 0.0371 0.0260 0.0305 SF-ALL-Micro UMass_IESL_KB_XLING_1 ALL 0.1165 0.0409 0.0606 SF-ALL-Macro UMass_IESL_KB_XLING_1 0 0.0919 0.0849 0.0752 SF-ALL-Macro UMass_IESL_KB_XLING_1 1 0.0242 0.0258 0.0234 SF-ALL-Macro UMass_IESL_KB_XLING_1 ALL 0.0573 0.0546 0.0487 LDC-MAX-ALL-Micro UMass_IESL_KB_XLING_1 0 0.2958 0.0935 0.1420 LDC-MAX-ALL-Micro UMass_IESL_KB_XLING_1 1 0.0510 0.0377 0.0434 LDC-MAX-ALL-Micro UMass_IESL_KB_XLING_1 ALL 0.1633 0.0748 0.1026 LDC-MAX-ALL-Macro UMass_IESL_KB_XLING_1 0 0.1710 0.1512 0.1370 LDC-MAX-ALL-Macro UMass_IESL_KB_XLING_1 1 0.0405 0.0406 0.0379 LDC-MAX-ALL-Macro UMass_IESL_KB_XLING_1 ALL 0.1065 0.0965 0.0880 LDC-MEAN-ALL-Macro UMass_IESL_KB_XLING_1 0 0.0900 0.0881 0.0769 LDC-MEAN-ALL-Macro UMass_IESL_KB_XLING_1 1 0.0233 0.0246 0.0226 LDC-MEAN-ALL-Macro UMass_IESL_KB_XLING_1 ALL 0.0570 0.0567 0.0500 *ALL-Macro Prec, Recall and F1 refer to mean-precision, mean-recall and mean-F1. NIL-DETECTION P/R/F1: 0.2807 0.8537 0.4225 ************************************************************* Run ID: UMass_IESL_KB_XLING_2 Did the run access the live Web during the evaluation window: No Did the run extract relations from the Cold Start source corpus: Yes Did the run generate meaningful confidence values: Yes Run number of the English KB system that is most closely configured to the English component of this run: 4 Run number of the Spanish KB system that is most closely configured to the Spanish component of this run: 2 Run number of the Chinese KB system that is most closely configured to the Chinese component of this run: NA Entity Discovery Evaluation: ALL English, Chinese, and Spanish documents: Prec Recall F1 Metric 0.860 0.250 0.388 strong_mention_match 0.755 0.219 0.340 strong_typed_mention_match 0.000 0.000 0.000 entity_match 0.824 0.098 0.175 b_cubed 0.714 0.208 0.322 mention_ceaf 0.671 0.195 0.302 typed_mention_ceaf ONLY English documents: Prec Recall F1 Metric 0.874 0.409 0.557 strong_mention_match 0.818 0.382 0.521 strong_typed_mention_match 0.000 0.000 0.000 entity_match 0.845 0.233 0.366 b_cubed 0.787 0.368 0.501 mention_ceaf 0.752 0.352 0.479 typed_mention_ceaf ONLY Chinese documents: Prec Recall F1 Metric 0.000 0.000 0.000 strong_mention_match 0.000 0.000 0.000 strong_typed_mention_match 0.000 0.000 0.000 entity_match 0.000 0.000 0.000 b_cubed 0.000 0.000 0.000 mention_ceaf 0.000 0.000 0.000 typed_mention_ceaf ONLY Spanish documents: Prec Recall F1 Metric 0.840 0.358 0.502 strong_mention_match 0.662 0.282 0.396 strong_typed_mention_match 0.000 0.000 0.000 entity_match 0.801 0.176 0.289 b_cubed 0.701 0.298 0.418 mention_ceaf 0.613 0.261 0.366 typed_mention_ceaf Slot Filling Evaluation: Metric RunID Hop Prec Recall F1 SF-ALL-Micro UMass_IESL_KB_XLING_2 0 0.2556 0.0445 0.0757 SF-ALL-Micro UMass_IESL_KB_XLING_2 1 0.0714 0.0265 0.0386 SF-ALL-Micro UMass_IESL_KB_XLING_2 ALL 0.1625 0.0386 0.0624 SF-ALL-Macro UMass_IESL_KB_XLING_2 0 0.0899 0.0907 0.0778 SF-ALL-Macro UMass_IESL_KB_XLING_2 1 0.0219 0.0257 0.0218 SF-ALL-Macro UMass_IESL_KB_XLING_2 ALL 0.0551 0.0574 0.0491 LDC-MAX-ALL-Micro UMass_IESL_KB_XLING_2 0 0.2658 0.0835 0.1271 LDC-MAX-ALL-Micro UMass_IESL_KB_XLING_2 1 0.0708 0.0410 0.0519 LDC-MAX-ALL-Micro UMass_IESL_KB_XLING_2 ALL 0.1719 0.0693 0.0987 LDC-MAX-ALL-Macro UMass_IESL_KB_XLING_2 0 0.1678 0.1593 0.1390 LDC-MAX-ALL-Macro UMass_IESL_KB_XLING_2 1 0.0347 0.0364 0.0328 LDC-MAX-ALL-Macro UMass_IESL_KB_XLING_2 ALL 0.1020 0.0985 0.0865 LDC-MEAN-ALL-Macro UMass_IESL_KB_XLING_2 0 0.0903 0.0974 0.0806 LDC-MEAN-ALL-Macro UMass_IESL_KB_XLING_2 1 0.0210 0.0247 0.0209 LDC-MEAN-ALL-Macro UMass_IESL_KB_XLING_2 ALL 0.0560 0.0614 0.0511 *ALL-Macro Prec, Recall and F1 refer to mean-precision, mean-recall and mean-F1. NIL-DETECTION P/R/F1: 0.2902 0.8943 0.4382 ************************************************************* Run ID: UMass_IESL_KB_XLING_3 Did the run access the live Web during the evaluation window: No Did the run extract relations from the Cold Start source corpus: Yes Did the run generate meaningful confidence values: Yes Run number of the English KB system that is most closely configured to the English component of this run: 5 Run number of the Spanish KB system that is most closely configured to the Spanish component of this run: 3 Run number of the Chinese KB system that is most closely configured to the Chinese component of this run: NA Entity Discovery Evaluation: ALL English, Chinese, and Spanish documents: Prec Recall F1 Metric 0.860 0.250 0.388 strong_mention_match 0.755 0.219 0.340 strong_typed_mention_match 0.000 0.000 0.000 entity_match 0.824 0.098 0.174 b_cubed 0.713 0.207 0.321 mention_ceaf 0.671 0.195 0.302 typed_mention_ceaf ONLY English documents: Prec Recall F1 Metric 0.874 0.409 0.557 strong_mention_match 0.818 0.382 0.521 strong_typed_mention_match 0.000 0.000 0.000 entity_match 0.845 0.233 0.366 b_cubed 0.787 0.368 0.501 mention_ceaf 0.752 0.352 0.479 typed_mention_ceaf ONLY Chinese documents: Prec Recall F1 Metric 0.000 0.000 0.000 strong_mention_match 0.000 0.000 0.000 strong_typed_mention_match 0.000 0.000 0.000 entity_match 0.000 0.000 0.000 b_cubed 0.000 0.000 0.000 mention_ceaf 0.000 0.000 0.000 typed_mention_ceaf ONLY Spanish documents: Prec Recall F1 Metric 0.840 0.358 0.502 strong_mention_match 0.662 0.282 0.396 strong_typed_mention_match 0.000 0.000 0.000 entity_match 0.800 0.176 0.288 b_cubed 0.698 0.297 0.417 mention_ceaf 0.612 0.261 0.366 typed_mention_ceaf Slot Filling Evaluation: Metric RunID Hop Prec Recall F1 SF-ALL-Micro UMass_IESL_KB_XLING_3 0 0.2314 0.0579 0.0926 SF-ALL-Micro UMass_IESL_KB_XLING_3 1 0.0215 0.0387 0.0276 SF-ALL-Micro UMass_IESL_KB_XLING_3 ALL 0.0687 0.0517 0.0590 SF-ALL-Macro UMass_IESL_KB_XLING_3 0 0.0965 0.0985 0.0835 SF-ALL-Macro UMass_IESL_KB_XLING_3 1 0.0323 0.0373 0.0325 SF-ALL-Macro UMass_IESL_KB_XLING_3 ALL 0.0637 0.0672 0.0574 LDC-MAX-ALL-Micro UMass_IESL_KB_XLING_3 0 0.2753 0.1125 0.1597 LDC-MAX-ALL-Micro UMass_IESL_KB_XLING_3 1 0.0723 0.0557 0.0630 LDC-MAX-ALL-Micro UMass_IESL_KB_XLING_3 ALL 0.1763 0.0935 0.1222 LDC-MAX-ALL-Macro UMass_IESL_KB_XLING_3 0 0.1804 0.1778 0.1542 LDC-MAX-ALL-Macro UMass_IESL_KB_XLING_3 1 0.0469 0.0552 0.0462 LDC-MAX-ALL-Macro UMass_IESL_KB_XLING_3 ALL 0.1144 0.1172 0.1008 LDC-MEAN-ALL-Macro UMass_IESL_KB_XLING_3 0 0.0954 0.1021 0.0856 LDC-MEAN-ALL-Macro UMass_IESL_KB_XLING_3 1 0.0287 0.0338 0.0290 LDC-MEAN-ALL-Macro UMass_IESL_KB_XLING_3 ALL 0.0624 0.0683 0.0576 *ALL-Macro Prec, Recall and F1 refer to mean-precision, mean-recall and mean-F1. NIL-DETECTION P/R/F1: 0.2670 0.7967 0.4000 ************************************************************* Run ID: UMass_IESL_KB_XLING_4 Did the run access the live Web during the evaluation window: No Did the run extract relations from the Cold Start source corpus: Yes Did the run generate meaningful confidence values: Yes Run number of the English KB system that is most closely configured to the English component of this run: 2 Run number of the Spanish KB system that is most closely configured to the Spanish component of this run: 1 Run number of the Chinese KB system that is most closely configured to the Chinese component of this run: NA Entity Discovery Evaluation: ALL English, Chinese, and Spanish documents: Prec Recall F1 Metric 0.860 0.250 0.388 strong_mention_match 0.755 0.219 0.340 strong_typed_mention_match 0.000 0.000 0.000 entity_match 0.824 0.098 0.175 b_cubed 0.714 0.208 0.322 mention_ceaf 0.671 0.195 0.302 typed_mention_ceaf ONLY English documents: Prec Recall F1 Metric 0.874 0.409 0.557 strong_mention_match 0.818 0.382 0.521 strong_typed_mention_match 0.000 0.000 0.000 entity_match 0.845 0.233 0.366 b_cubed 0.787 0.368 0.501 mention_ceaf 0.752 0.352 0.479 typed_mention_ceaf ONLY Chinese documents: Prec Recall F1 Metric 0.000 0.000 0.000 strong_mention_match 0.000 0.000 0.000 strong_typed_mention_match 0.000 0.000 0.000 entity_match 0.000 0.000 0.000 b_cubed 0.000 0.000 0.000 mention_ceaf 0.000 0.000 0.000 typed_mention_ceaf ONLY Spanish documents: Prec Recall F1 Metric 0.840 0.358 0.502 strong_mention_match 0.662 0.282 0.396 strong_typed_mention_match 0.000 0.000 0.000 entity_match 0.801 0.176 0.289 b_cubed 0.701 0.298 0.418 mention_ceaf 0.613 0.261 0.366 typed_mention_ceaf Slot Filling Evaluation: Metric RunID Hop Prec Recall F1 SF-ALL-Micro UMass_IESL_KB_XLING_4 0 0.3953 0.0166 0.0319 SF-ALL-Micro UMass_IESL_KB_XLING_4 1 0.1600 0.0041 0.0079 SF-ALL-Micro UMass_IESL_KB_XLING_4 ALL 0.3423 0.0125 0.0242 SF-ALL-Macro UMass_IESL_KB_XLING_4 0 0.0558 0.0363 0.0390 SF-ALL-Macro UMass_IESL_KB_XLING_4 1 0.0049 0.0031 0.0035 SF-ALL-Macro UMass_IESL_KB_XLING_4 ALL 0.0298 0.0193 0.0209 LDC-MAX-ALL-Micro UMass_IESL_KB_XLING_4 0 0.5278 0.0314 0.0593 LDC-MAX-ALL-Micro UMass_IESL_KB_XLING_4 1 0.1667 0.0066 0.0126 LDC-MAX-ALL-Micro UMass_IESL_KB_XLING_4 ALL 0.4375 0.0231 0.0439 LDC-MAX-ALL-Macro UMass_IESL_KB_XLING_4 0 0.0977 0.0645 0.0699 LDC-MAX-ALL-Macro UMass_IESL_KB_XLING_4 1 0.0082 0.0053 0.0059 LDC-MAX-ALL-Macro UMass_IESL_KB_XLING_4 ALL 0.0534 0.0352 0.0382 LDC-MEAN-ALL-Macro UMass_IESL_KB_XLING_4 0 0.0553 0.0410 0.0430 LDC-MEAN-ALL-Macro UMass_IESL_KB_XLING_4 1 0.0067 0.0047 0.0051 LDC-MEAN-ALL-Macro UMass_IESL_KB_XLING_4 ALL 0.0313 0.0231 0.0243 *ALL-Macro Prec, Recall and F1 refer to mean-precision, mean-recall and mean-F1. NIL-DETECTION P/R/F1: 0.3085 0.9756 0.4688 ************************************************************* Run ID: UMass_IESL_KB_XLING_5 Did the run access the live Web during the evaluation window: No Did the run extract relations from the Cold Start source corpus: Yes Did the run generate meaningful confidence values: Yes Run number of the English KB system that is most closely configured to the English component of this run: 2 Run number of the Spanish KB system that is most closely configured to the Spanish component of this run: 1 Run number of the Chinese KB system that is most closely configured to the Chinese component of this run: NA Entity Discovery Evaluation: ALL English, Chinese, and Spanish documents: Prec Recall F1 Metric 0.860 0.250 0.388 strong_mention_match 0.755 0.219 0.340 strong_typed_mention_match 0.000 0.000 0.000 entity_match 0.824 0.098 0.175 b_cubed 0.714 0.208 0.322 mention_ceaf 0.671 0.195 0.302 typed_mention_ceaf ONLY English documents: Prec Recall F1 Metric 0.874 0.409 0.557 strong_mention_match 0.818 0.382 0.521 strong_typed_mention_match 0.000 0.000 0.000 entity_match 0.845 0.233 0.366 b_cubed 0.787 0.368 0.501 mention_ceaf 0.752 0.352 0.479 typed_mention_ceaf ONLY Chinese documents: Prec Recall F1 Metric 0.000 0.000 0.000 strong_mention_match 0.000 0.000 0.000 strong_typed_mention_match 0.000 0.000 0.000 entity_match 0.000 0.000 0.000 b_cubed 0.000 0.000 0.000 mention_ceaf 0.000 0.000 0.000 typed_mention_ceaf ONLY Spanish documents: Prec Recall F1 Metric 0.840 0.358 0.502 strong_mention_match 0.662 0.282 0.396 strong_typed_mention_match 0.000 0.000 0.000 entity_match 0.801 0.176 0.289 b_cubed 0.701 0.298 0.418 mention_ceaf 0.613 0.261 0.366 typed_mention_ceaf Slot Filling Evaluation: Metric RunID Hop Prec Recall F1 SF-ALL-Micro UMass_IESL_KB_XLING_5 0 0.3132 0.0393 0.0699 SF-ALL-Micro UMass_IESL_KB_XLING_5 1 0.0669 0.0188 0.0294 SF-ALL-Micro UMass_IESL_KB_XLING_5 ALL 0.1856 0.0327 0.0556 SF-ALL-Macro UMass_IESL_KB_XLING_5 0 0.0906 0.0733 0.0683 SF-ALL-Macro UMass_IESL_KB_XLING_5 1 0.0204 0.0222 0.0197 SF-ALL-Macro UMass_IESL_KB_XLING_5 ALL 0.0547 0.0471 0.0434 LDC-MAX-ALL-Micro UMass_IESL_KB_XLING_5 0 0.3457 0.0769 0.1258 LDC-MAX-ALL-Micro UMass_IESL_KB_XLING_5 1 0.0874 0.0295 0.0441 LDC-MAX-ALL-Micro UMass_IESL_KB_XLING_5 ALL 0.2337 0.0610 0.0968 LDC-MAX-ALL-Macro UMass_IESL_KB_XLING_5 0 0.1571 0.1244 0.1187 LDC-MAX-ALL-Macro UMass_IESL_KB_XLING_5 1 0.0335 0.0345 0.0311 LDC-MAX-ALL-Macro UMass_IESL_KB_XLING_5 ALL 0.0959 0.0799 0.0753 LDC-MEAN-ALL-Macro UMass_IESL_KB_XLING_5 0 0.0816 0.0707 0.0648 LDC-MEAN-ALL-Macro UMass_IESL_KB_XLING_5 1 0.0213 0.0225 0.0204 LDC-MEAN-ALL-Macro UMass_IESL_KB_XLING_5 ALL 0.0518 0.0468 0.0428 *ALL-Macro Prec, Recall and F1 refer to mean-precision, mean-recall and mean-F1. NIL-DETECTION P/R/F1: 0.2902 0.8943 0.4382