======================================================= TAC KBP 2016 SPANISH KB CONSTRUCTION EVALUATION RESULTS ======================================================= Team ID: UMass_IESL Organization: College of Information and Computer Sciences, University of Massachusetts Amherst ************************************************************* Run ID: UMass_IESL_KB_SPA_1 Did the run access the live Web during the evaluation window: No Did the run extract relations from the Cold Start source corpus: Yes Did the run generate meaningful confidence values: Yes Entity Discovery Evaluation: ONLY Spanish documents: Prec Recall F1 Metric 0.840 0.358 0.502 strong_mention_match 0.662 0.282 0.396 strong_typed_mention_match 0.000 0.000 0.000 entity_match 0.801 0.176 0.289 b_cubed 0.701 0.298 0.418 mention_ceaf 0.613 0.261 0.366 typed_mention_ceaf Slot Filling Evaluation: Metric RunID Hop Prec Recall F1 SF-ALL-Micro UMass_IESL_KB_SPA_1 0 0.2785 0.0663 0.1071 SF-ALL-Micro UMass_IESL_KB_SPA_1 1 0.0000 0.0000 0.0000 SF-ALL-Micro UMass_IESL_KB_SPA_1 ALL 0.1366 0.0404 0.0623 SF-ALL-Macro UMass_IESL_KB_SPA_1 0 0.0726 0.0704 0.0613 SF-ALL-Macro UMass_IESL_KB_SPA_1 1 0.0000 0.0000 0.0000 SF-ALL-Macro UMass_IESL_KB_SPA_1 ALL 0.0484 0.0469 0.0409 LDC-MAX-ALL-Micro UMass_IESL_KB_SPA_1 0 0.2698 0.0683 0.1090 LDC-MAX-ALL-Micro UMass_IESL_KB_SPA_1 1 0.0000 0.0000 0.0000 LDC-MAX-ALL-Micro UMass_IESL_KB_SPA_1 ALL 0.1288 0.0453 0.0671 LDC-MAX-ALL-Macro UMass_IESL_KB_SPA_1 0 0.0701 0.0706 0.0592 LDC-MAX-ALL-Macro UMass_IESL_KB_SPA_1 1 0.0000 0.0000 0.0000 LDC-MAX-ALL-Macro UMass_IESL_KB_SPA_1 ALL 0.0471 0.0474 0.0398 LDC-MEAN-ALL-Macro UMass_IESL_KB_SPA_1 0 0.0641 0.0683 0.0560 LDC-MEAN-ALL-Macro UMass_IESL_KB_SPA_1 1 0.0000 0.0000 0.0000 LDC-MEAN-ALL-Macro UMass_IESL_KB_SPA_1 ALL 0.0431 0.0459 0.0376 *ALL-Macro Prec, Recall and F1 refer to mean-precision, mean-recall and mean-F1. NIL-DETECTION P/R/F1: 0.3085 0.9756 0.4688 ************************************************************* Run ID: UMass_IESL_KB_SPA_2 Did the run access the live Web during the evaluation window: No Did the run extract relations from the Cold Start source corpus: Yes Did the run generate meaningful confidence values: Yes Entity Discovery Evaluation: ONLY Spanish documents: Prec Recall F1 Metric 0.840 0.358 0.502 strong_mention_match 0.662 0.282 0.396 strong_typed_mention_match 0.000 0.000 0.000 entity_match 0.801 0.176 0.289 b_cubed 0.701 0.298 0.418 mention_ceaf 0.613 0.261 0.366 typed_mention_ceaf Slot Filling Evaluation: Metric RunID Hop Prec Recall F1 SF-ALL-Micro UMass_IESL_KB_SPA_2 0 0.2908 0.1235 0.1734 SF-ALL-Micro UMass_IESL_KB_SPA_2 1 0.0376 0.0329 0.0351 SF-ALL-Micro UMass_IESL_KB_SPA_2 ALL 0.1468 0.0881 0.1101 SF-ALL-Macro UMass_IESL_KB_SPA_2 0 0.1044 0.1429 0.1056 SF-ALL-Macro UMass_IESL_KB_SPA_2 1 0.0215 0.0404 0.0250 SF-ALL-Macro UMass_IESL_KB_SPA_2 ALL 0.0768 0.1087 0.0787 LDC-MAX-ALL-Micro UMass_IESL_KB_SPA_2 0 0.2743 0.1245 0.1713 LDC-MAX-ALL-Micro UMass_IESL_KB_SPA_2 1 0.0409 0.0556 0.0471 LDC-MAX-ALL-Micro UMass_IESL_KB_SPA_2 ALL 0.1338 0.1013 0.1153 LDC-MAX-ALL-Macro UMass_IESL_KB_SPA_2 0 0.1056 0.1395 0.1056 LDC-MAX-ALL-Macro UMass_IESL_KB_SPA_2 1 0.0306 0.0576 0.0357 LDC-MAX-ALL-Macro UMass_IESL_KB_SPA_2 ALL 0.0810 0.1126 0.0827 LDC-MEAN-ALL-Macro UMass_IESL_KB_SPA_2 0 0.0986 0.1329 0.0990 LDC-MEAN-ALL-Macro UMass_IESL_KB_SPA_2 1 0.0153 0.0288 0.0179 LDC-MEAN-ALL-Macro UMass_IESL_KB_SPA_2 ALL 0.0713 0.0987 0.0724 *ALL-Macro Prec, Recall and F1 refer to mean-precision, mean-recall and mean-F1. NIL-DETECTION P/R/F1: 0.3067 0.9675 0.4658 ************************************************************* Run ID: UMass_IESL_KB_SPA_3 Did the run access the live Web during the evaluation window: No Did the run extract relations from the Cold Start source corpus: Yes Did the run generate meaningful confidence values: Yes Entity Discovery Evaluation: ONLY Spanish documents: Prec Recall F1 Metric 0.840 0.358 0.502 strong_mention_match 0.662 0.282 0.396 strong_typed_mention_match 0.000 0.000 0.000 entity_match 0.800 0.176 0.288 b_cubed 0.698 0.297 0.417 mention_ceaf 0.612 0.261 0.366 typed_mention_ceaf Slot Filling Evaluation: Metric RunID Hop Prec Recall F1 SF-ALL-Micro UMass_IESL_KB_SPA_3 0 0.2955 0.1175 0.1681 SF-ALL-Micro UMass_IESL_KB_SPA_3 1 0.0513 0.0282 0.0364 SF-ALL-Micro UMass_IESL_KB_SPA_3 ALL 0.1807 0.0826 0.1134 SF-ALL-Macro UMass_IESL_KB_SPA_3 0 0.1008 0.1326 0.1001 SF-ALL-Macro UMass_IESL_KB_SPA_3 1 0.0120 0.0301 0.0156 SF-ALL-Macro UMass_IESL_KB_SPA_3 ALL 0.0712 0.0984 0.0719 LDC-MAX-ALL-Micro UMass_IESL_KB_SPA_3 0 0.2804 0.1205 0.1685 LDC-MAX-ALL-Micro UMass_IESL_KB_SPA_3 1 0.0600 0.0476 0.0531 LDC-MAX-ALL-Micro UMass_IESL_KB_SPA_3 ALL 0.1739 0.0960 0.1237 LDC-MAX-ALL-Macro UMass_IESL_KB_SPA_3 0 0.1042 0.1323 0.1028 LDC-MAX-ALL-Macro UMass_IESL_KB_SPA_3 1 0.0172 0.0429 0.0223 LDC-MAX-ALL-Macro UMass_IESL_KB_SPA_3 ALL 0.0756 0.1029 0.0763 LDC-MEAN-ALL-Macro UMass_IESL_KB_SPA_3 0 0.0966 0.1257 0.0958 LDC-MEAN-ALL-Macro UMass_IESL_KB_SPA_3 1 0.0086 0.0214 0.0111 LDC-MEAN-ALL-Macro UMass_IESL_KB_SPA_3 ALL 0.0677 0.0914 0.0680 *ALL-Macro Prec, Recall and F1 refer to mean-precision, mean-recall and mean-F1. NIL-DETECTION P/R/F1: 0.3067 0.9675 0.4658 ************************************************************* Run ID: UMass_IESL_KB_SPA_4 Did the run access the live Web during the evaluation window: No Did the run extract relations from the Cold Start source corpus: Yes Did the run generate meaningful confidence values: Yes Entity Discovery Evaluation: ONLY Spanish documents: Prec Recall F1 Metric 0.840 0.358 0.502 strong_mention_match 0.662 0.282 0.396 strong_typed_mention_match 0.000 0.000 0.000 entity_match 0.801 0.176 0.289 b_cubed 0.701 0.298 0.418 mention_ceaf 0.613 0.261 0.366 typed_mention_ceaf Slot Filling Evaluation: Metric RunID Hop Prec Recall F1 SF-ALL-Micro UMass_IESL_KB_SPA_4 0 0.2785 0.0663 0.1071 SF-ALL-Micro UMass_IESL_KB_SPA_4 1 0.0000 0.0000 0.0000 SF-ALL-Micro UMass_IESL_KB_SPA_4 ALL 0.1366 0.0404 0.0623 SF-ALL-Macro UMass_IESL_KB_SPA_4 0 0.0726 0.0704 0.0613 SF-ALL-Macro UMass_IESL_KB_SPA_4 1 0.0000 0.0000 0.0000 SF-ALL-Macro UMass_IESL_KB_SPA_4 ALL 0.0484 0.0469 0.0409 LDC-MAX-ALL-Micro UMass_IESL_KB_SPA_4 0 0.2742 0.0683 0.1093 LDC-MAX-ALL-Micro UMass_IESL_KB_SPA_4 1 0.0000 0.0000 0.0000 LDC-MAX-ALL-Micro UMass_IESL_KB_SPA_4 ALL 0.1298 0.0453 0.0672 LDC-MAX-ALL-Macro UMass_IESL_KB_SPA_4 0 0.0701 0.0706 0.0592 LDC-MAX-ALL-Macro UMass_IESL_KB_SPA_4 1 0.0000 0.0000 0.0000 LDC-MAX-ALL-Macro UMass_IESL_KB_SPA_4 ALL 0.0471 0.0474 0.0398 LDC-MEAN-ALL-Macro UMass_IESL_KB_SPA_4 0 0.0641 0.0683 0.0560 LDC-MEAN-ALL-Macro UMass_IESL_KB_SPA_4 1 0.0000 0.0000 0.0000 LDC-MEAN-ALL-Macro UMass_IESL_KB_SPA_4 ALL 0.0431 0.0459 0.0376 *ALL-Macro Prec, Recall and F1 refer to mean-precision, mean-recall and mean-F1. NIL-DETECTION P/R/F1: 0.3085 0.9756 0.4688 ************************************************************* Run ID: UMass_IESL_KB_SPA_5 Did the run access the live Web during the evaluation window: No Did the run extract relations from the Cold Start source corpus: Yes Did the run generate meaningful confidence values: Yes Entity Discovery Evaluation: ONLY Spanish documents: Prec Recall F1 Metric 0.840 0.358 0.502 strong_mention_match 0.662 0.282 0.396 strong_typed_mention_match 0.000 0.000 0.000 entity_match 0.800 0.176 0.288 b_cubed 0.698 0.297 0.417 mention_ceaf 0.612 0.261 0.366 typed_mention_ceaf Slot Filling Evaluation: Metric RunID Hop Prec Recall F1 SF-ALL-Micro UMass_IESL_KB_SPA_5 0 0.2917 0.0633 0.1040 SF-ALL-Micro UMass_IESL_KB_SPA_5 1 0.0000 0.0000 0.0000 SF-ALL-Micro UMass_IESL_KB_SPA_5 ALL 0.2039 0.0385 0.0648 SF-ALL-Macro UMass_IESL_KB_SPA_5 0 0.0777 0.0678 0.0640 SF-ALL-Macro UMass_IESL_KB_SPA_5 1 0.0000 0.0000 0.0000 SF-ALL-Macro UMass_IESL_KB_SPA_5 ALL 0.0518 0.0452 0.0427 LDC-MAX-ALL-Micro UMass_IESL_KB_SPA_5 0 0.2963 0.0643 0.1056 LDC-MAX-ALL-Micro UMass_IESL_KB_SPA_5 1 0.0000 0.0000 0.0000 LDC-MAX-ALL-Micro UMass_IESL_KB_SPA_5 ALL 0.2192 0.0427 0.0714 LDC-MAX-ALL-Macro UMass_IESL_KB_SPA_5 0 0.0773 0.0670 0.0630 LDC-MAX-ALL-Macro UMass_IESL_KB_SPA_5 1 0.0000 0.0000 0.0000 LDC-MAX-ALL-Macro UMass_IESL_KB_SPA_5 ALL 0.0519 0.0450 0.0423 LDC-MEAN-ALL-Macro UMass_IESL_KB_SPA_5 0 0.0701 0.0647 0.0595 LDC-MEAN-ALL-Macro UMass_IESL_KB_SPA_5 1 0.0000 0.0000 0.0000 LDC-MEAN-ALL-Macro UMass_IESL_KB_SPA_5 ALL 0.0471 0.0434 0.0399 *ALL-Macro Prec, Recall and F1 refer to mean-precision, mean-recall and mean-F1. NIL-DETECTION P/R/F1: 0.3103 0.9837 0.4718