========================================================== TAC KBP 2016 CROSS-LINGUAL SLOT FILLING EVALUATION RESULTS ========================================================== Team ID: UMass_IESL Organization: College of Information and Computer Sciences, University of Massachusetts Amherst ************************************************************* Run ID: UMass_IESL_SF_XLING_1 Did the run access the live Web during the evaluation window: No Did the run extract relations from the Cold Start source corpus: Yes Did the run generate meaningful confidence values: Yes Run number of the English SF system that is most closely configured to the English component of this run: 1 Run number of the Spanish SF system that is most closely configured to the Spanish component of this run: 1 Run number of the Chinese SF system that is most closely configured to the Chinese component of this run: NA Slot Filling Evaluation: Metric RunID Hop Prec Recall F1 SF-ALL-Micro UMass_IESL_SF_XLING_1 0 0.1606 0.0933 0.1180 SF-ALL-Micro UMass_IESL_SF_XLING_1 1 0.2143 0.0046 0.0090 SF-ALL-Micro UMass_IESL_SF_XLING_1 ALL 0.1615 0.0645 0.0922 SF-ALL-Macro UMass_IESL_SF_XLING_1 0 0.1365 0.1592 0.1282 SF-ALL-Macro UMass_IESL_SF_XLING_1 1 0.0066 0.0073 0.0069 SF-ALL-Macro UMass_IESL_SF_XLING_1 ALL 0.0701 0.0815 0.0661 LDC-MAX-ALL-Micro UMass_IESL_SF_XLING_1 0 0.1829 0.1555 0.1681 LDC-MAX-ALL-Micro UMass_IESL_SF_XLING_1 1 0.2857 0.0066 0.0128 LDC-MAX-ALL-Micro UMass_IESL_SF_XLING_1 ALL 0.1843 0.1056 0.1342 LDC-MAX-ALL-Macro UMass_IESL_SF_XLING_1 0 0.2114 0.2475 0.2007 LDC-MAX-ALL-Macro UMass_IESL_SF_XLING_1 1 0.0094 0.0106 0.0099 LDC-MAX-ALL-Macro UMass_IESL_SF_XLING_1 ALL 0.1115 0.1303 0.1063 LDC-MEAN-ALL-Macro UMass_IESL_SF_XLING_1 0 0.1356 0.1617 0.1301 LDC-MEAN-ALL-Macro UMass_IESL_SF_XLING_1 1 0.0068 0.0080 0.0073 LDC-MEAN-ALL-Macro UMass_IESL_SF_XLING_1 ALL 0.0719 0.0856 0.0693 *ALL-Macro Prec, Recall and F1 refer to mean-precision, mean-recall and mean-F1. NIL-DETECTION P/R/F1: 0.2111 0.5854 0.3103 ************************************************************* Run ID: UMass_IESL_SF_XLING_2 Did the run access the live Web during the evaluation window: No Did the run extract relations from the Cold Start source corpus: Yes Did the run generate meaningful confidence values: Yes Run number of the English SF system that is most closely configured to the English component of this run: 2 Run number of the Spanish SF system that is most closely configured to the Spanish component of this run: 2 Run number of the Chinese SF system that is most closely configured to the Chinese component of this run: NA Slot Filling Evaluation: Metric RunID Hop Prec Recall F1 SF-ALL-Micro UMass_IESL_SF_XLING_2 0 0.1743 0.0840 0.1134 SF-ALL-Micro UMass_IESL_SF_XLING_2 1 0.1477 0.0066 0.0127 SF-ALL-Micro UMass_IESL_SF_XLING_2 ALL 0.1731 0.0589 0.0879 SF-ALL-Macro UMass_IESL_SF_XLING_2 0 0.1282 0.1380 0.1176 SF-ALL-Macro UMass_IESL_SF_XLING_2 1 0.0114 0.0114 0.0114 SF-ALL-Macro UMass_IESL_SF_XLING_2 ALL 0.0685 0.0733 0.0633 LDC-MAX-ALL-Micro UMass_IESL_SF_XLING_2 0 0.1876 0.1423 0.1618 LDC-MAX-ALL-Micro UMass_IESL_SF_XLING_2 1 0.1304 0.0098 0.0183 LDC-MAX-ALL-Micro UMass_IESL_SF_XLING_2 ALL 0.1848 0.0979 0.1280 LDC-MAX-ALL-Macro UMass_IESL_SF_XLING_2 0 0.2025 0.2144 0.1853 LDC-MAX-ALL-Macro UMass_IESL_SF_XLING_2 1 0.0176 0.0176 0.0176 LDC-MAX-ALL-Macro UMass_IESL_SF_XLING_2 ALL 0.1110 0.1170 0.1023 LDC-MEAN-ALL-Macro UMass_IESL_SF_XLING_2 0 0.1250 0.1388 0.1175 LDC-MEAN-ALL-Macro UMass_IESL_SF_XLING_2 1 0.0133 0.0133 0.0133 LDC-MEAN-ALL-Macro UMass_IESL_SF_XLING_2 ALL 0.0697 0.0767 0.0659 *ALL-Macro Prec, Recall and F1 refer to mean-precision, mean-recall and mean-F1. NIL-DETECTION P/R/F1: 0.2358 0.6748 0.3495 ************************************************************* Run ID: UMass_IESL_SF_XLING_3 Did the run access the live Web during the evaluation window: No Did the run extract relations from the Cold Start source corpus: Yes Did the run generate meaningful confidence values: Yes Run number of the English SF system that is most closely configured to the English component of this run: 1 Run number of the Spanish SF system that is most closely configured to the Spanish component of this run: 3 Run number of the Chinese SF system that is most closely configured to the Chinese component of this run: NA Slot Filling Evaluation: Metric RunID Hop Prec Recall F1 SF-ALL-Micro UMass_IESL_SF_XLING_3 0 0.0655 0.1041 0.0804 SF-ALL-Micro UMass_IESL_SF_XLING_3 1 0.2143 0.0046 0.0090 SF-ALL-Micro UMass_IESL_SF_XLING_3 ALL 0.0665 0.0718 0.0690 SF-ALL-Macro UMass_IESL_SF_XLING_3 0 0.1203 0.1561 0.1181 SF-ALL-Macro UMass_IESL_SF_XLING_3 1 0.0066 0.0073 0.0069 SF-ALL-Macro UMass_IESL_SF_XLING_3 ALL 0.0621 0.0800 0.0612 LDC-MAX-ALL-Micro UMass_IESL_SF_XLING_3 0 0.0786 0.1803 0.1095 LDC-MAX-ALL-Micro UMass_IESL_SF_XLING_3 1 0.2000 0.0066 0.0127 LDC-MAX-ALL-Micro UMass_IESL_SF_XLING_3 ALL 0.0795 0.1220 0.0962 LDC-MAX-ALL-Macro UMass_IESL_SF_XLING_3 0 0.1892 0.2463 0.1853 LDC-MAX-ALL-Macro UMass_IESL_SF_XLING_3 1 0.0094 0.0106 0.0099 LDC-MAX-ALL-Macro UMass_IESL_SF_XLING_3 ALL 0.1002 0.1296 0.0985 LDC-MEAN-ALL-Macro UMass_IESL_SF_XLING_3 0 0.1205 0.1587 0.1205 LDC-MEAN-ALL-Macro UMass_IESL_SF_XLING_3 1 0.0068 0.0080 0.0073 LDC-MEAN-ALL-Macro UMass_IESL_SF_XLING_3 ALL 0.0642 0.0841 0.0645 *ALL-Macro Prec, Recall and F1 refer to mean-precision, mean-recall and mean-F1. NIL-DETECTION P/R/F1: 0.2135 0.5935 0.3140 ************************************************************* Run ID: UMass_IESL_SF_XLING_4 Did the run access the live Web during the evaluation window: No Did the run extract relations from the Cold Start source corpus: Yes Did the run generate meaningful confidence values: Yes Run number of the English SF system that is most closely configured to the English component of this run: 2 Run number of the Spanish SF system that is most closely configured to the Spanish component of this run: 1 Run number of the Chinese SF system that is most closely configured to the Chinese component of this run: NA Slot Filling Evaluation: Metric RunID Hop Prec Recall F1 SF-ALL-Micro UMass_IESL_SF_XLING_4 0 0.1673 0.0889 0.1161 SF-ALL-Micro UMass_IESL_SF_XLING_4 1 0.1477 0.0066 0.0127 SF-ALL-Micro UMass_IESL_SF_XLING_4 ALL 0.1665 0.0622 0.0906 SF-ALL-Macro UMass_IESL_SF_XLING_4 0 0.1289 0.1443 0.1191 SF-ALL-Macro UMass_IESL_SF_XLING_4 1 0.0114 0.0114 0.0114 SF-ALL-Macro UMass_IESL_SF_XLING_4 ALL 0.0688 0.0764 0.0640 LDC-MAX-ALL-Micro UMass_IESL_SF_XLING_4 0 0.1750 0.1505 0.1618 LDC-MAX-ALL-Micro UMass_IESL_SF_XLING_4 1 0.1304 0.0098 0.0183 LDC-MAX-ALL-Micro UMass_IESL_SF_XLING_4 ALL 0.1731 0.1034 0.1294 LDC-MAX-ALL-Macro UMass_IESL_SF_XLING_4 0 0.2012 0.2279 0.1878 LDC-MAX-ALL-Macro UMass_IESL_SF_XLING_4 1 0.0176 0.0176 0.0176 LDC-MAX-ALL-Macro UMass_IESL_SF_XLING_4 ALL 0.1104 0.1239 0.1036 LDC-MEAN-ALL-Macro UMass_IESL_SF_XLING_4 0 0.1258 0.1455 0.1190 LDC-MEAN-ALL-Macro UMass_IESL_SF_XLING_4 1 0.0133 0.0133 0.0133 LDC-MEAN-ALL-Macro UMass_IESL_SF_XLING_4 ALL 0.0701 0.0801 0.0667 *ALL-Macro Prec, Recall and F1 refer to mean-precision, mean-recall and mean-F1. NIL-DETECTION P/R/F1: 0.2292 0.6504 0.3390 ************************************************************* Run ID: UMass_IESL_SF_XLING_5 Did the run access the live Web during the evaluation window: No Did the run extract relations from the Cold Start source corpus: Yes Did the run generate meaningful confidence values: Yes Run number of the English SF system that is most closely configured to the English component of this run: 4 Run number of the Spanish SF system that is most closely configured to the Spanish component of this run: 2 Run number of the Chinese SF system that is most closely configured to the Chinese component of this run: NA Slot Filling Evaluation: Metric RunID Hop Prec Recall F1 SF-ALL-Micro UMass_IESL_SF_XLING_5 0 0.1666 0.0784 0.1066 SF-ALL-Micro UMass_IESL_SF_XLING_5 1 0.3103 0.0046 0.0090 SF-ALL-Micro UMass_IESL_SF_XLING_5 ALL 0.1687 0.0545 0.0823 SF-ALL-Macro UMass_IESL_SF_XLING_5 0 0.1276 0.1353 0.1186 SF-ALL-Macro UMass_IESL_SF_XLING_5 1 0.0066 0.0073 0.0069 SF-ALL-Macro UMass_IESL_SF_XLING_5 ALL 0.0657 0.0698 0.0614 LDC-MAX-ALL-Micro UMass_IESL_SF_XLING_5 0 0.1946 0.1323 0.1576 LDC-MAX-ALL-Micro UMass_IESL_SF_XLING_5 1 0.2857 0.0066 0.0128 LDC-MAX-ALL-Micro UMass_IESL_SF_XLING_5 ALL 0.1962 0.0902 0.1235 LDC-MAX-ALL-Macro UMass_IESL_SF_XLING_5 0 0.2000 0.2061 0.1837 LDC-MAX-ALL-Macro UMass_IESL_SF_XLING_5 1 0.0094 0.0106 0.0099 LDC-MAX-ALL-Macro UMass_IESL_SF_XLING_5 ALL 0.1057 0.1094 0.0977 LDC-MEAN-ALL-Macro UMass_IESL_SF_XLING_5 0 0.1253 0.1369 0.1191 LDC-MEAN-ALL-Macro UMass_IESL_SF_XLING_5 1 0.0068 0.0080 0.0073 LDC-MEAN-ALL-Macro UMass_IESL_SF_XLING_5 ALL 0.0667 0.0731 0.0638 *ALL-Macro Prec, Recall and F1 refer to mean-precision, mean-recall and mean-F1. NIL-DETECTION P/R/F1: 0.2225 0.6260 0.3283