======================================================= TAC KBP 2015 COLD START SLOT FILLING EVALUATION RESULTS ======================================================= Team ID: UMass_IESL Organization: University of Massachusetts, Amherst ************************************************************* Run ID: UMass_IESL1 Did the run access the live Web during the evaluation window: No Run Number of most similar Cold Start KB Construction task submission: 1 Did the run extract relations from the Cold Start source corpus: Yes Did the run generate meaningful confidence values: Yes Slot Filling Evaluation: Metric Hop GoldT Submit Correct Incorrect Inexact PIncorrect Dup Right Wrong Prec Recall F1 CSSF micro 0 4840 2693 907 1636 150 0 66 841 1852 0.3123 0.1738 0.2233 CSSF micro 1 3954 2159 210 695 34 1220 5 205 1954 0.0950 0.0518 0.0671 CSSF micro ALL 8794 4852 1117 2331 184 1220 71 1046 3806 0.2156 0.1189 0.1533 LDC-MEAN macro 0 0.1983 LDC-MEAN macro 1 0.0806 LDC-MEAN macro ALL 0.1562 LDC-MAX micro 0 1268 713 255 416 42 0 15 240 473 0.3366 0.1893 0.2423 LDC-MAX micro 1 900 574 56 187 8 323 1 55 519 0.0958 0.0611 0.0746 LDC-MAX micro ALL 2168 1287 311 603 50 323 16 295 992 0.2292 0.1361 0.1708 LDC-MAX macro 0 0.2285 LDC-MAX macro 1 0.0930 LDC-MAX macro ALL 0.1800 ************************************************************* Run ID: UMass_IESL2 Did the run access the live Web during the evaluation window: No Run Number of most similar Cold Start KB Construction task submission: 1 Did the run extract relations from the Cold Start source corpus: Yes Did the run generate meaningful confidence values: Yes Slot Filling Evaluation: Metric Hop GoldT Submit Correct Incorrect Inexact PIncorrect Dup Right Wrong Prec Recall F1 CSSF micro 0 4840 4731 970 3601 160 0 72 898 3833 0.1898 0.1855 0.1877 CSSF micro 1 3954 10090 278 1434 39 8339 7 271 9819 0.0269 0.0685 0.0386 CSSF micro ALL 8794 14821 1248 5035 199 8339 79 1169 13652 0.0789 0.1329 0.0990 LDC-MEAN macro 0 0.1997 LDC-MEAN macro 1 0.0894 LDC-MEAN macro ALL 0.1602 LDC-MAX micro 0 1268 1281 282 954 45 0 17 265 1016 0.2069 0.2090 0.2079 LDC-MAX micro 1 900 2258 81 483 11 1683 3 78 2180 0.0345 0.0867 0.0494 LDC-MAX micro ALL 2168 3539 363 1437 56 1683 20 343 3196 0.0969 0.1582 0.1202 LDC-MAX macro 0 0.2535 LDC-MAX macro 1 0.1110 LDC-MAX macro ALL 0.2025 ************************************************************* Run ID: UMass_IESL3 Did the run access the live Web during the evaluation window: No Run Number of most similar Cold Start KB Construction task submission: 2 Did the run extract relations from the Cold Start source corpus: Yes Did the run generate meaningful confidence values: Yes Slot Filling Evaluation: Metric Hop GoldT Submit Correct Incorrect Inexact PIncorrect Dup Right Wrong Prec Recall F1 CSSF micro 0 4840 3174 886 2138 150 0 65 821 2353 0.2587 0.1696 0.2049 CSSF micro 1 3954 3136 235 1063 34 1804 5 230 2906 0.0733 0.0582 0.0649 CSSF micro ALL 8794 6310 1121 3201 184 1804 70 1051 5259 0.1666 0.1195 0.1392 LDC-MEAN macro 0 0.1955 LDC-MEAN macro 1 0.0640 LDC-MEAN macro ALL 0.1484 LDC-MAX micro 0 1268 854 267 544 43 0 15 252 602 0.2951 0.1987 0.2375 LDC-MAX micro 1 900 700 69 255 7 369 1 68 632 0.0971 0.0756 0.0850 LDC-MAX micro ALL 2168 1554 336 799 50 369 16 320 1234 0.2059 0.1476 0.1720 LDC-MAX macro 0 0.2481 LDC-MAX macro 1 0.0901 LDC-MAX macro ALL 0.1915 ************************************************************* Run ID: UMass_IESL4 Did the run access the live Web during the evaluation window: No Run Number of most similar Cold Start KB Construction task submission: 1 Did the run extract relations from the Cold Start source corpus: Yes Did the run generate meaningful confidence values: Yes Slot Filling Evaluation: Metric Hop GoldT Submit Correct Incorrect Inexact PIncorrect Dup Right Wrong Prec Recall F1 CSSF micro 0 4840 1980 721 1150 109 0 51 670 1310 0.3384 0.1384 0.1965 CSSF micro 1 3954 1810 131 613 25 1041 5 126 1684 0.0696 0.0319 0.0437 CSSF micro ALL 8794 3790 852 1763 134 1041 56 796 2994 0.2100 0.0905 0.1265 LDC-MEAN macro 0 0.1965 LDC-MEAN macro 1 0.0451 LDC-MEAN macro ALL 0.1423 LDC-MAX micro 0 1268 562 212 311 39 0 11 201 361 0.3577 0.1585 0.2197 LDC-MAX micro 1 900 544 36 160 6 342 1 35 509 0.0643 0.0389 0.0485 LDC-MAX micro ALL 2168 1106 248 471 45 342 12 236 870 0.2134 0.1089 0.1442 LDC-MAX macro 0 0.2281 LDC-MAX macro 1 0.0541 LDC-MAX macro ALL 0.1658 ************************************************************* Run ID: UMass_IESL5 Did the run access the live Web during the evaluation window: No Run Number of most similar Cold Start KB Construction task submission: 1 Did the run extract relations from the Cold Start source corpus: Yes Did the run generate meaningful confidence values: Yes Slot Filling Evaluation: Metric Hop GoldT Submit Correct Incorrect Inexact PIncorrect Dup Right Wrong Prec Recall F1 CSSF micro 0 4840 2589 834 1608 147 0 66 768 1821 0.2966 0.1587 0.2068 CSSF micro 1 3954 2144 193 676 24 1251 5 188 1956 0.0877 0.0475 0.0617 CSSF micro ALL 8794 4733 1027 2284 171 1251 71 956 3777 0.2020 0.1087 0.1413 LDC-MEAN macro 0 0.1884 LDC-MEAN macro 1 0.0733 LDC-MEAN macro ALL 0.1472 LDC-MAX micro 0 1268 691 235 413 43 0 15 220 471 0.3184 0.1735 0.2246 LDC-MAX micro 1 900 554 52 195 6 301 1 51 503 0.0921 0.0567 0.0702 LDC-MAX micro ALL 2168 1245 287 608 49 301 16 271 974 0.2177 0.1250 0.1588 LDC-MAX macro 0 0.2157 LDC-MAX macro 1 0.0853 LDC-MAX macro ALL 0.1691