======================================================= TAC KBP 2015 COLD START SLOT FILLING EVALUATION RESULTS ======================================================= Team ID: CIS Organization: Ludwig-Maximilians-Universitaet Muenchen ************************************************************* Run ID: CIS1 Did the run access the live Web during the evaluation window: No Run Number of most similar Cold Start KB Construction task submission: NA Did the run extract relations from the Cold Start source corpus: Yes Did the run generate meaningful confidence values: Yes Slot Filling Evaluation: Metric Hop GoldT Submit Correct Incorrect Inexact PIncorrect Dup Right Wrong Prec Recall F1 CSSF micro 0 4840 985 539 410 36 0 6 533 452 0.5411 0.1101 0.1830 CSSF micro 1 3954 449 76 114 9 250 0 76 373 0.1693 0.0192 0.0345 CSSF micro ALL 8794 1434 615 524 45 250 6 609 825 0.4247 0.0693 0.1191 LDC-MEAN macro 0 0.2052 LDC-MEAN macro 1 0.0187 LDC-MEAN macro ALL 0.1384 LDC-MAX micro 0 1268 283 165 107 11 0 2 163 120 0.5760 0.1285 0.2102 LDC-MAX micro 1 900 107 17 26 2 62 0 17 90 0.1589 0.0189 0.0338 LDC-MAX micro ALL 2168 390 182 133 13 62 2 180 210 0.4615 0.0830 0.1407 LDC-MAX macro 0 0.2243 LDC-MAX macro 1 0.0187 LDC-MAX macro ALL 0.1507 ************************************************************* Run ID: CIS2 Did the run access the live Web during the evaluation window: No Run Number of most similar Cold Start KB Construction task submission: NA Did the run extract relations from the Cold Start source corpus: Yes Did the run generate meaningful confidence values: Yes Slot Filling Evaluation: Metric Hop GoldT Submit Correct Incorrect Inexact PIncorrect Dup Right Wrong Prec Recall F1 CSSF micro 0 4840 3524 1043 2342 139 0 20 1023 2501 0.2903 0.2114 0.2446 CSSF micro 1 3954 2267 240 557 57 1413 2 238 2029 0.1050 0.0602 0.0765 CSSF micro ALL 8794 5791 1283 2899 196 1413 22 1261 4530 0.2178 0.1434 0.1729 LDC-MEAN macro 0 0.2995 LDC-MEAN macro 1 0.0665 LDC-MEAN macro ALL 0.2161 LDC-MAX micro 0 1268 960 309 614 37 0 5 304 656 0.3167 0.2397 0.2729 LDC-MAX micro 1 900 545 58 129 14 344 1 57 488 0.1046 0.0633 0.0789 LDC-MAX micro ALL 2168 1505 367 743 51 344 6 361 1144 0.2399 0.1665 0.1966 LDC-MAX macro 0 0.3238 LDC-MAX macro 1 0.0753 LDC-MAX macro ALL 0.2348 ************************************************************* Run ID: CIS3 Did the run access the live Web during the evaluation window: No Run Number of most similar Cold Start KB Construction task submission: NA Did the run extract relations from the Cold Start source corpus: Yes Did the run generate meaningful confidence values: Yes Slot Filling Evaluation: Metric Hop GoldT Submit Correct Incorrect Inexact PIncorrect Dup Right Wrong Prec Recall F1 CSSF micro 0 4840 4230 1166 2880 184 0 15 1151 3079 0.2721 0.2378 0.2538 CSSF micro 1 3954 1456 195 300 77 884 2 193 1263 0.1326 0.0488 0.0713 CSSF micro ALL 8794 5686 1361 3180 261 884 17 1344 4342 0.2364 0.1528 0.1856 LDC-MEAN macro 0 0.3066 LDC-MEAN macro 1 0.0587 LDC-MEAN macro ALL 0.2178 LDC-MAX micro 0 1268 1125 340 732 53 0 4 336 789 0.2987 0.2650 0.2808 LDC-MAX micro 1 900 375 54 78 17 226 1 53 322 0.1413 0.0589 0.0831 LDC-MAX micro ALL 2168 1500 394 810 70 226 5 389 1111 0.2593 0.1794 0.2121 LDC-MAX macro 0 0.3286 LDC-MAX macro 1 0.0676 LDC-MAX macro ALL 0.2352 ************************************************************* Run ID: CIS4 Did the run access the live Web during the evaluation window: No Run Number of most similar Cold Start KB Construction task submission: NA Did the run extract relations from the Cold Start source corpus: Yes Did the run generate meaningful confidence values: Yes Slot Filling Evaluation: Metric Hop GoldT Submit Correct Incorrect Inexact PIncorrect Dup Right Wrong Prec Recall F1 CSSF micro 0 4840 3535 1049 2348 138 0 20 1029 2506 0.2911 0.2126 0.2457 CSSF micro 1 3954 2289 267 567 60 1395 2 265 2024 0.1158 0.0670 0.0849 CSSF micro ALL 8794 5824 1316 2915 198 1395 22 1294 4530 0.2222 0.1471 0.1770 LDC-MEAN macro 0 0.3024 LDC-MEAN macro 1 0.0686 LDC-MEAN macro ALL 0.2187 LDC-MAX micro 0 1268 965 311 615 39 0 5 306 659 0.3171 0.2413 0.2741 LDC-MAX micro 1 900 533 64 122 14 333 1 63 470 0.1182 0.0700 0.0879 LDC-MAX micro ALL 2168 1498 375 737 53 333 6 369 1129 0.2463 0.1702 0.2013 LDC-MAX macro 0 0.3267 LDC-MAX macro 1 0.0778 LDC-MAX macro ALL 0.2376 ************************************************************* Run ID: CIS5 Did the run access the live Web during the evaluation window: No Run Number of most similar Cold Start KB Construction task submission: NA Did the run extract relations from the Cold Start source corpus: Yes Did the run generate meaningful confidence values: Yes Slot Filling Evaluation: Metric Hop GoldT Submit Correct Incorrect Inexact PIncorrect Dup Right Wrong Prec Recall F1 CSSF micro 0 4840 5837 961 4729 147 0 6 955 4882 0.1636 0.1973 0.1789 CSSF micro 1 3954 2902 132 370 66 2334 0 132 2770 0.0455 0.0334 0.0385 CSSF micro ALL 8794 8739 1093 5099 213 2334 6 1087 7652 0.1244 0.1236 0.1240 LDC-MEAN macro 0 0.2677 LDC-MEAN macro 1 0.0430 LDC-MEAN macro ALL 0.1872 LDC-MAX micro 0 1268 1481 285 1151 45 0 2 283 1198 0.1911 0.2232 0.2059 LDC-MAX micro 1 900 729 37 92 15 585 0 37 692 0.0508 0.0411 0.0454 LDC-MAX micro ALL 2168 2210 322 1243 60 585 2 320 1890 0.1448 0.1476 0.1462 LDC-MAX macro 0 0.2906 LDC-MAX macro 1 0.0525 LDC-MAX macro ALL 0.2054