======================================================= TAC KBP 2015 COLD START SLOT FILLING EVALUATION RESULTS ======================================================= Team ID: Stanford Organization: Stanford University ************************************************************* Run ID: Stanford1 Did the run access the live Web during the evaluation window: No Run Number of most similar Cold Start KB Construction task submission: 1 Did the run extract relations from the Cold Start source corpus: Yes Did the run generate meaningful confidence values: Yes Slot Filling Evaluation: Metric Hop GoldT Submit Correct Incorrect Inexact PIncorrect Dup Right Wrong Prec Recall F1 CSSF micro 0 4840 2133 1099 916 118 0 12 1087 1046 0.5096 0.2246 0.3118 CSSF micro 1 3954 1008 324 301 37 346 3 321 687 0.3185 0.0812 0.1294 CSSF micro ALL 8794 3141 1423 1217 155 346 15 1408 1733 0.4483 0.1601 0.2359 LDC-MEAN macro 0 0.2817 LDC-MEAN macro 1 0.1346 LDC-MEAN macro ALL 0.2290 LDC-MAX micro 0 1268 640 337 267 36 0 4 333 307 0.5203 0.2626 0.3491 LDC-MAX micro 1 900 287 93 74 9 111 1 92 195 0.3206 0.1022 0.1550 LDC-MAX micro ALL 2168 927 430 341 45 111 5 425 502 0.4585 0.1960 0.2746 LDC-MAX macro 0 0.3116 LDC-MAX macro 1 0.1674 LDC-MAX macro ALL 0.2600 ************************************************************* Run ID: Stanford2 Did the run access the live Web during the evaluation window: No Run Number of most similar Cold Start KB Construction task submission: 2 Did the run extract relations from the Cold Start source corpus: Yes Did the run generate meaningful confidence values: Yes Slot Filling Evaluation: Metric Hop GoldT Submit Correct Incorrect Inexact PIncorrect Dup Right Wrong Prec Recall F1 CSSF micro 0 4840 1300 816 423 61 0 30 786 514 0.6046 0.1624 0.2560 CSSF micro 1 3954 724 278 213 38 195 8 270 454 0.3729 0.0683 0.1154 CSSF micro ALL 8794 2024 1094 636 99 195 38 1056 968 0.5217 0.1201 0.1952 LDC-MEAN macro 0 0.2329 LDC-MEAN macro 1 0.1082 LDC-MEAN macro ALL 0.1883 LDC-MAX micro 0 1268 392 252 121 19 0 8 244 148 0.6224 0.1924 0.2940 LDC-MAX micro 1 900 208 80 55 9 64 2 78 130 0.3750 0.0867 0.1408 LDC-MAX micro ALL 2168 600 332 176 28 64 10 322 278 0.5367 0.1485 0.2327 LDC-MAX macro 0 0.2617 LDC-MAX macro 1 0.1282 LDC-MAX macro ALL 0.2139 ************************************************************* Run ID: Stanford3 Did the run access the live Web during the evaluation window: No Run Number of most similar Cold Start KB Construction task submission: 2 Did the run extract relations from the Cold Start source corpus: Yes Did the run generate meaningful confidence values: Yes Slot Filling Evaluation: Metric Hop GoldT Submit Correct Incorrect Inexact PIncorrect Dup Right Wrong Prec Recall F1 CSSF micro 0 4840 4074 1446 2488 140 0 44 1402 2672 0.3441 0.2897 0.3146 CSSF micro 1 3954 4564 1072 1917 122 1453 8 1064 3500 0.2331 0.2691 0.2498 CSSF micro ALL 8794 8638 2518 4405 262 1453 52 2466 6172 0.2855 0.2804 0.2829 LDC-MEAN macro 0 0.3321 LDC-MEAN macro 1 0.1925 LDC-MEAN macro ALL 0.2821 LDC-MAX micro 0 1268 1149 446 657 46 0 13 433 716 0.3768 0.3415 0.3583 LDC-MAX micro 1 900 1177 267 491 28 391 2 265 912 0.2251 0.2944 0.2552 LDC-MAX micro ALL 2168 2326 713 1148 74 391 15 698 1628 0.3001 0.3220 0.3106 LDC-MAX macro 0 0.3700 LDC-MAX macro 1 0.2252 LDC-MAX macro ALL 0.3181 ************************************************************* Run ID: Stanford4 Did the run access the live Web during the evaluation window: No Run Number of most similar Cold Start KB Construction task submission: 1 Did the run extract relations from the Cold Start source corpus: Yes Did the run generate meaningful confidence values: Yes Slot Filling Evaluation: Metric Hop GoldT Submit Correct Incorrect Inexact PIncorrect Dup Right Wrong Prec Recall F1 CSSF micro 0 4840 1214 772 381 61 0 30 742 472 0.6112 0.1533 0.2451 CSSF micro 1 3954 581 252 169 28 132 8 244 337 0.4200 0.0617 0.1076 CSSF micro ALL 8794 1795 1024 550 89 132 38 986 809 0.5493 0.1121 0.1862 LDC-MEAN macro 0 0.2278 LDC-MEAN macro 1 0.0897 LDC-MEAN macro ALL 0.1784 LDC-MAX micro 0 1268 370 241 109 20 0 8 233 137 0.6297 0.1838 0.2845 LDC-MAX micro 1 900 160 71 42 7 40 2 69 91 0.4313 0.0767 0.1302 LDC-MAX micro ALL 2168 530 312 151 27 40 10 302 228 0.5698 0.1393 0.2239 LDC-MAX macro 0 0.2556 LDC-MAX macro 1 0.1118 LDC-MAX macro ALL 0.2041