======================================================= TAC KBP 2015 COLD START SLOT FILLING EVALUATION RESULTS ======================================================= Team ID: StaRAI2015 Organization: Indiana University Bloomington and University of Wisconsin Madison ************************************************************* Run ID: StaRAI20151 Did the run access the live Web during the evaluation window: No Run Number of most similar Cold Start KB Construction task submission: NA Did the run extract relations from the Cold Start source corpus: Yes Did the run generate meaningful confidence values: Yes Slot Filling Evaluation: Metric Hop GoldT Submit Correct Incorrect Inexact PIncorrect Dup Right Wrong Prec Recall F1 CSSF micro 0 4840 2263 253 1933 77 0 143 110 2153 0.0486 0.0227 0.0310 CSSF micro 1 3954 483 0 84 2 397 0 0 483 0.0000 0.0000 0.0000 CSSF micro ALL 8794 2746 253 2017 79 397 143 110 2636 0.0401 0.0125 0.0191 LDC-MEAN macro 0 0.0359 LDC-MEAN macro 1 0.0000 LDC-MEAN macro ALL 0.0230 LDC-MAX micro 0 1268 603 93 489 21 0 49 44 559 0.0730 0.0347 0.0470 LDC-MAX micro 1 900 133 0 28 1 104 0 0 133 0.0000 0.0000 0.0000 LDC-MAX micro ALL 2168 736 93 517 22 104 49 44 692 0.0598 0.0203 0.0303 LDC-MAX macro 0 0.0396 LDC-MAX macro 1 0.0000 LDC-MAX macro ALL 0.0254 ************************************************************* Run ID: StaRAI20152 Did the run access the live Web during the evaluation window: No Run Number of most similar Cold Start KB Construction task submission: NA Did the run extract relations from the Cold Start source corpus: Yes Did the run generate meaningful confidence values: Yes Slot Filling Evaluation: Metric Hop GoldT Submit Correct Incorrect Inexact PIncorrect Dup Right Wrong Prec Recall F1 CSSF micro 0 4840 1150 102 1045 3 0 46 56 1094 0.0487 0.0116 0.0187 CSSF micro 1 3954 761 0 38 6 717 0 0 761 0.0000 0.0000 0.0000 CSSF micro ALL 8794 1911 102 1083 9 717 46 56 1855 0.0293 0.0064 0.0105 LDC-MEAN macro 0 0.0162 LDC-MEAN macro 1 0.0000 LDC-MEAN macro ALL 0.0104 LDC-MAX micro 0 1268 334 37 295 2 0 17 20 314 0.0599 0.0158 0.0250 LDC-MAX micro 1 900 248 0 13 3 232 0 0 248 0.0000 0.0000 0.0000 LDC-MAX micro ALL 2168 582 37 308 5 232 17 20 562 0.0344 0.0092 0.0145 LDC-MAX macro 0 0.0167 LDC-MAX macro 1 0.0000 LDC-MAX macro ALL 0.0107 ************************************************************* Run ID: StaRAI20153 Did the run access the live Web during the evaluation window: No Run Number of most similar Cold Start KB Construction task submission: NA Did the run extract relations from the Cold Start source corpus: Yes Did the run generate meaningful confidence values: Yes Slot Filling Evaluation: Metric Hop GoldT Submit Correct Incorrect Inexact PIncorrect Dup Right Wrong Prec Recall F1 CSSF micro 0 4840 2286 251 1956 79 0 142 109 2177 0.0477 0.0225 0.0306 CSSF micro 1 3954 426 0 79 0 347 0 0 426 0.0000 0.0000 0.0000 CSSF micro ALL 8794 2712 251 2035 79 347 142 109 2603 0.0402 0.0124 0.0189 LDC-MEAN macro 0 0.0354 LDC-MEAN macro 1 0.0000 LDC-MEAN macro ALL 0.0228 LDC-MAX micro 0 1268 605 90 493 22 0 48 42 563 0.0694 0.0331 0.0448 LDC-MAX micro 1 900 117 0 25 0 92 0 0 117 0.0000 0.0000 0.0000 LDC-MAX micro ALL 2168 722 90 518 22 92 48 42 680 0.0582 0.0194 0.0291 LDC-MAX macro 0 0.0365 LDC-MAX macro 1 0.0000 LDC-MAX macro ALL 0.0234 ************************************************************* Run ID: StaRAI20154 Did the run access the live Web during the evaluation window: No Run Number of most similar Cold Start KB Construction task submission: NA Did the run extract relations from the Cold Start source corpus: Yes Did the run generate meaningful confidence values: Yes Slot Filling Evaluation: Metric Hop GoldT Submit Correct Incorrect Inexact PIncorrect Dup Right Wrong Prec Recall F1 CSSF micro 0 4840 1141 118 1011 12 0 45 73 1068 0.0640 0.0151 0.0244 CSSF micro 1 3954 555 0 48 0 507 0 0 555 0.0000 0.0000 0.0000 CSSF micro ALL 8794 1696 118 1059 12 507 45 73 1623 0.0430 0.0083 0.0139 LDC-MEAN macro 0 0.0187 LDC-MEAN macro 1 0.0000 LDC-MEAN macro ALL 0.0120 LDC-MAX micro 0 1268 351 40 306 5 0 15 25 326 0.0712 0.0197 0.0309 LDC-MAX micro 1 900 186 0 12 0 174 0 0 186 0.0000 0.0000 0.0000 LDC-MAX micro ALL 2168 537 40 318 5 174 15 25 512 0.0466 0.0115 0.0185 LDC-MAX macro 0 0.0222 LDC-MAX macro 1 0.0000 LDC-MAX macro ALL 0.0143 ************************************************************* Run ID: StaRAI20155 Did the run access the live Web during the evaluation window: No Run Number of most similar Cold Start KB Construction task submission: NA Did the run extract relations from the Cold Start source corpus: Yes Did the run generate meaningful confidence values: Yes Slot Filling Evaluation: Metric Hop GoldT Submit Correct Incorrect Inexact PIncorrect Dup Right Wrong Prec Recall F1 CSSF micro 0 4840 851 96 748 7 0 37 59 792 0.0693 0.0122 0.0207 CSSF micro 1 3954 371 0 0 0 371 0 0 371 0.0000 0.0000 0.0000 CSSF micro ALL 8794 1222 96 748 7 371 37 59 1163 0.0483 0.0067 0.0118 LDC-MEAN macro 0 0.0170 LDC-MEAN macro 1 0.0000 LDC-MEAN macro ALL 0.0109 LDC-MAX micro 0 1268 264 32 228 4 0 13 19 245 0.0720 0.0150 0.0248 LDC-MAX micro 1 900 121 0 0 0 121 0 0 121 0.0000 0.0000 0.0000 LDC-MAX micro ALL 2168 385 32 228 4 121 13 19 366 0.0494 0.0088 0.0149 LDC-MAX macro 0 0.0203 LDC-MAX macro 1 0.0000 LDC-MAX macro ALL 0.0130