======================================================= TAC KBP 2015 COLD START SLOT FILLING EVALUATION RESULTS ======================================================= Team ID: UWashington Organization: University of Washington ************************************************************* Run ID: UWashington1 Did the run access the live Web during the evaluation window: No Run Number of most similar Cold Start KB Construction task submission: NA Did the run extract relations from the Cold Start source corpus: Yes Did the run generate meaningful confidence values: Yes Slot Filling Evaluation: Metric Hop GoldT Submit Correct Incorrect Inexact PIncorrect Dup Right Wrong Prec Recall F1 CSSF micro 0 4840 1202 559 556 87 0 23 536 666 0.4459 0.1107 0.1774 CSSF micro 1 3954 1344 243 732 67 302 9 234 1110 0.1741 0.0592 0.0883 CSSF micro ALL 8794 2546 802 1288 154 302 32 770 1776 0.3024 0.0876 0.1358 LDC-MEAN macro 0 0.1441 LDC-MEAN macro 1 0.0436 LDC-MEAN macro ALL 0.1081 LDC-MAX micro 0 1268 376 184 165 27 0 8 176 200 0.4681 0.1388 0.2141 LDC-MAX micro 1 900 363 65 179 15 104 2 63 300 0.1736 0.0700 0.0998 LDC-MAX micro ALL 2168 739 249 344 42 104 10 239 500 0.3234 0.1102 0.1644 LDC-MAX macro 0 0.1697 LDC-MAX macro 1 0.0512 LDC-MAX macro ALL 0.1273 ************************************************************* Run ID: UWashington2 Did the run access the live Web during the evaluation window: No Run Number of most similar Cold Start KB Construction task submission: NA Did the run extract relations from the Cold Start source corpus: Yes Did the run generate meaningful confidence values: Yes Slot Filling Evaluation: Metric Hop GoldT Submit Correct Incorrect Inexact PIncorrect Dup Right Wrong Prec Recall F1 CSSF micro 0 4840 318 168 111 39 0 0 168 150 0.5283 0.0347 0.0651 CSSF micro 1 3954 47 27 0 0 20 0 27 20 0.5745 0.0068 0.0135 CSSF micro ALL 8794 365 195 111 39 20 0 195 170 0.5342 0.0222 0.0426 LDC-MEAN macro 0 0.0686 LDC-MEAN macro 1 0.0104 LDC-MEAN macro ALL 0.0477 LDC-MAX micro 0 1268 122 65 46 11 0 0 65 57 0.5328 0.0513 0.0935 LDC-MAX micro 1 900 14 10 0 0 4 0 10 4 0.7143 0.0111 0.0219 LDC-MAX micro ALL 2168 136 75 46 11 4 0 75 61 0.5515 0.0346 0.0651 LDC-MAX macro 0 0.0799 LDC-MAX macro 1 0.0156 LDC-MAX macro ALL 0.0569 ************************************************************* Run ID: UWashington3 Did the run access the live Web during the evaluation window: No Run Number of most similar Cold Start KB Construction task submission: NA Did the run extract relations from the Cold Start source corpus: Yes Did the run generate meaningful confidence values: Yes Slot Filling Evaluation: Metric Hop GoldT Submit Correct Incorrect Inexact PIncorrect Dup Right Wrong Prec Recall F1 CSSF micro 0 4840 535 290 220 25 0 12 278 257 0.5196 0.0574 0.1034 CSSF micro 1 3954 68 20 3 0 45 0 20 48 0.2941 0.0051 0.0099 CSSF micro ALL 8794 603 310 223 25 45 12 298 305 0.4942 0.0339 0.0634 LDC-MEAN macro 0 0.0526 LDC-MEAN macro 1 0.0223 LDC-MEAN macro ALL 0.0417 LDC-MAX micro 0 1268 161 90 63 8 0 4 86 75 0.5342 0.0678 0.1204 LDC-MAX micro 1 900 27 10 3 0 14 0 10 17 0.3704 0.0111 0.0216 LDC-MAX micro ALL 2168 188 100 66 8 14 4 96 92 0.5106 0.0443 0.0815 LDC-MAX macro 0 0.0679 LDC-MAX macro 1 0.0262 LDC-MAX macro ALL 0.0530 ************************************************************* Run ID: UWashington4 Did the run access the live Web during the evaluation window: No Run Number of most similar Cold Start KB Construction task submission: NA Did the run extract relations from the Cold Start source corpus: Yes Did the run generate meaningful confidence values: Yes Slot Filling Evaluation: Metric Hop GoldT Submit Correct Incorrect Inexact PIncorrect Dup Right Wrong Prec Recall F1 CSSF micro 0 4840 481 188 264 29 0 5 183 298 0.3805 0.0378 0.0688 CSSF micro 1 3954 202 80 19 35 68 0 80 122 0.3960 0.0202 0.0385 CSSF micro ALL 8794 683 268 283 64 68 5 263 420 0.3851 0.0299 0.0555 LDC-MEAN macro 0 0.0456 LDC-MEAN macro 1 0.0051 LDC-MEAN macro ALL 0.0311 LDC-MAX micro 0 1268 142 57 76 9 0 1 56 86 0.3944 0.0442 0.0794 LDC-MAX micro 1 900 44 16 4 7 17 0 16 28 0.3636 0.0178 0.0339 LDC-MAX micro ALL 2168 186 73 80 16 17 1 72 114 0.3871 0.0332 0.0612 LDC-MAX macro 0 0.0523 LDC-MAX macro 1 0.0051 LDC-MAX macro ALL 0.0354