======================================================= TAC KBP 2015 COLD START SLOT FILLING EVALUATION RESULTS ======================================================= Team ID: MSIIPL_THU Organization: Tsinghua University ************************************************************* Run ID: MSIIPL_THU1 Did the run access the live Web during the evaluation window: No Run Number of most similar Cold Start KB Construction task submission: NA Did the run extract relations from the Cold Start source corpus: Yes Did the run generate meaningful confidence values: Yes Slot Filling Evaluation: Metric Hop GoldT Submit Correct Incorrect Inexact PIncorrect Dup Right Wrong Prec Recall F1 CSSF micro 0 4840 1547 301 1061 185 0 8 293 1254 0.1894 0.0605 0.0917 CSSF micro 1 3954 1589 17 111 4 1457 2 15 1574 0.0094 0.0038 0.0054 CSSF micro ALL 8794 3136 318 1172 189 1457 10 308 2828 0.0982 0.0350 0.0516 LDC-MEAN macro 0 0.0819 LDC-MEAN macro 1 0.0042 LDC-MEAN macro ALL 0.0540 LDC-MAX micro 0 1268 475 106 316 53 0 2 104 371 0.2189 0.0820 0.1193 LDC-MAX micro 1 900 366 6 35 2 323 1 5 361 0.0137 0.0056 0.0079 LDC-MAX micro ALL 2168 841 112 351 55 323 3 109 732 0.1296 0.0503 0.0724 LDC-MAX macro 0 0.1026 LDC-MAX macro 1 0.0083 LDC-MAX macro ALL 0.0688 ************************************************************* Run ID: MSIIPL_THU2 Did the run access the live Web during the evaluation window: No Run Number of most similar Cold Start KB Construction task submission: NA Did the run extract relations from the Cold Start source corpus: Yes Did the run generate meaningful confidence values: Yes Slot Filling Evaluation: Metric Hop GoldT Submit Correct Incorrect Inexact PIncorrect Dup Right Wrong Prec Recall F1 CSSF micro 0 4840 1292 275 861 156 0 10 265 1027 0.2051 0.0548 0.0864 CSSF micro 1 3954 1336 15 94 2 1225 0 15 1321 0.0112 0.0038 0.0057 CSSF micro ALL 8794 2628 290 955 158 1225 10 280 2348 0.1065 0.0318 0.0490 LDC-MEAN macro 0 0.0789 LDC-MEAN macro 1 0.0044 LDC-MEAN macro ALL 0.0522 LDC-MAX micro 0 1268 393 96 254 43 0 3 93 300 0.2366 0.0733 0.1120 LDC-MAX micro 1 900 312 5 27 1 279 0 5 307 0.0160 0.0056 0.0083 LDC-MAX micro ALL 2168 705 101 281 44 279 3 98 607 0.1390 0.0452 0.0682 LDC-MAX macro 0 0.0992 LDC-MAX macro 1 0.0085 LDC-MAX macro ALL 0.0668 ************************************************************* Run ID: MSIIPL_THU4 Did the run access the live Web during the evaluation window: No Run Number of most similar Cold Start KB Construction task submission: 2 Did the run extract relations from the Cold Start source corpus: Yes Did the run generate meaningful confidence values: Yes Slot Filling Evaluation: Metric Hop GoldT Submit Correct Incorrect Inexact PIncorrect Dup Right Wrong Prec Recall F1 CSSF micro 0 4840 718 22 683 13 0 0 22 696 0.0306 0.0045 0.0079 CSSF micro 1 3954 81 0 0 0 81 0 0 81 0.0000 0.0000 0.0000 CSSF micro ALL 8794 799 22 683 13 81 0 22 777 0.0275 0.0025 0.0046 LDC-MEAN macro 0 0.0058 LDC-MEAN macro 1 0.0000 LDC-MEAN macro ALL 0.0037 LDC-MAX micro 0 1268 185 7 174 4 0 0 7 178 0.0378 0.0055 0.0096 LDC-MAX micro 1 900 19 0 0 0 19 0 0 19 0.0000 0.0000 0.0000 LDC-MAX micro ALL 2168 204 7 174 4 19 0 7 197 0.0343 0.0032 0.0059 LDC-MAX macro 0 0.0072 LDC-MAX macro 1 0.0000 LDC-MAX macro ALL 0.0046 ************************************************************* Run ID: MSIIPL_THU5 Did the run access the live Web during the evaluation window: No Run Number of most similar Cold Start KB Construction task submission: 2 Did the run extract relations from the Cold Start source corpus: Yes Did the run generate meaningful confidence values: Yes Slot Filling Evaluation: Metric Hop GoldT Submit Correct Incorrect Inexact PIncorrect Dup Right Wrong Prec Recall F1 CSSF micro 0 4840 625 46 565 14 0 0 46 579 0.0736 0.0095 0.0168 CSSF micro 1 3954 0 0 0 0 0 0 0 0 0.0000 0.0000 0.0000 CSSF micro ALL 8794 625 46 565 14 0 0 46 579 0.0736 0.0052 0.0098 LDC-MEAN macro 0 0.0175 LDC-MEAN macro 1 0.0000 LDC-MEAN macro ALL 0.0112 LDC-MAX micro 0 1268 167 15 149 3 0 0 15 152 0.0898 0.0118 0.0209 LDC-MAX micro 1 900 0 0 0 0 0 0 0 0 0.0000 0.0000 0.0000 LDC-MAX micro ALL 2168 167 15 149 3 0 0 15 152 0.0898 0.0069 0.0128 LDC-MAX macro 0 0.0238 LDC-MAX macro 1 0.0000 LDC-MAX macro ALL 0.0153