======================================================= TAC KBP 2015 COLD START SLOT FILLING EVALUATION RESULTS ======================================================= Team ID: BUPT_PRIS Organization: Beijing University of Posts and Telecommunications ************************************************************* Run ID: BUPT_PRIS1 Did the run access the live Web during the evaluation window: No Run Number of most similar Cold Start KB Construction task submission: NA Did the run extract relations from the Cold Start source corpus: Yes Did the run generate meaningful confidence values: No Slot Filling Evaluation: Metric Hop GoldT Submit Correct Incorrect Inexact PIncorrect Dup Right Wrong Prec Recall F1 CSSF micro 0 4840 3149 752 2181 216 0 32 720 2429 0.2286 0.1488 0.1802 CSSF micro 1 3954 2920 169 510 42 2199 15 154 2766 0.0527 0.0389 0.0448 CSSF micro ALL 8794 6069 921 2691 258 2199 47 874 5195 0.1440 0.0994 0.1176 LDC-MEAN macro 0 0.1960 LDC-MEAN macro 1 0.0657 LDC-MEAN macro ALL 0.1494 LDC-MAX micro 0 1268 894 250 585 59 0 8 242 652 0.2707 0.1909 0.2239 LDC-MAX micro 1 900 752 50 153 13 536 4 46 706 0.0612 0.0511 0.0557 LDC-MAX micro ALL 2168 1646 300 738 72 536 12 288 1358 0.1750 0.1328 0.1510 LDC-MAX macro 0 0.2358 LDC-MAX macro 1 0.0782 LDC-MAX macro ALL 0.1794 ************************************************************* Run ID: BUPT_PRIS2 Did the run access the live Web during the evaluation window: No Run Number of most similar Cold Start KB Construction task submission: NA Did the run extract relations from the Cold Start source corpus: Yes Did the run generate meaningful confidence values: No Slot Filling Evaluation: Metric Hop GoldT Submit Correct Incorrect Inexact PIncorrect Dup Right Wrong Prec Recall F1 CSSF micro 0 4840 2649 335 2181 133 0 10 325 2324 0.1227 0.0671 0.0868 CSSF micro 1 3954 0 0 0 0 0 0 0 0 0.0000 0.0000 0.0000 CSSF micro ALL 8794 2649 335 2181 133 0 10 325 2324 0.1227 0.0370 0.0568 LDC-MEAN macro 0 0.1035 LDC-MEAN macro 1 0.0000 LDC-MEAN macro ALL 0.0664 LDC-MAX micro 0 1268 741 127 581 33 0 5 122 619 0.1646 0.0962 0.1215 LDC-MAX micro 1 900 0 0 0 0 0 0 0 0 0.0000 0.0000 0.0000 LDC-MAX micro ALL 2168 741 127 581 33 0 5 122 619 0.1646 0.0563 0.0839 LDC-MAX macro 0 0.1288 LDC-MAX macro 1 0.0000 LDC-MAX macro ALL 0.0827 ************************************************************* Run ID: BUPT_PRIS3 Did the run access the live Web during the evaluation window: No Run Number of most similar Cold Start KB Construction task submission: NA Did the run extract relations from the Cold Start source corpus: Yes Did the run generate meaningful confidence values: No Slot Filling Evaluation: Metric Hop GoldT Submit Correct Incorrect Inexact PIncorrect Dup Right Wrong Prec Recall F1 CSSF micro 0 4840 3149 750 2181 218 0 32 718 2431 0.2280 0.1483 0.1797 CSSF micro 1 3954 0 0 0 0 0 0 0 0 0.0000 0.0000 0.0000 CSSF micro ALL 8794 3149 750 2181 218 0 32 718 2431 0.2280 0.0816 0.1202 LDC-MEAN macro 0 0.1937 LDC-MEAN macro 1 0.0000 LDC-MEAN macro ALL 0.1243 LDC-MAX micro 0 1268 889 249 579 61 0 8 241 648 0.2711 0.1901 0.2235 LDC-MAX micro 1 900 0 0 0 0 0 0 0 0 0.0000 0.0000 0.0000 LDC-MAX micro ALL 2168 889 249 579 61 0 8 241 648 0.2711 0.1112 0.1577 LDC-MAX macro 0 0.2334 LDC-MAX macro 1 0.0000 LDC-MAX macro ALL 0.1498 ************************************************************* Run ID: BUPT_PRIS4 Did the run access the live Web during the evaluation window: No Run Number of most similar Cold Start KB Construction task submission: NA Did the run extract relations from the Cold Start source corpus: Yes Did the run generate meaningful confidence values: No Slot Filling Evaluation: Metric Hop GoldT Submit Correct Incorrect Inexact PIncorrect Dup Right Wrong Prec Recall F1 CSSF micro 0 4840 2649 335 2181 133 0 10 325 2324 0.1227 0.0671 0.0868 CSSF micro 1 3954 1665 25 208 15 1417 3 22 1643 0.0132 0.0056 0.0078 CSSF micro ALL 8794 4314 360 2389 148 1417 13 347 3967 0.0804 0.0395 0.0529 LDC-MEAN macro 0 0.1035 LDC-MEAN macro 1 0.0126 LDC-MEAN macro ALL 0.0709 LDC-MAX micro 0 1268 738 127 574 37 0 5 122 616 0.1653 0.0962 0.1216 LDC-MAX micro 1 900 457 11 69 4 373 1 10 447 0.0219 0.0111 0.0147 LDC-MAX micro ALL 2168 1195 138 643 41 373 6 132 1063 0.1105 0.0609 0.0785 LDC-MAX macro 0 0.1288 LDC-MAX macro 1 0.0210 LDC-MAX macro ALL 0.0902 ************************************************************* Run ID: BUPT_PRIS5 Did the run access the live Web during the evaluation window: No Run Number of most similar Cold Start KB Construction task submission: NA Did the run extract relations from the Cold Start source corpus: Yes Did the run generate meaningful confidence values: No Slot Filling Evaluation: Metric Hop GoldT Submit Correct Incorrect Inexact PIncorrect Dup Right Wrong Prec Recall F1 CSSF micro 0 4840 2649 335 2178 136 0 10 325 2324 0.1227 0.0671 0.0868 CSSF micro 1 3954 1665 25 208 15 1417 3 22 1643 0.0132 0.0056 0.0078 CSSF micro ALL 8794 4314 360 2386 151 1417 13 347 3967 0.0804 0.0395 0.0529 LDC-MEAN macro 0 0.1035 LDC-MEAN macro 1 0.0126 LDC-MEAN macro ALL 0.0709 LDC-MAX micro 0 1268 754 127 591 36 0 5 122 632 0.1618 0.0962 0.1207 LDC-MAX micro 1 900 469 11 69 4 385 1 10 459 0.0213 0.0111 0.0146 LDC-MAX micro ALL 2168 1223 138 660 40 385 6 132 1091 0.1079 0.0609 0.0779 LDC-MAX macro 0 0.1288 LDC-MAX macro 1 0.0210 LDC-MAX macro ALL 0.0902