=========================================================== TAC KBP 2015 COLD START KB CONSTRUCTION EVALUATION RESULTS =========================================================== Team ID: Stanford Organization: Stanford University ************************************************************* Run ID: Stanford1 Did the run access the live Web during the evaluation window: No Run Number of most similar Cold Start Slot Filling task submission: 4 Did the run extract relations from the Cold Start source corpus: Yes Did the run generate meaningful confidence values: Yes Entity Discovery Evaluation: Prec Recall F1 Metric 0.631 0.710 0.668 strong_mention_match 0.579 0.651 0.613 strong_typed_mention_match 0.000 0.000 0.000 entity_match 0.522 0.580 0.550 b_cubed 0.571 0.643 0.605 mention_ceaf 0.531 0.598 0.562 typed_mention_ceaf 0.531 0.598 0.562 typed_mention_ceaf_plus Slot Filling Evaluation: Metric Hop GoldT Submit Correct Incorrect Inexact PIncorrect Dup Right Wrong Prec Recall F1 CSSF micro 0 4840 1153 662 394 97 0 38 624 529 0.5412 0.1289 0.2082 CSSF micro 1 3954 538 219 136 24 159 11 208 330 0.3866 0.0526 0.0926 CSSF micro ALL 8794 1691 881 530 121 159 49 832 859 0.4920 0.0946 0.1587 LDC-MEAN macro 0 0.2059 LDC-MEAN macro 1 0.0613 LDC-MEAN macro ALL 0.1541 LDC-MAX micro 0 1268 392 229 138 25 0 13 216 176 0.5510 0.1703 0.2602 LDC-MAX micro 1 900 153 62 32 8 51 3 59 94 0.3856 0.0656 0.1121 LDC-MAX micro ALL 2168 545 291 170 33 51 16 275 270 0.5046 0.1268 0.2027 LDC-MAX macro 0 0.2408 LDC-MAX macro 1 0.0839 LDC-MAX macro ALL 0.1846 ************************************************************* Run ID: Stanford2 Did the run access the live Web during the evaluation window: No Run Number of most similar Cold Start Slot Filling task submission: 3 Did the run extract relations from the Cold Start source corpus: Yes Did the run generate meaningful confidence values: Yes Entity Discovery Evaluation: Prec Recall F1 Metric 0.631 0.710 0.668 strong_mention_match 0.579 0.652 0.614 strong_typed_mention_match 0.000 0.000 0.000 entity_match 0.522 0.580 0.550 b_cubed 0.571 0.643 0.605 mention_ceaf 0.531 0.598 0.563 typed_mention_ceaf 0.531 0.598 0.563 typed_mention_ceaf_plus Slot Filling Evaluation: Metric Hop GoldT Submit Correct Incorrect Inexact PIncorrect Dup Right Wrong Prec Recall F1 CSSF micro 0 4840 4089 1211 2671 207 0 54 1157 2932 0.2830 0.2390 0.2592 CSSF micro 1 3954 7078 1143 2689 177 3069 16 1127 5951 0.1592 0.2850 0.2043 CSSF micro ALL 8794 11167 2354 5360 384 3069 70 2284 8883 0.2045 0.2597 0.2288 LDC-MEAN macro 0 0.2838 LDC-MEAN macro 1 0.1378 LDC-MEAN macro ALL 0.2315 LDC-MAX micro 0 1268 1198 411 725 62 0 20 391 807 0.3264 0.3084 0.3171 LDC-MAX micro 1 900 2181 278 661 47 1195 4 274 1907 0.1256 0.3044 0.1779 LDC-MAX micro ALL 2168 3379 689 1386 109 1195 24 665 2714 0.1968 0.3067 0.2398 LDC-MAX macro 0 0.3254 LDC-MAX macro 1 0.1722 LDC-MAX macro ALL 0.2705