=========================================================== TAC KBP 2015 COLD START KB CONSTRUCTION EVALUATION RESULTS =========================================================== Team ID: BBN Organization: BBN ************************************************************* Run ID: BBN1 Did the run access the live Web during the evaluation window: No Run Number of most similar Cold Start Slot Filling task submission: NA Did the run extract relations from the Cold Start source corpus: Yes Did the run generate meaningful confidence values: Yes Entity Discovery Evaluation: Prec Recall F1 Metric 0.814 0.735 0.772 strong_mention_match 0.771 0.696 0.731 strong_typed_mention_match 0.000 0.000 0.000 entity_match 0.748 0.612 0.673 b_cubed 0.752 0.678 0.713 mention_ceaf 0.722 0.651 0.684 typed_mention_ceaf 0.722 0.651 0.684 typed_mention_ceaf_plus Slot Filling Evaluation: Metric Hop GoldT Submit Correct Incorrect Inexact PIncorrect Dup Right Wrong Prec Recall F1 CSSF micro 0 4840 3472 1668 1657 147 0 74 1594 1878 0.4591 0.3293 0.3835 CSSF micro 1 3954 3701 848 944 94 1815 47 801 2905 0.2164 0.2026 0.2093 CSSF micro ALL 8794 7173 2516 2601 241 1815 121 2395 4783 0.3339 0.2723 0.3000 LDC-MEAN macro 0 0.3275 LDC-MEAN macro 1 0.1539 LDC-MEAN macro ALL 0.2654 LDC-MAX micro 0 1268 1008 505 459 44 0 19 486 522 0.4821 0.3833 0.4271 LDC-MAX micro 1 900 1242 219 242 23 758 10 209 1034 0.1683 0.2322 0.1951 LDC-MAX micro ALL 2168 2250 724 701 67 758 29 695 1556 0.3089 0.3206 0.3146 LDC-MAX macro 0 0.3749 LDC-MAX macro 1 0.1815 LDC-MAX macro ALL 0.3057 ************************************************************* Run ID: BBN2 Did the run access the live Web during the evaluation window: No Run Number of most similar Cold Start Slot Filling task submission: NA Did the run extract relations from the Cold Start source corpus: Yes Did the run generate meaningful confidence values: Yes Entity Discovery Evaluation: Prec Recall F1 Metric 0.814 0.735 0.772 strong_mention_match 0.771 0.696 0.731 strong_typed_mention_match 0.000 0.000 0.000 entity_match 0.753 0.588 0.661 b_cubed 0.728 0.656 0.690 mention_ceaf 0.697 0.629 0.661 typed_mention_ceaf 0.697 0.629 0.661 typed_mention_ceaf_plus Slot Filling Evaluation: Metric Hop GoldT Submit Correct Incorrect Inexact PIncorrect Dup Right Wrong Prec Recall F1 CSSF micro 0 4840 2971 1509 1340 122 0 71 1438 1533 0.4840 0.2971 0.3682 CSSF micro 1 3954 2131 703 754 68 606 42 661 1475 0.3102 0.1672 0.2173 CSSF micro ALL 8794 5102 2212 2094 190 606 113 2099 3008 0.4114 0.2387 0.3021 LDC-MEAN macro 0 0.3224 LDC-MEAN macro 1 0.1447 LDC-MEAN macro ALL 0.2588 LDC-MAX micro 0 1268 890 457 394 39 0 18 439 451 0.4933 0.3462 0.4069 LDC-MAX micro 1 900 655 189 198 18 250 9 180 476 0.2748 0.2000 0.2315 LDC-MAX micro ALL 2168 1545 646 592 57 250 27 619 927 0.4006 0.2855 0.3334 LDC-MAX macro 0 0.3683 LDC-MAX macro 1 0.1732 LDC-MAX macro ALL 0.2985 ************************************************************* Run ID: BBN3 Did the run access the live Web during the evaluation window: No Run Number of most similar Cold Start Slot Filling task submission: NA Did the run extract relations from the Cold Start source corpus: Yes Did the run generate meaningful confidence values: Yes Entity Discovery Evaluation: Prec Recall F1 Metric 0.682 0.745 0.712 strong_mention_match 0.644 0.703 0.672 strong_typed_mention_match 0.000 0.000 0.000 entity_match 0.533 0.617 0.572 b_cubed 0.620 0.678 0.648 mention_ceaf 0.594 0.649 0.620 typed_mention_ceaf 0.594 0.649 0.620 typed_mention_ceaf_plus Slot Filling Evaluation: Metric Hop GoldT Submit Correct Incorrect Inexact PIncorrect Dup Right Wrong Prec Recall F1 CSSF micro 0 4840 3726 1663 1902 161 0 73 1590 2136 0.4267 0.3285 0.3712 CSSF micro 1 3954 3897 859 1075 103 1860 43 816 3086 0.2094 0.2064 0.2079 CSSF micro ALL 8794 7623 2522 2977 264 1860 116 2406 5222 0.3156 0.2736 0.2931 LDC-MEAN macro 0 0.3102 LDC-MEAN macro 1 0.1539 LDC-MEAN macro ALL 0.2543 LDC-MAX micro 0 1268 1088 508 533 47 0 19 489 599 0.4494 0.3856 0.4151 LDC-MAX micro 1 900 1306 222 274 25 785 9 213 1094 0.1631 0.2367 0.1931 LDC-MAX micro ALL 2168 2394 730 807 72 785 28 702 1693 0.2932 0.3238 0.3078 LDC-MAX macro 0 0.3639 LDC-MAX macro 1 0.1828 LDC-MAX macro ALL 0.2991 ************************************************************* Run ID: BBN4 Did the run access the live Web during the evaluation window: No Run Number of most similar Cold Start Slot Filling task submission: NA Did the run extract relations from the Cold Start source corpus: Yes Did the run generate meaningful confidence values: Yes Entity Discovery Evaluation: Prec Recall F1 Metric 0.814 0.735 0.772 strong_mention_match 0.771 0.696 0.731 strong_typed_mention_match 0.000 0.000 0.000 entity_match 0.748 0.612 0.673 b_cubed 0.752 0.678 0.713 mention_ceaf 0.722 0.651 0.684 typed_mention_ceaf 0.722 0.651 0.684 typed_mention_ceaf_plus Slot Filling Evaluation: Metric Hop GoldT Submit Correct Incorrect Inexact PIncorrect Dup Right Wrong Prec Recall F1 CSSF micro 0 4840 5096 1882 3030 184 0 76 1806 3290 0.3544 0.3731 0.3635 CSSF micro 1 3954 6216 1166 2108 133 2809 64 1102 5119 0.1773 0.2787 0.2167 CSSF micro ALL 8794 11312 3048 5138 317 2809 140 2908 8409 0.2571 0.3307 0.2893 LDC-MEAN macro 0 0.3322 LDC-MEAN macro 1 0.2032 LDC-MEAN macro ALL 0.2860 LDC-MAX micro 0 1268 1641 558 1031 52 0 20 538 1103 0.3278 0.4243 0.3699 LDC-MAX micro 1 900 1934 298 542 34 1060 14 284 1651 0.1468 0.3156 0.2004 LDC-MAX micro ALL 2168 3575 856 1573 86 1060 34 822 2754 0.2299 0.3792 0.2863 LDC-MAX macro 0 0.3793 LDC-MAX macro 1 0.2308 LDC-MAX macro ALL 0.3261 ************************************************************* Run ID: BBN5 Did the run access the live Web during the evaluation window: No Run Number of most similar Cold Start Slot Filling task submission: NA Did the run extract relations from the Cold Start source corpus: Yes Did the run generate meaningful confidence values: Yes Entity Discovery Evaluation: Prec Recall F1 Metric 0.810 0.751 0.779 strong_mention_match 0.768 0.712 0.739 strong_typed_mention_match 0.000 0.000 0.000 entity_match 0.742 0.620 0.675 b_cubed 0.735 0.682 0.707 mention_ceaf 0.705 0.654 0.679 typed_mention_ceaf 0.705 0.654 0.679 typed_mention_ceaf_plus Slot Filling Evaluation: Metric Hop GoldT Submit Correct Incorrect Inexact PIncorrect Dup Right Wrong Prec Recall F1 CSSF micro 0 4840 3313 1542 1595 176 0 60 1482 1831 0.4473 0.3062 0.3635 CSSF micro 1 3954 3867 834 971 98 1964 32 802 3070 0.2074 0.2028 0.2051 CSSF micro ALL 8794 7180 2376 2566 274 1964 92 2284 4901 0.3181 0.2597 0.2860 LDC-MEAN macro 0 0.2929 LDC-MEAN macro 1 0.1384 LDC-MEAN macro ALL 0.2376 LDC-MAX micro 0 1268 958 475 434 49 0 15 460 498 0.4802 0.3628 0.4133 LDC-MAX micro 1 900 1340 231 256 25 828 7 224 1117 0.1672 0.2489 0.2000 LDC-MAX micro ALL 2168 2298 706 690 74 828 22 684 1615 0.2977 0.3155 0.3063 LDC-MAX macro 0 0.3483 LDC-MAX macro 1 0.1657 LDC-MAX macro ALL 0.2829