=========================================================== TAC KBP 2015 COLD START KB CONSTRUCTION EVALUATION RESULTS =========================================================== Team ID: hltcoe Organization: Human Language Technology Center of Excellence ************************************************************* Run ID: hltcoe1 Did the run access the live Web during the evaluation window: No Run Number of most similar Cold Start Slot Filling task submission: NA Did the run extract relations from the Cold Start source corpus: Yes Did the run generate meaningful confidence values: No Entity Discovery Evaluation: Prec Recall F1 Metric 0.713 0.734 0.723 strong_mention_match 0.668 0.687 0.677 strong_typed_mention_match 0.000 0.000 0.000 entity_match 0.605 0.545 0.573 b_cubed 0.597 0.614 0.605 mention_ceaf 0.570 0.587 0.579 typed_mention_ceaf 0.570 0.587 0.579 typed_mention_ceaf_plus Slot Filling Evaluation: Metric Hop GoldT Submit Correct Incorrect Inexact PIncorrect Dup Right Wrong Prec Recall F1 CSSF micro 0 4840 1701 912 686 103 0 98 814 887 0.4785 0.1682 0.2489 CSSF micro 1 3954 1399 394 371 47 587 21 373 1026 0.2666 0.0943 0.1394 CSSF micro ALL 8794 3100 1306 1057 150 587 119 1187 1913 0.3829 0.1350 0.1996 LDC-MEAN macro 0 0.1923 LDC-MEAN macro 1 0.0695 LDC-MEAN macro ALL 0.1483 LDC-MAX micro 0 1268 573 317 225 31 0 25 292 281 0.5096 0.2303 0.3172 LDC-MAX micro 1 900 551 117 127 16 291 5 112 439 0.2033 0.1244 0.1544 LDC-MAX micro ALL 2168 1124 434 352 47 291 30 404 720 0.3594 0.1863 0.2454 LDC-MAX macro 0 0.2559 LDC-MAX macro 1 0.0952 LDC-MAX macro ALL 0.1983 ************************************************************* Run ID: hltcoe2 Did the run access the live Web during the evaluation window: No Run Number of most similar Cold Start Slot Filling task submission: NA Did the run extract relations from the Cold Start source corpus: Yes Did the run generate meaningful confidence values: No Entity Discovery Evaluation: Prec Recall F1 Metric 0.713 0.734 0.723 strong_mention_match 0.668 0.687 0.677 strong_typed_mention_match 0.000 0.000 0.000 entity_match 0.604 0.567 0.585 b_cubed 0.616 0.634 0.625 mention_ceaf 0.591 0.608 0.599 typed_mention_ceaf 0.591 0.608 0.599 typed_mention_ceaf_plus Slot Filling Evaluation: Metric Hop GoldT Submit Correct Incorrect Inexact PIncorrect Dup Right Wrong Prec Recall F1 CSSF micro 0 4840 2369 1181 1057 131 0 103 1078 1291 0.4550 0.2227 0.2991 CSSF micro 1 3954 2705 434 401 73 1797 15 419 2286 0.1549 0.1060 0.1258 CSSF micro ALL 8794 5074 1615 1458 204 1797 118 1497 3577 0.2950 0.1702 0.2159 LDC-MEAN macro 0 0.2152 LDC-MEAN macro 1 0.0943 LDC-MEAN macro ALL 0.1719 LDC-MAX micro 0 1268 642 347 260 35 0 25 322 320 0.5016 0.2539 0.3372 LDC-MAX micro 1 900 531 122 131 24 254 3 119 412 0.2241 0.1322 0.1663 LDC-MAX micro ALL 2168 1173 469 391 59 254 28 441 732 0.3760 0.2034 0.2640 LDC-MAX macro 0 0.2710 LDC-MAX macro 1 0.1183 LDC-MAX macro ALL 0.2164 ************************************************************* Run ID: hltcoe3 Did the run access the live Web during the evaluation window: No Run Number of most similar Cold Start Slot Filling task submission: NA Did the run extract relations from the Cold Start source corpus: Yes Did the run generate meaningful confidence values: No Entity Discovery Evaluation: Prec Recall F1 Metric 0.713 0.734 0.723 strong_mention_match 0.668 0.687 0.677 strong_typed_mention_match 0.000 0.000 0.000 entity_match 0.603 0.552 0.576 b_cubed 0.602 0.619 0.611 mention_ceaf 0.575 0.592 0.584 typed_mention_ceaf 0.575 0.592 0.584 typed_mention_ceaf_plus Slot Filling Evaluation: Metric Hop GoldT Submit Correct Incorrect Inexact PIncorrect Dup Right Wrong Prec Recall F1 CSSF micro 0 4840 1811 950 746 115 0 79 871 940 0.4809 0.1800 0.2619 CSSF micro 1 3954 1440 445 425 43 527 16 429 1011 0.2979 0.1085 0.1591 CSSF micro ALL 8794 3251 1395 1171 158 527 95 1300 1951 0.3999 0.1478 0.2159 LDC-MEAN macro 0 0.2142 LDC-MEAN macro 1 0.0912 LDC-MEAN macro ALL 0.1702 LDC-MAX micro 0 1268 595 327 234 34 0 22 305 290 0.5126 0.2405 0.3274 LDC-MAX micro 1 900 457 133 146 15 163 4 129 328 0.2823 0.1433 0.1901 LDC-MAX micro ALL 2168 1052 460 380 49 163 26 434 618 0.4125 0.2002 0.2696 LDC-MAX macro 0 0.2805 LDC-MAX macro 1 0.1144 LDC-MAX macro ALL 0.2210 ************************************************************* Run ID: hltcoe4 Did the run access the live Web during the evaluation window: No Run Number of most similar Cold Start Slot Filling task submission: NA Did the run extract relations from the Cold Start source corpus: Yes Did the run generate meaningful confidence values: No Entity Discovery Evaluation: Prec Recall F1 Metric 0.713 0.734 0.723 strong_mention_match 0.668 0.687 0.677 strong_typed_mention_match 0.000 0.000 0.000 entity_match 0.605 0.568 0.586 b_cubed 0.617 0.635 0.626 mention_ceaf 0.591 0.608 0.600 typed_mention_ceaf 0.591 0.608 0.600 typed_mention_ceaf_plus Slot Filling Evaluation: Metric Hop GoldT Submit Correct Incorrect Inexact PIncorrect Dup Right Wrong Prec Recall F1 CSSF micro 0 4840 2415 1208 1076 131 0 114 1094 1321 0.4530 0.2260 0.3016 CSSF micro 1 3954 2693 421 397 79 1796 5 416 2277 0.1545 0.1052 0.1252 CSSF micro ALL 8794 5108 1629 1473 210 1796 119 1510 3598 0.2956 0.1717 0.2172 LDC-MEAN macro 0 0.2194 LDC-MEAN macro 1 0.0946 LDC-MEAN macro ALL 0.1747 LDC-MAX micro 0 1268 708 357 315 36 0 28 329 379 0.4647 0.2595 0.3330 LDC-MAX micro 1 900 483 119 127 26 211 1 118 365 0.2443 0.1311 0.1706 LDC-MAX micro ALL 2168 1191 476 442 62 211 29 447 744 0.3753 0.2062 0.2662 LDC-MAX macro 0 0.2761 LDC-MAX macro 1 0.1192 LDC-MAX macro ALL 0.2199 ************************************************************* Run ID: hltcoe5 Did the run access the live Web during the evaluation window: No Run Number of most similar Cold Start Slot Filling task submission: NA Did the run extract relations from the Cold Start source corpus: Yes Did the run generate meaningful confidence values: No Entity Discovery Evaluation: Prec Recall F1 Metric 0.713 0.734 0.723 strong_mention_match 0.668 0.687 0.677 strong_typed_mention_match 0.000 0.000 0.000 entity_match 0.604 0.569 0.586 b_cubed 0.618 0.636 0.627 mention_ceaf 0.592 0.609 0.600 typed_mention_ceaf 0.592 0.609 0.600 typed_mention_ceaf_plus Slot Filling Evaluation: Metric Hop GoldT Submit Correct Incorrect Inexact PIncorrect Dup Right Wrong Prec Recall F1 CSSF micro 0 4840 2405 1218 1060 127 0 120 1098 1307 0.4565 0.2269 0.3031 CSSF micro 1 3954 2169 426 401 80 1262 10 416 1753 0.1918 0.1052 0.1359 CSSF micro ALL 8794 4574 1644 1461 207 1262 130 1514 3060 0.3310 0.1722 0.2265 LDC-MEAN macro 0 0.2209 LDC-MEAN macro 1 0.0854 LDC-MEAN macro ALL 0.1724 LDC-MAX micro 0 1268 707 362 311 34 0 30 332 375 0.4696 0.2618 0.3362 LDC-MAX micro 1 900 967 119 132 27 689 2 117 850 0.1210 0.1300 0.1253 LDC-MAX micro ALL 2168 1674 481 443 61 689 32 449 1225 0.2682 0.2071 0.2337 LDC-MAX macro 0 0.2806 LDC-MAX macro 1 0.1102 LDC-MAX macro ALL 0.2196