=========================================================== TAC KBP 2015 COLD START KB CONSTRUCTION EVALUATION RESULTS =========================================================== Team ID: UMass_IESL Organization: University of Massachusetts, Amherst ************************************************************* Run ID: UMass_IESL1 Did the run access the live Web during the evaluation window: No Run Number of most similar Cold Start Slot Filling task submission: 5 Did the run extract relations from the Cold Start source corpus: Yes Did the run generate meaningful confidence values: Yes Entity Discovery Evaluation: Prec Recall F1 Metric 0.746 0.636 0.687 strong_mention_match 0.693 0.590 0.637 strong_typed_mention_match 0.000 0.000 0.000 entity_match 0.696 0.471 0.562 b_cubed 0.667 0.568 0.614 mention_ceaf 0.641 0.546 0.590 typed_mention_ceaf 0.641 0.546 0.590 typed_mention_ceaf_plus Slot Filling Evaluation: Metric Hop GoldT Submit Correct Incorrect Inexact PIncorrect Dup Right Wrong Prec Recall F1 CSSF micro 0 4840 2865 776 1916 173 0 72 704 2161 0.2457 0.1455 0.1827 CSSF micro 1 3954 4791 373 479 41 3898 37 336 4455 0.0701 0.0850 0.0768 CSSF micro ALL 8794 7656 1149 2395 214 3898 109 1040 6616 0.1358 0.1183 0.1264 LDC-MEAN macro 0 0.1622 LDC-MEAN macro 1 0.0753 LDC-MEAN macro ALL 0.1311 LDC-MAX micro 0 1268 989 254 679 56 0 20 234 755 0.2366 0.1845 0.2074 LDC-MAX micro 1 900 1337 104 161 12 1060 10 94 1243 0.0703 0.1044 0.0840 LDC-MAX micro ALL 2168 2326 358 840 68 1060 30 328 1998 0.1410 0.1513 0.1460 LDC-MAX macro 0 0.1952 LDC-MAX macro 1 0.0922 LDC-MAX macro ALL 0.1583 ************************************************************* Run ID: UMass_IESL2 Did the run access the live Web during the evaluation window: No Run Number of most similar Cold Start Slot Filling task submission: 3 Did the run extract relations from the Cold Start source corpus: Yes Did the run generate meaningful confidence values: Yes Entity Discovery Evaluation: Prec Recall F1 Metric 0.746 0.636 0.687 strong_mention_match 0.693 0.590 0.637 strong_typed_mention_match 0.000 0.000 0.000 entity_match 0.696 0.471 0.562 b_cubed 0.667 0.568 0.614 mention_ceaf 0.641 0.546 0.590 typed_mention_ceaf 0.641 0.546 0.590 typed_mention_ceaf_plus Slot Filling Evaluation: Metric Hop GoldT Submit Correct Incorrect Inexact PIncorrect Dup Right Wrong Prec Recall F1 CSSF micro 0 4840 3708 857 2666 185 0 77 780 2928 0.2104 0.1612 0.1825 CSSF micro 1 3954 7762 416 788 45 6513 40 376 7386 0.0484 0.0951 0.0642 CSSF micro ALL 8794 11470 1273 3454 230 6513 117 1156 10314 0.1008 0.1315 0.1141 LDC-MEAN macro 0 0.1672 LDC-MEAN macro 1 0.0837 LDC-MEAN macro ALL 0.1373 LDC-MAX micro 0 1268 1356 279 1016 61 0 21 258 1098 0.1903 0.2035 0.1966 LDC-MAX micro 1 900 2305 128 277 14 1886 12 116 2189 0.0503 0.1289 0.0724 LDC-MAX micro ALL 2168 3661 407 1293 75 1886 33 374 3287 0.1022 0.1725 0.1283 LDC-MAX macro 0 0.2041 LDC-MAX macro 1 0.1029 LDC-MAX macro ALL 0.1679 ************************************************************* Run ID: UMass_IESL3 Did the run access the live Web during the evaluation window: No Run Number of most similar Cold Start Slot Filling task submission: 5 Did the run extract relations from the Cold Start source corpus: Yes Did the run generate meaningful confidence values: Yes Entity Discovery Evaluation: Prec Recall F1 Metric 0.746 0.636 0.687 strong_mention_match 0.693 0.590 0.637 strong_typed_mention_match 0.000 0.000 0.000 entity_match 0.696 0.471 0.562 b_cubed 0.667 0.568 0.614 mention_ceaf 0.641 0.546 0.590 typed_mention_ceaf 0.641 0.546 0.590 typed_mention_ceaf_plus Slot Filling Evaluation: Metric Hop GoldT Submit Correct Incorrect Inexact PIncorrect Dup Right Wrong Prec Recall F1 CSSF micro 0 4840 2396 593 1658 145 0 65 528 1868 0.2204 0.1091 0.1459 CSSF micro 1 3954 3444 201 283 27 2933 10 191 3253 0.0555 0.0483 0.0516 CSSF micro ALL 8794 5840 794 1941 172 2933 75 719 5121 0.1231 0.0818 0.0983 LDC-MEAN macro 0 0.1461 LDC-MEAN macro 1 0.0572 LDC-MEAN macro ALL 0.1143 LDC-MAX micro 0 1268 835 202 584 49 0 18 184 651 0.2204 0.1451 0.1750 LDC-MAX micro 1 900 1143 63 83 8 989 3 60 1083 0.0525 0.0667 0.0587 LDC-MAX micro ALL 2168 1978 265 667 57 989 21 244 1734 0.1234 0.1125 0.1177 LDC-MAX macro 0 0.1759 LDC-MAX macro 1 0.0666 LDC-MAX macro ALL 0.1367 ************************************************************* Run ID: UMass_IESL4 Did the run access the live Web during the evaluation window: No Run Number of most similar Cold Start Slot Filling task submission: 3 Did the run extract relations from the Cold Start source corpus: Yes Did the run generate meaningful confidence values: Yes Entity Discovery Evaluation: Prec Recall F1 Metric 0.746 0.636 0.687 strong_mention_match 0.693 0.590 0.637 strong_typed_mention_match 0.000 0.000 0.000 entity_match 0.696 0.471 0.562 b_cubed 0.667 0.568 0.614 mention_ceaf 0.641 0.546 0.590 typed_mention_ceaf 0.641 0.546 0.590 typed_mention_ceaf_plus Slot Filling Evaluation: Metric Hop GoldT Submit Correct Incorrect Inexact PIncorrect Dup Right Wrong Prec Recall F1 CSSF micro 0 4840 2459 653 1660 146 0 65 588 1871 0.2391 0.1215 0.1611 CSSF micro 1 3954 3093 183 264 24 2622 5 178 2915 0.0575 0.0450 0.0505 CSSF micro ALL 8794 5552 836 1924 170 2622 70 766 4786 0.1380 0.0871 0.1068 LDC-MEAN macro 0 0.1489 LDC-MEAN macro 1 0.0644 LDC-MEAN macro ALL 0.1187 LDC-MAX micro 0 1268 871 220 601 50 0 18 202 669 0.2319 0.1593 0.1889 LDC-MAX micro 1 900 1144 60 77 7 1000 2 58 1086 0.0507 0.0644 0.0568 LDC-MAX micro ALL 2168 2015 280 678 57 1000 20 260 1755 0.1290 0.1199 0.1243 LDC-MAX macro 0 0.1809 LDC-MAX macro 1 0.0756 LDC-MAX macro ALL 0.1432