=========================================================== TAC KBP 2015 COLD START KB CONSTRUCTION EVALUATION RESULTS =========================================================== Team ID: NYU Organization: New York University ************************************************************* Run ID: NYU1 Did the run access the live Web during the evaluation window: No Run Number of most similar Cold Start Slot Filling task submission: NA Did the run extract relations from the Cold Start source corpus: Yes Did the run generate meaningful confidence values: No Entity Discovery Evaluation: Prec Recall F1 Metric 0.799 0.664 0.725 strong_mention_match 0.711 0.592 0.646 strong_typed_mention_match 0.000 0.000 0.000 entity_match 0.728 0.503 0.595 b_cubed 0.704 0.585 0.639 mention_ceaf 0.656 0.545 0.595 typed_mention_ceaf 0.656 0.545 0.595 typed_mention_ceaf_plus Slot Filling Evaluation: Metric Hop GoldT Submit Correct Incorrect Inexact PIncorrect Dup Right Wrong Prec Recall F1 CSSF micro 0 4840 1734 751 883 100 0 69 682 1052 0.3933 0.1409 0.2075 CSSF micro 1 3954 3330 426 1680 60 1164 20 406 2925 0.1219 0.1027 0.1115 CSSF micro ALL 8794 5064 1177 2563 160 1164 89 1088 3977 0.2148 0.1237 0.1570 LDC-MEAN macro 0 0.1666 LDC-MEAN macro 1 0.0724 LDC-MEAN macro ALL 0.1329 LDC-MAX micro 0 1268 575 272 273 30 0 20 252 323 0.4383 0.1987 0.2735 LDC-MAX micro 1 900 965 126 414 17 408 5 121 845 0.1254 0.1344 0.1298 LDC-MAX micro ALL 2168 1540 398 687 47 408 25 373 1168 0.2422 0.1720 0.2012 LDC-MAX macro 0 0.2315 LDC-MAX macro 1 0.1079 LDC-MAX macro ALL 0.1873 ************************************************************* Run ID: NYU2 Did the run access the live Web during the evaluation window: No Run Number of most similar Cold Start Slot Filling task submission: NA Did the run extract relations from the Cold Start source corpus: Yes Did the run generate meaningful confidence values: No Entity Discovery Evaluation: Prec Recall F1 Metric 0.799 0.664 0.725 strong_mention_match 0.711 0.592 0.646 strong_typed_mention_match 0.000 0.000 0.000 entity_match 0.728 0.503 0.595 b_cubed 0.704 0.585 0.639 mention_ceaf 0.656 0.545 0.595 typed_mention_ceaf 0.656 0.545 0.595 typed_mention_ceaf_plus Slot Filling Evaluation: Metric Hop GoldT Submit Correct Incorrect Inexact PIncorrect Dup Right Wrong Prec Recall F1 CSSF micro 0 4840 1328 709 515 104 0 65 644 684 0.4849 0.1331 0.2088 CSSF micro 1 3954 1346 284 480 41 541 20 264 1083 0.1961 0.0668 0.0996 CSSF micro ALL 8794 2674 993 995 145 541 85 908 1767 0.3396 0.1033 0.1584 LDC-MEAN macro 0 0.1551 LDC-MEAN macro 1 0.0614 LDC-MEAN macro ALL 0.1216 LDC-MAX micro 0 1268 439 252 161 26 0 20 232 207 0.5285 0.1830 0.2718 LDC-MAX micro 1 900 415 97 139 14 165 5 92 324 0.2217 0.1022 0.1399 LDC-MAX micro ALL 2168 854 349 300 40 165 25 324 531 0.3794 0.1494 0.2144 LDC-MAX macro 0 0.2261 LDC-MAX macro 1 0.0963 LDC-MAX macro ALL 0.1796 ************************************************************* Run ID: NYU3 Did the run access the live Web during the evaluation window: No Run Number of most similar Cold Start Slot Filling task submission: NA Did the run extract relations from the Cold Start source corpus: Yes Did the run generate meaningful confidence values: No Entity Discovery Evaluation: Prec Recall F1 Metric 0.799 0.664 0.725 strong_mention_match 0.711 0.592 0.646 strong_typed_mention_match 0.000 0.000 0.000 entity_match 0.728 0.503 0.595 b_cubed 0.704 0.585 0.639 mention_ceaf 0.656 0.545 0.595 typed_mention_ceaf 0.656 0.545 0.595 typed_mention_ceaf_plus Slot Filling Evaluation: Metric Hop GoldT Submit Correct Incorrect Inexact PIncorrect Dup Right Wrong Prec Recall F1 CSSF micro 0 4840 1199 670 422 107 0 62 608 591 0.5071 0.1256 0.2014 CSSF micro 1 3954 1212 273 475 36 428 20 253 960 0.2087 0.0640 0.0979 CSSF micro ALL 8794 2411 943 897 143 428 82 861 1551 0.3571 0.0979 0.1537 LDC-MEAN macro 0 0.1479 LDC-MEAN macro 1 0.0599 LDC-MEAN macro ALL 0.1164 LDC-MAX micro 0 1268 413 237 144 32 0 19 218 195 0.5278 0.1719 0.2594 LDC-MAX micro 1 900 371 91 136 12 132 5 86 286 0.2318 0.0956 0.1353 LDC-MAX micro ALL 2168 784 328 280 44 132 24 304 481 0.3878 0.1402 0.2060 LDC-MAX macro 0 0.2152 LDC-MAX macro 1 0.0922 LDC-MAX macro ALL 0.1712 ************************************************************* Run ID: NYU4 Did the run access the live Web during the evaluation window: No Run Number of most similar Cold Start Slot Filling task submission: NA Did the run extract relations from the Cold Start source corpus: Yes Did the run generate meaningful confidence values: No Entity Discovery Evaluation: Prec Recall F1 Metric 0.798 0.671 0.729 strong_mention_match 0.710 0.597 0.649 strong_typed_mention_match 0.000 0.000 0.000 entity_match 0.727 0.512 0.600 b_cubed 0.703 0.592 0.643 mention_ceaf 0.655 0.551 0.598 typed_mention_ceaf 0.655 0.551 0.598 typed_mention_ceaf_plus Slot Filling Evaluation: Metric Hop GoldT Submit Correct Incorrect Inexact PIncorrect Dup Right Wrong Prec Recall F1 CSSF micro 0 4840 1739 751 888 100 0 69 682 1057 0.3922 0.1409 0.2073 CSSF micro 1 3954 3691 442 2016 68 1165 20 422 3270 0.1143 0.1067 0.1104 CSSF micro ALL 8794 5430 1193 2904 168 1165 89 1104 4327 0.2033 0.1255 0.1552 LDC-MEAN macro 0 0.1666 LDC-MEAN macro 1 0.0850 LDC-MEAN macro ALL 0.1374 LDC-MAX micro 0 1268 585 272 283 30 0 20 252 333 0.4308 0.1987 0.2720 LDC-MAX micro 1 900 1177 132 498 19 528 5 127 1051 0.1079 0.1411 0.1223 LDC-MAX micro ALL 2168 1762 404 781 49 528 25 379 1384 0.2151 0.1748 0.1929 LDC-MAX macro 0 0.2315 LDC-MAX macro 1 0.1206 LDC-MAX macro ALL 0.1918 ************************************************************* Run ID: NYU5 Did the run access the live Web during the evaluation window: No Run Number of most similar Cold Start Slot Filling task submission: NA Did the run extract relations from the Cold Start source corpus: Yes Did the run generate meaningful confidence values: No Entity Discovery Evaluation: Prec Recall F1 Metric 0.798 0.671 0.729 strong_mention_match 0.710 0.597 0.649 strong_typed_mention_match 0.000 0.000 0.000 entity_match 0.727 0.512 0.600 b_cubed 0.703 0.592 0.643 mention_ceaf 0.655 0.551 0.598 typed_mention_ceaf 0.655 0.551 0.598 typed_mention_ceaf_plus Slot Filling Evaluation: Metric Hop GoldT Submit Correct Incorrect Inexact PIncorrect Dup Right Wrong Prec Recall F1 CSSF micro 0 4840 1330 709 517 104 0 65 644 686 0.4842 0.1331 0.2088 CSSF micro 1 3954 1730 296 836 57 541 20 276 1455 0.1595 0.0698 0.0971 CSSF micro ALL 8794 3060 1005 1353 161 541 85 920 2141 0.3007 0.1046 0.1552 LDC-MEAN macro 0 0.1554 LDC-MEAN macro 1 0.0615 LDC-MEAN macro ALL 0.1218 LDC-MAX micro 0 1268 448 252 167 29 0 20 232 216 0.5179 0.1830 0.2704 LDC-MAX micro 1 900 511 100 228 18 165 5 95 417 0.1859 0.1056 0.1347 LDC-MAX micro ALL 2168 959 352 395 47 165 25 327 633 0.3410 0.1508 0.2091 LDC-MAX macro 0 0.2265 LDC-MAX macro 1 0.0963 LDC-MAX macro ALL 0.1799