========================================= TAC 2019 SM-KBP TASK 3 EVALUATION RESULTS ========================================= Team ID: GAIA Organization: USC-ISI; UIUC; Columbia University; University of Florida ********************** Description of Columns in Task 3 scores: - Column 1: Topic ID - Column 2: edges_submitted * number of edges in the truncated submitted hypotheses - Column 3: edges_correct * number of edges that have a correct justification (i.e., predicate justification is correct and linkable to object justification) - Column 4: edges_wrong * number of edges that have no correct justification - Column 5: edges_duplicate * number of correct edges that are in the same edge equivalence class as a previously seen correct edge for the same hypothesis - Column 6: edges_skipped * number of correct edges (for relations) that cannot be assigned to an edge equivalence class because the assessment is missing a subject equivalence class * these edges are ignored for the puposes of computing correctness * these edges are counted for all other purposes (e.g. when reporting coherence, relevance, and coverage) - Column 7: edges_coherent * number of edges assessed as coherent - Column 8: KE_submitted * number of event or relation clusters in the truncated submitted hypotheses - Column 9: KE_coherent * number of event or relation clusters assessed as coherent - Column 10: KE_Frel * number of event or relation clusters assessed as FullyRelevant - Column 11: KE_Prel * number of event or relation clusters assessed as PartiallyRelevant - Column 12: hyotheses_submitted * number of hypotheses submitted - Column 13: theories_matched * number of prevailing theories (partially) matched by the submitted hypotheses - Column 14: Mean hypothesis-level correctness * hypothesis-level correctness = (edges_correct - edges_duplicate - edges_skipped) / (edges_submitted - edges_duplicate - edges_skipped) , restricted to edges in hypothesis - Column 15: Mean hypothesis-level edge_coherence * hypothesis-level edge_coherence = edges_coherent / edges_submitted , restricted to edges in hypothesis - Column 16: Mean hypothesis-level KE_coherence * hypothesis-level KE_coherence = KE_coherent / KE_submitted , restricted to KEs in hypothesis - Column 17: Mean hypothesis-level relevance_strict * hypothesis-level relevance_strict = KE_Frel / KE_submitted , restricted to KEs in hypothesis - Column 18: Mean hypothesis-level relevance_lenient * hypothesis-level relevance_lenient = (KE_Frel + KE_Prel) / KE_submitted , restricted to KEs in hypothesis - Column 19: coverage * Recall of edges in prevailing theories - Column 20: Run ID ********************** Task 3 scores for each topic Topic #2 #3 #4 #5 #6 #7 #8 #9 #10 #11 #12 #13 #14 #15 #16 #17 #18 #19 Run_ID E101 71 42 29 2 14 42 41 32 15 14 4 1 0.5331 0.6375 0.7739 0.4360 0.7239 0.0065 BBN_1.BBN_TA2_v2.GAIA_2 E101 20 8 12 0 2 8 14 6 6 0 5 0 0.2190 0.2667 0.3033 0.3033 0.3033 0.0000 GAIA_1.GAIA_2.GAIA_1 E101 416 284 132 37 14 284 197 167 103 58 8 1 0.6398 0.6837 0.8479 0.5229 0.8175 0.0195 GAIA_1.GAIA_2.GAIA_2_v2 E101 413 270 143 21 13 262 198 153 90 59 8 1 0.6239 0.6345 0.7725 0.4542 0.7521 0.0179 GAIA_1.GAIA_2p.GAIA_2 E101 23 11 12 0 2 11 15 8 6 2 6 1 0.4972 0.5063 0.5444 0.4778 0.5444 0.0089 GAIA_2.GAIA_2.GAIA_1 E101 12 11 1 0 3 11 10 10 9 1 10 1 0.9000 0.9500 1.0000 0.9000 1.0000 0.0026 LDC_2.LDC_2.GAIA_1 E101 296 250 46 28 0 250 78 78 48 30 8 1 0.8281 0.8442 1.0000 0.6153 1.0000 0.0053 LDC_2.LDC_2.GAIA_2 E101 177 81 96 0 31 81 93 66 24 42 8 0 0.3451 0.4581 0.7105 0.2588 0.7105 0.0000 OPERA_TA1a_hans_V2.BBN_TA2_v1.GAIA_2 E101 5 4 1 0 4 4 4 4 3 1 4 0 0.7500 0.8750 1.0000 0.7500 1.0000 0.0000 OPERA_TA1a_hans_V3.OPERA_TA2_hans_V5.GAIA_1 E102 158 109 49 3 21 109 71 69 32 36 8 1 0.6478 0.6986 0.9725 0.4685 0.9547 0.0093 BBN_1.BBN_TA2_v2.GAIA_2 E102 40 14 26 0 1 13 24 10 6 3 4 0 0.1875 0.1833 0.2361 0.1458 0.2153 0.0000 GAIA_1.GAIA_2.GAIA_1 E102 192 96 96 6 9 96 73 49 12 37 8 3 0.4587 0.5016 0.6764 0.1559 0.6764 0.0296 GAIA_1.GAIA_2.GAIA_2_v2 E102 208 90 118 7 12 90 83 55 24 31 8 1 0.3899 0.4419 0.6636 0.2913 0.6636 0.0103 GAIA_1.GAIA_2p.GAIA_2 E102 22 12 10 0 0 12 16 12 7 5 3 0 0.3889 0.3889 0.5185 0.2963 0.5185 0.0000 GAIA_2.GAIA_2.GAIA_1 E102 3 0 3 0 0 0 2 0 0 0 2 0 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 LDC_2.LDC_2.GAIA_1 E102 110 86 24 10 0 86 24 24 15 9 8 1 0.7668 0.7809 1.0000 0.6250 1.0000 0.0105 LDC_2.LDC_2.GAIA_2 E102 199 95 104 0 3 95 107 57 41 16 8 0 0.4711 0.4778 0.5323 0.3827 0.5323 0.0000 OPERA_TA1a_hans_V2.BBN_TA2_v1.GAIA_2 E102 3 3 0 0 0 3 2 2 2 0 2 0 1.0000 1.0000 1.0000 1.0000 1.0000 0.0000 OPERA_TA1a_hans_V3.OPERA_TA2_hans_V5.GAIA_1 E103 136 68 68 13 16 68 49 33 12 21 8 0 0.3625 0.5007 0.6767 0.2692 0.6767 0.0000 BBN_1.BBN_TA2_v2.GAIA_2 E103 36 15 21 0 2 15 29 13 9 4 4 0 0.4045 0.4442 0.4833 0.3208 0.4833 0.0000 GAIA_1.GAIA_2.GAIA_1 E103 256 69 187 0 15 69 90 42 16 26 8 2 0.2195 0.2650 0.4722 0.1709 0.4722 0.0651 GAIA_1.GAIA_2.GAIA_2_v2 E103 224 94 130 3 26 94 85 58 25 33 7 2 0.3282 0.4080 0.6538 0.2922 0.6538 0.0651 GAIA_1.GAIA_2p.GAIA_2 E103 83 35 48 2 2 35 59 27 18 7 6 0 0.3983 0.4177 0.4415 0.3141 0.4137 0.0000 GAIA_2.GAIA_2.GAIA_1 E103 27 27 0 8 1 27 9 9 9 0 9 1 1.0000 1.0000 1.0000 1.0000 1.0000 0.0078 LDC_2.LDC_2.GAIA_1 E103 165 148 17 37 0 148 32 32 17 15 8 0 0.8672 0.8967 1.0000 0.5312 1.0000 0.0000 LDC_2.LDC_2.GAIA_2 E103 133 42 91 0 4 42 68 25 20 5 8 0 0.4431 0.4552 0.4845 0.4298 0.4845 0.0000 OPERA_TA1a_hans_V2.BBN_TA2_v1.GAIA_2 ALL 365 219 146 18 51 219 161 134 59 71 20 2 0.5107 0.6072 0.8145 0.3823 0.7973 0.0061 BBN_1.BBN_TA2_v2.GAIA_2 ALL 96 37 59 0 5 36 67 29 21 7 13 0 0.2664 0.2957 0.3380 0.2603 0.3316 0.0000 GAIA_1.GAIA_2.GAIA_1 ALL 864 449 415 43 38 449 360 258 131 121 24 6 0.4393 0.4834 0.6655 0.2832 0.6554 0.0320 GAIA_1.GAIA_2.GAIA_2_v2 ALL 845 454 391 31 51 446 366 266 139 123 23 4 0.4525 0.4986 0.6985 0.3482 0.6914 0.0248 GAIA_1.GAIA_2p.GAIA_2 ALL 128 58 70 2 4 58 90 47 31 14 15 1 0.4360 0.4474 0.4981 0.3760 0.4870 0.0042 GAIA_2.GAIA_2.GAIA_1 ALL 42 38 4 8 4 38 21 19 18 1 21 2 0.8571 0.8810 0.9048 0.8571 0.9048 0.0028 LDC_2.LDC_2.GAIA_1 ALL 571 484 87 75 0 484 134 134 80 54 24 2 0.8207 0.8406 1.0000 0.5905 1.0000 0.0060 LDC_2.LDC_2.GAIA_2 ALL 509 218 291 0 38 218 268 148 85 63 24 0 0.4197 0.4637 0.5758 0.3571 0.5758 0.0000 OPERA_TA1a_hans_V2.BBN_TA2_v1.GAIA_2 ALL 8 7 1 0 4 7 6 6 5 1 6 0 0.8333 0.9167 1.0000 0.8333 1.0000 0.0000 OPERA_TA1a_hans_V3.OPERA_TA2_hans_V5.GAIA_1