========================================= TAC 2019 SM-KBP TASK 3 EVALUATION RESULTS ========================================= Team ID: UTexas Organization: The University of Texas at Austin ********************** Description of Columns in Task 3 scores: - Column 1: Topic ID - Column 2: edges_submitted * number of edges in the truncated submitted hypotheses - Column 3: edges_correct * number of edges that have a correct justification (i.e., predicate justification is correct and linkable to object justification) - Column 4: edges_wrong * number of edges that have no correct justification - Column 5: edges_duplicate * number of correct edges that are in the same edge equivalence class as a previously seen correct edge for the same hypothesis - Column 6: edges_skipped * number of correct edges (for relations) that cannot be assigned to an edge equivalence class because the assessment is missing a subject equivalence class * these edges are ignored for the puposes of computing correctness * these edges are counted for all other purposes (e.g. when reporting coherence, relevance, and coverage) - Column 7: edges_coherent * number of edges assessed as coherent - Column 8: KE_submitted * number of event or relation clusters in the truncated submitted hypotheses - Column 9: KE_coherent * number of event or relation clusters assessed as coherent - Column 10: KE_Frel * number of event or relation clusters assessed as FullyRelevant - Column 11: KE_Prel * number of event or relation clusters assessed as PartiallyRelevant - Column 12: hyotheses_submitted * number of hypotheses submitted - Column 13: theories_matched * number of prevailing theories (partially) matched by the submitted hypotheses - Column 14: Mean hypothesis-level correctness * hypothesis-level correctness = (edges_correct - edges_duplicate - edges_skipped) / (edges_submitted - edges_duplicate - edges_skipped) , restricted to edges in hypothesis - Column 15: Mean hypothesis-level edge_coherence * hypothesis-level edge_coherence = edges_coherent / edges_submitted , restricted to edges in hypothesis - Column 16: Mean hypothesis-level KE_coherence * hypothesis-level KE_coherence = KE_coherent / KE_submitted , restricted to KEs in hypothesis - Column 17: Mean hypothesis-level relevance_strict * hypothesis-level relevance_strict = KE_Frel / KE_submitted , restricted to KEs in hypothesis - Column 18: Mean hypothesis-level relevance_lenient * hypothesis-level relevance_lenient = (KE_Frel + KE_Prel) / KE_submitted , restricted to KEs in hypothesis - Column 19: coverage * Recall of edges in prevailing theories - Column 20: Run ID ********************** Task 3 scores for each topic Topic #2 #3 #4 #5 #6 #7 #8 #9 #10 #11 #12 #13 #14 #15 #16 #17 #18 #19 Run_ID E101 447 197 250 14 28 192 230 158 58 100 14 0 0.3862 0.4333 0.6903 0.2545 0.6903 0.0000 GAIA_1-OPERA_3.Colorado_1.UTexas_2 E101 349 319 30 2 60 318 216 193 190 4 14 0 0.8987 0.9123 0.8944 0.8807 0.8991 0.0000 LDC_2.LDC_2.UTexas_3 E101 341 135 206 14 0 129 248 133 98 21 14 0 0.3719 0.3806 0.5398 0.3964 0.4828 0.0000 OPERA_4.Colorado_1.UTexas_2 E101 330 136 194 6 0 134 234 123 69 42 14 0 0.3987 0.4030 0.5262 0.2936 0.4739 0.0000 OPERA_TA1a_aditi_V5.OPERA_TA2_aditi_V5.UTexas_2 E102 346 138 208 0 110 111 180 97 0 124 14 0 0.1187 0.3201 0.5375 0.0000 0.6886 0.0000 GAIA_1-OPERA_3.Colorado_1.UTexas_2 E102 320 264 56 49 17 260 140 131 115 16 14 3 0.7928 0.8205 0.9311 0.8237 0.9349 0.0208 LDC_2.LDC_2.UTexas_3 E102 279 108 171 6 0 107 165 93 49 38 14 0 0.3733 0.3836 0.5670 0.2872 0.5316 0.0000 OPERA_4.Colorado_1.UTexas_2 E102 178 85 93 42 5 84 118 85 47 28 14 0 0.2923 0.4713 0.7198 0.3913 0.6341 0.0000 OPERA_TA1a_aditi_V5.OPERA_TA2_aditi_V5.UTexas_2 E103 530 172 358 0 39 172 235 139 48 87 14 0 0.2748 0.3286 0.6045 0.2153 0.5866 0.0000 GAIA_1-OPERA_3.Colorado_1.UTexas_2 E103 428 364 64 130 6 363 181 156 142 15 14 1 0.7884 0.8507 0.8684 0.7892 0.8763 0.0155 LDC_2.LDC_2.UTexas_3 E103 427 147 280 0 14 147 182 126 28 35 14 1 0.3207 0.3430 0.6970 0.1455 0.3485 0.0155 OPERA_4.Colorado_1.UTexas_2 E103 273 128 145 0 0 128 182 93 58 7 14 1 0.4683 0.4683 0.5094 0.3036 0.3427 0.0159 OPERA_TA1a_aditi_V5.OPERA_TA2_aditi_V5.UTexas_2 ALL 1323 507 816 14 177 475 645 394 106 311 42 0 0.2599 0.3607 0.6108 0.1566 0.6552 0.0000 GAIA_1-OPERA_3.Colorado_1.UTexas_2 ALL 1097 947 150 181 83 941 537 480 447 35 42 4 0.8266 0.8612 0.8979 0.8312 0.9034 0.0100 LDC_2.LDC_2.UTexas_3 ALL 1047 390 657 20 14 383 595 352 175 94 42 1 0.3553 0.3691 0.6012 0.2763 0.4543 0.0031 OPERA_4.Colorado_1.UTexas_2 ALL 781 349 432 48 5 346 534 301 174 77 42 1 0.3864 0.4475 0.5851 0.3295 0.4836 0.0032 OPERA_TA1a_aditi_V5.OPERA_TA2_aditi_V5.UTexas_2