========================================= TAC 2019 SM-KBP TASK 3 EVALUATION RESULTS ========================================= Team ID: OPERA Organization: Carnegie Mellon University; USC Information Sciences Institute (ISI) ********************** Description of Columns in Task 3 scores: - Column 1: Topic ID - Column 2: edges_submitted * number of edges in the truncated submitted hypotheses - Column 3: edges_correct * number of edges that have a correct justification (i.e., predicate justification is correct and linkable to object justification) - Column 4: edges_wrong * number of edges that have no correct justification - Column 5: edges_duplicate * number of correct edges that are in the same edge equivalence class as a previously seen correct edge for the same hypothesis - Column 6: edges_skipped * number of correct edges (for relations) that cannot be assigned to an edge equivalence class because the assessment is missing a subject equivalence class * these edges are ignored for the puposes of computing correctness * these edges are counted for all other purposes (e.g. when reporting coherence, relevance, and coverage) - Column 7: edges_coherent * number of edges assessed as coherent - Column 8: KE_submitted * number of event or relation clusters in the truncated submitted hypotheses - Column 9: KE_coherent * number of event or relation clusters assessed as coherent - Column 10: KE_Frel * number of event or relation clusters assessed as FullyRelevant - Column 11: KE_Prel * number of event or relation clusters assessed as PartiallyRelevant - Column 12: hyotheses_submitted * number of hypotheses submitted - Column 13: theories_matched * number of prevailing theories (partially) matched by the submitted hypotheses - Column 14: Mean hypothesis-level correctness * hypothesis-level correctness = (edges_correct - edges_duplicate - edges_skipped) / (edges_submitted - edges_duplicate - edges_skipped) , restricted to edges in hypothesis - Column 15: Mean hypothesis-level edge_coherence * hypothesis-level edge_coherence = edges_coherent / edges_submitted , restricted to edges in hypothesis - Column 16: Mean hypothesis-level KE_coherence * hypothesis-level KE_coherence = KE_coherent / KE_submitted , restricted to KEs in hypothesis - Column 17: Mean hypothesis-level relevance_strict * hypothesis-level relevance_strict = KE_Frel / KE_submitted , restricted to KEs in hypothesis - Column 18: Mean hypothesis-level relevance_lenient * hypothesis-level relevance_lenient = (KE_Frel + KE_Prel) / KE_submitted , restricted to KEs in hypothesis - Column 19: coverage * Recall of edges in prevailing theories - Column 20: Run ID ********************** Task 3 scores for each topic Topic #2 #3 #4 #5 #6 #7 #8 #9 #10 #11 #12 #13 #14 #15 #16 #17 #18 #19 Run_ID E101 262 63 199 6 24 63 147 55 21 33 6 1 0.1450 0.2423 0.3703 0.1436 0.3636 0.0034 GAIA_1.GAIA_2.OPERA_TA3a_1 E101 839 481 358 14 42 479 350 272 115 157 14 1 0.5428 0.5709 0.7771 0.3286 0.7771 0.0238 LDC_2.LDC_2.OPERA_TA3b_2 E101 50 16 34 0 10 16 50 16 16 0 2 1 0.1500 0.3200 0.3200 0.3200 0.3200 0.0026 OPERA_TA1a_aditi_V5.OPERA_TA2_aditi_V5.OPERA_TA3a_1 E101 14 0 14 0 0 0 14 0 0 0 14 0 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 OPERA_TA1a_hans_V3.OPERA_TA2_hans_V5.OPERA_TA3a_2 E102 560 126 434 14 84 126 308 126 56 56 14 0 0.0606 0.2250 0.4091 0.1818 0.3636 0.0000 GAIA_1.GAIA_2.OPERA_TA3a_1 E102 848 522 326 8 62 522 350 301 127 174 14 0 0.5807 0.6156 0.8600 0.3629 0.8600 0.0000 LDC_2.LDC_2.OPERA_TA3b_2 E102 281 56 225 0 14 56 239 56 28 0 14 0 0.1573 0.1993 0.2344 0.1172 0.1172 0.0000 OPERA_TA1a_aditi_V5.OPERA_TA2_aditi_V5.OPERA_TA3a_1 E102 553 306 247 29 45 305 278 188 91 93 14 2 0.3816 0.4352 0.5371 0.2600 0.5257 0.0152 OPERA_TA1a_hans_V3.OPERA_TA2_hans_V5.OPERA_TA3a_2 E103 350 70 280 0 28 70 168 56 0 56 14 1 0.1304 0.2000 0.3333 0.0000 0.3333 0.0317 GAIA_1.GAIA_2.OPERA_TA3a_1 E103 858 590 268 22 39 590 350 308 160 148 14 1 0.6647 0.6881 0.8800 0.4571 0.8800 0.0635 LDC_2.LDC_2.OPERA_TA3b_2 E103 246 68 178 0 21 68 224 63 52 0 14 0 0.2382 0.3019 0.3022 0.2394 0.2394 0.0000 OPERA_TA1a_aditi_V5.OPERA_TA2_aditi_V5.OPERA_TA3a_1 E103 352 138 214 0 32 138 189 105 45 60 14 2 0.4005 0.4329 0.7317 0.1429 0.7317 0.0381 OPERA_TA1a_hans_V3.OPERA_TA2_hans_V5.OPERA_TA3a_2 ALL 1172 259 913 20 136 259 623 237 77 145 34 2 0.1042 0.2178 0.3711 0.1002 0.3512 0.0079 GAIA_1.GAIA_2.OPERA_TA3a_1 ALL 2545 1593 952 44 143 1591 1050 881 402 479 42 2 0.5961 0.6249 0.8390 0.3829 0.8390 0.0238 LDC_2.LDC_2.OPERA_TA3b_2 ALL 577 140 437 0 45 140 513 135 96 0 30 1 0.1946 0.2552 0.2717 0.1878 0.1878 0.0012 OPERA_TA1a_aditi_V5.OPERA_TA2_aditi_V5.OPERA_TA3a_1 ALL 919 444 475 29 77 443 481 293 136 153 42 4 0.2607 0.2894 0.4230 0.1343 0.4192 0.0127 OPERA_TA1a_hans_V3.OPERA_TA2_hans_V5.OPERA_TA3a_2