========================================= TAC 2019 SM-KBP TASK 3 EVALUATION RESULTS ========================================= Team ID: Hyperthesis Organization: Pacific Northwest National Laboratory; American University; Georgetown University ********************** Description of Columns in Task 3 scores: - Column 1: Topic ID - Column 2: edges_submitted * number of edges in the truncated submitted hypotheses - Column 3: edges_correct * number of edges that have a correct justification (i.e., predicate justification is correct and linkable to object justification) - Column 4: edges_wrong * number of edges that have no correct justification - Column 5: edges_duplicate * number of correct edges that are in the same edge equivalence class as a previously seen correct edge for the same hypothesis - Column 6: edges_skipped * number of correct edges (for relations) that cannot be assigned to an edge equivalence class because the assessment is missing a subject equivalence class * these edges are ignored for the puposes of computing correctness * these edges are counted for all other purposes (e.g. when reporting coherence, relevance, and coverage) - Column 7: edges_coherent * number of edges assessed as coherent - Column 8: KE_submitted * number of event or relation clusters in the truncated submitted hypotheses - Column 9: KE_coherent * number of event or relation clusters assessed as coherent - Column 10: KE_Frel * number of event or relation clusters assessed as FullyRelevant - Column 11: KE_Prel * number of event or relation clusters assessed as PartiallyRelevant - Column 12: hyotheses_submitted * number of hypotheses submitted - Column 13: theories_matched * number of prevailing theories (partially) matched by the submitted hypotheses - Column 14: Mean hypothesis-level correctness * hypothesis-level correctness = (edges_correct - edges_duplicate - edges_skipped) / (edges_submitted - edges_duplicate - edges_skipped) , restricted to edges in hypothesis - Column 15: Mean hypothesis-level edge_coherence * hypothesis-level edge_coherence = edges_coherent / edges_submitted , restricted to edges in hypothesis - Column 16: Mean hypothesis-level KE_coherence * hypothesis-level KE_coherence = KE_coherent / KE_submitted , restricted to KEs in hypothesis - Column 17: Mean hypothesis-level relevance_strict * hypothesis-level relevance_strict = KE_Frel / KE_submitted , restricted to KEs in hypothesis - Column 18: Mean hypothesis-level relevance_lenient * hypothesis-level relevance_lenient = (KE_Frel + KE_Prel) / KE_submitted , restricted to KEs in hypothesis - Column 19: coverage * Recall of edges in prevailing theories - Column 20: Run ID ********************** Task 3 scores for each topic Topic #2 #3 #4 #5 #6 #7 #8 #9 #10 #11 #12 #13 #14 #15 #16 #17 #18 #19 Run_ID E101 1 1 0 0 0 1 1 1 1 0 1 0 1.0000 1.0000 1.0000 1.0000 1.0000 0.0000 BBN_1-Michigan_1-OPERA_TA1a_hans_V3.PNNL_4.PNNL_sheafbox_10 E101 5 2 3 0 0 2 2 1 1 0 1 0 0.4000 0.4000 0.5000 0.5000 0.5000 0.0000 GAIA_1-OPERA_3.Colorado_1.PNNL_sheafbox_10 E101 3 1 2 0 0 1 1 1 0 1 1 0 0.3333 0.3333 1.0000 0.0000 1.0000 0.0000 GAIA_2.Colorado_1.PNNL_sheafbox_10 E101 5 4 1 0 0 4 3 2 2 0 3 1 0.6667 0.6667 0.6667 0.6667 0.6667 0.0026 GAIA_2.GAIA_2.PNNL_sheafbox_10 E101 283 249 34 5 0 249 138 138 104 34 14 0 0.8757 0.8783 1.0000 0.7483 1.0000 0.0000 LDC_2.LDC_2.PNNL_HyperQA_2 E101 128 119 9 11 0 119 32 32 24 8 3 0 0.9240 0.9307 1.0000 0.7515 1.0000 0.0000 LDC_2.LDC_2.PNNL_sheafbox_10 E101 1 0 1 0 0 0 1 0 0 0 1 0 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 OPERA_TA1a_hans_V2.BBN_TA2_v1.PNNL_sheafbox_10 E101 1 1 0 0 0 1 1 1 1 0 1 0 1.0000 1.0000 1.0000 1.0000 1.0000 0.0000 OPERA_TA1a_hans_V2.GAIA_1.PNNL_sheafbox_10 E101 1 1 0 0 0 1 1 1 1 0 1 0 1.0000 1.0000 1.0000 1.0000 1.0000 0.0000 OPERA_TA1a_hans_V3.OPERA_TA2_hans_V5.PNNL_HyperQA_2 E101 2 1 1 0 0 1 2 1 1 0 1 0 0.5000 0.5000 0.5000 0.5000 0.5000 0.0000 OPERA_TA1a_hans_V3.OPERA_TA2_hans_V5.PNNL_sheafbox_10 E102 8 3 5 0 0 3 4 3 0 3 2 0 0.3750 0.3750 0.7500 0.0000 0.7500 0.0000 GAIA_1.GAIA_2.PNNL_sheafbox_10 E102 2 0 2 0 0 0 1 0 0 0 1 0 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 GAIA_2.GAIA_2.PNNL_sheafbox_10 E102 73 64 9 0 4 62 35 34 26 9 14 2 0.8879 0.8622 0.9643 0.7857 1.0000 0.0104 LDC_2.LDC_2.PNNL_HyperQA_2 E102 158 141 17 32 0 141 64 64 47 17 14 1 0.8650 0.8923 1.0000 0.7333 1.0000 0.0154 LDC_2.LDC_2.PNNL_sheafbox_10 E103 4 4 0 0 0 4 1 1 1 0 1 0 1.0000 1.0000 1.0000 1.0000 1.0000 0.0000 BBN_1-Michigan_1-OPERA_TA1a_hans_V3.PNNL_4.PNNL_sheafbox_10 E103 4 1 3 0 0 1 1 1 0 1 1 0 0.2500 0.2500 1.0000 0.0000 1.0000 0.0000 BBN_1.Colorado_1.PNNL_sheafbox_10 E103 1 1 0 0 0 1 1 1 1 0 1 0 1.0000 1.0000 1.0000 1.0000 1.0000 0.0000 GAIA_1.GAIA_2.PNNL_sheafbox_10 E103 4 2 2 0 0 2 1 1 0 1 1 1 0.5000 0.5000 1.0000 0.0000 1.0000 0.0159 GAIA_2.Colorado_1.PNNL_sheafbox_10 E103 14 12 2 0 0 12 6 5 5 0 2 0 0.8571 0.8571 0.8333 0.8333 0.8333 0.0000 LDC_2.LDC_2.PNNL_HyperQA_2 E103 108 99 9 0 0 99 36 36 27 9 12 0 0.9167 0.9167 1.0000 0.7500 1.0000 0.0000 LDC_2.LDC_2.PNNL_sheafbox_10 E103 5 1 4 0 0 1 2 1 0 1 1 1 0.2000 0.2000 0.5000 0.0000 0.5000 0.0111 OPERA_4.Colorado_1.PNNL_sheafbox_10 E103 2 2 0 0 0 2 1 1 1 0 1 1 1.0000 1.0000 1.0000 1.0000 1.0000 0.0111 OPERA_TA1a_hans_V2.BBN_TA2_v1.PNNL_sheafbox_10 E103 2 2 0 0 0 2 1 1 1 0 1 1 1.0000 1.0000 1.0000 1.0000 1.0000 0.0111 OPERA_TA1a_hans_V3.OPERA_TA2_hans_V5.PNNL_sheafbox_10 ALL 5 5 0 0 0 5 2 2 2 0 2 0 1.0000 1.0000 1.0000 1.0000 1.0000 0.0000 BBN_1-Michigan_1-OPERA_TA1a_hans_V3.PNNL_4.PNNL_sheafbox_10 ALL 4 1 3 0 0 1 1 1 0 1 1 0 0.2500 0.2500 1.0000 0.0000 1.0000 0.0000 BBN_1.Colorado_1.PNNL_sheafbox_10 ALL 5 2 3 0 0 2 2 1 1 0 1 0 0.4000 0.4000 0.5000 0.5000 0.5000 0.0000 GAIA_1-OPERA_3.Colorado_1.PNNL_sheafbox_10 ALL 9 4 5 0 0 4 5 4 1 3 3 0 0.5833 0.5833 0.8333 0.3333 0.8333 0.0000 GAIA_1.GAIA_2.PNNL_sheafbox_10 ALL 7 3 4 0 0 3 2 2 0 2 2 1 0.4167 0.4167 1.0000 0.0000 1.0000 0.0032 GAIA_2.Colorado_1.PNNL_sheafbox_10 ALL 7 4 3 0 0 4 4 2 2 0 4 1 0.5000 0.5000 0.5000 0.5000 0.5000 0.0012 GAIA_2.GAIA_2.PNNL_sheafbox_10 ALL 370 325 45 5 4 323 179 177 135 43 30 2 0.8802 0.8694 0.9722 0.7714 0.9889 0.0035 LDC_2.LDC_2.PNNL_HyperQA_2 ALL 394 359 35 43 0 359 132 132 98 34 29 1 0.8925 0.9063 1.0000 0.7421 1.0000 0.0051 LDC_2.LDC_2.PNNL_sheafbox_10 ALL 5 1 4 0 0 1 2 1 0 1 1 1 0.2000 0.2000 0.5000 0.0000 0.5000 0.0022 OPERA_4.Colorado_1.PNNL_sheafbox_10 ALL 3 2 1 0 0 2 2 1 1 0 2 1 0.5000 0.5000 0.5000 0.5000 0.5000 0.0022 OPERA_TA1a_hans_V2.BBN_TA2_v1.PNNL_sheafbox_10 ALL 1 1 0 0 0 1 1 1 1 0 1 0 1.0000 1.0000 1.0000 1.0000 1.0000 0.0000 OPERA_TA1a_hans_V2.GAIA_1.PNNL_sheafbox_10 ALL 1 1 0 0 0 1 1 1 1 0 1 0 1.0000 1.0000 1.0000 1.0000 1.0000 0.0000 OPERA_TA1a_hans_V3.OPERA_TA2_hans_V5.PNNL_HyperQA_2 ALL 4 3 1 0 0 3 3 2 2 0 2 1 0.7500 0.7500 0.7500 0.7500 0.7500 0.0022 OPERA_TA1a_hans_V3.OPERA_TA2_hans_V5.PNNL_sheafbox_10