====================================================================== TAC KBP 2017 EVENT NUGGET DETECTION AND COREFERENCE EVALUATION RESULTS ====================================================================== Team ID: UTD Organization: University of Texas at Dallas ************************************************************* Run ID: UTD1 Did the run access the live Web during the evaluation window: No Languages addressed: Chinese, English Did the run return meaningful confidence values: No ==========English========== Mention Type Results: Type Prec Rec F1 #Gold #Sys conflictattack 0.39 0.43 0.41 410 447 conflictdemonstrate 0.57 0.55 0.56 209 201 contactbroadcast 0.42 0.25 0.31 639 371 contactcontact 0.16 0.26 0.20 238 389 contactcorrespondence 0.23 0.15 0.18 198 129 contactmeet 0.27 0.24 0.26 200 182 justicearrestjail 0.50 0.53 0.51 104 111 lifedie 0.65 0.61 0.63 218 205 lifeinjure 0.43 0.51 0.47 53 62 manufactureartifact 0.28 0.21 0.24 141 103 movementtransportartifact 0.29 0.23 0.26 242 192 movementtransportperson 0.25 0.28 0.27 447 497 personnelelect 0.63 0.51 0.56 152 123 personnelendposition 0.52 0.41 0.46 186 149 personnelstartposition 0.26 0.22 0.24 116 97 transactiontransaction 0.07 0.06 0.07 51 41 transactiontransfermoney 0.31 0.28 0.29 510 462 transactiontransferownership 0.27 0.34 0.30 261 325 Final Mention Detection Results: Micro Average Macro Average Attributes Prec Rec F1 Prec Rec F1 plain 61.74 57.66 59.63 61.59 58.84 60.18 mention_type 52.16 48.71 50.37 51.64 49.52 50.56 realis_status 42.36 39.56 40.91 42.49 41.70 42.09 mention_type+realis_status 35.01 32.70 33.81 34.82 34.39 34.61 Final Mention Coreference Results: Metric : bcub Score 39.57 Metric : ceafe Score 35.20 Metric : ceafm Score 38.60 * Metric : muc Score 33.96 Metric : blanc Score 25.91 Overall Average CoNLL score 33.66 * Score not included for final CoNLL score. ==========Chinese========== Mention Type Results: Type Prec Rec F1 #Gold #Sys conflictattack 0.38 0.36 0.37 820 776 conflictdemonstrate 0.63 0.60 0.62 101 96 contactbroadcast 0.36 0.26 0.30 775 572 contactcontact 0.01 0.03 0.02 62 158 contactcorrespondence 0.14 0.12 0.13 25 21 contactmeet 0.15 0.24 0.18 126 205 justicearrestjail 0.34 0.35 0.34 109 111 lifedie 0.50 0.63 0.56 298 372 lifeinjure 0.80 0.78 0.79 93 91 manufactureartifact 0.22 0.15 0.18 131 93 movementtransportartifact 0.06 0.05 0.06 79 64 movementtransportperson 0.17 0.28 0.21 360 570 personnelelect 0.37 0.44 0.40 176 210 personnelendposition 0.48 0.36 0.41 108 81 personnelstartposition 0.33 0.32 0.32 78 74 transactiontransaction 0.04 0.01 0.02 107 26 transactiontransfermoney 0.16 0.14 0.15 264 227 transactiontransferownership 0.15 0.14 0.14 172 162 Final Mention Detection Results: Micro Average Macro Average Attributes Prec Rec F1 Prec Rec F1 plain 52.69 53.02 52.85 52.14 53.32 52.72 mention_type 46.61 46.91 46.76 45.77 46.71 46.24 realis_status 35.08 35.30 35.19 33.90 34.83 34.36 mention_type+realis_status 31.07 31.27 31.17 29.87 30.66 30.26 Final Mention Coreference Results: Metric : bcub Score 34.18 Metric : ceafe Score 32.22 Metric : ceafm Score 33.90 * Metric : muc Score 27.07 Metric : blanc Score 18.57 Overall Average CoNLL score 28.01 * Score not included for final CoNLL score. ************************************************************* Run ID: UTD2 Did the run access the live Web during the evaluation window: No Languages addressed: Chinese, English Did the run return meaningful confidence values: No ==========English========== Mention Type Results: Type Prec Rec F1 #Gold #Sys conflictattack 0.39 0.43 0.41 410 447 conflictdemonstrate 0.57 0.55 0.56 209 201 contactbroadcast 0.42 0.25 0.31 639 371 contactcontact 0.16 0.26 0.20 238 389 contactcorrespondence 0.23 0.15 0.18 198 129 contactmeet 0.27 0.24 0.26 200 182 justicearrestjail 0.50 0.53 0.51 104 111 lifedie 0.65 0.61 0.63 218 205 lifeinjure 0.43 0.51 0.47 53 62 manufactureartifact 0.28 0.21 0.24 141 103 movementtransportartifact 0.29 0.23 0.26 242 192 movementtransportperson 0.25 0.28 0.27 447 497 personnelelect 0.63 0.51 0.56 152 123 personnelendposition 0.52 0.41 0.46 186 149 personnelstartposition 0.26 0.22 0.24 116 97 transactiontransaction 0.07 0.06 0.07 51 41 transactiontransfermoney 0.31 0.28 0.29 510 462 transactiontransferownership 0.27 0.34 0.30 261 325 Final Mention Detection Results: Micro Average Macro Average Attributes Prec Rec F1 Prec Rec F1 plain 61.74 57.66 59.63 61.59 58.84 60.18 mention_type 52.16 48.71 50.37 51.64 49.52 50.56 realis_status 42.36 39.56 40.91 42.49 41.70 42.09 mention_type+realis_status 35.01 32.70 33.81 34.82 34.39 34.61 Final Mention Coreference Results: Metric : bcub Score 39.88 Metric : ceafe Score 35.73 Metric : ceafm Score 38.90 * Metric : muc Score 33.79 Metric : blanc Score 26.06 Overall Average CoNLL score 33.87 * Score not included for final CoNLL score. ==========Chinese========== Mention Type Results: Type Prec Rec F1 #Gold #Sys conflictattack 0.38 0.36 0.37 820 776 conflictdemonstrate 0.63 0.60 0.62 101 96 contactbroadcast 0.36 0.26 0.30 775 572 contactcontact 0.01 0.03 0.02 62 158 contactcorrespondence 0.14 0.12 0.13 25 21 contactmeet 0.15 0.24 0.18 126 205 justicearrestjail 0.34 0.35 0.34 109 111 lifedie 0.50 0.63 0.56 298 372 lifeinjure 0.80 0.78 0.79 93 91 manufactureartifact 0.22 0.15 0.18 131 93 movementtransportartifact 0.06 0.05 0.06 79 64 movementtransportperson 0.17 0.28 0.21 360 570 personnelelect 0.37 0.44 0.40 176 210 personnelendposition 0.48 0.36 0.41 108 81 personnelstartposition 0.33 0.32 0.32 78 74 transactiontransaction 0.04 0.01 0.02 107 26 transactiontransfermoney 0.16 0.14 0.15 264 227 transactiontransferownership 0.15 0.14 0.14 172 162 Final Mention Detection Results: Micro Average Macro Average Attributes Prec Rec F1 Prec Rec F1 plain 52.69 53.02 52.85 52.14 53.32 52.72 mention_type 46.61 46.91 46.76 45.77 46.71 46.24 realis_status 35.08 35.30 35.19 33.90 34.83 34.36 mention_type+realis_status 31.07 31.27 31.17 29.87 30.66 30.26 Final Mention Coreference Results: Metric : bcub Score 34.28 Metric : ceafe Score 32.64 Metric : ceafm Score 34.03 * Metric : muc Score 26.40 Metric : blanc Score 18.40 Overall Average CoNLL score 27.93 * Score not included for final CoNLL score.