====================================================================== TAC KBP 2017 EVENT NUGGET DETECTION AND COREFERENCE EVALUATION RESULTS ====================================================================== Team ID: TAMUNLP Organization: Texas A&M University, College Station ************************************************************* Run ID: TAMUNLP1 Did the run access the live Web during the evaluation window: No Languages addressed: English Did the run return meaningful confidence values: No ==========English========== Mention Type Results: Type Prec Rec F1 #Gold #Sys conflictattack 0.33 0.36 0.34 410 451 conflictdemonstrate 0.48 0.55 0.51 209 239 contactbroadcast 0.41 0.27 0.33 639 423 contactcontact 0.16 0.44 0.24 238 655 contactcorrespondence 0.23 0.13 0.16 198 108 contactmeet 0.34 0.25 0.29 200 147 justicearrestjail 0.45 0.58 0.51 104 132 lifedie 0.50 0.51 0.50 218 224 lifeinjure 0.74 0.53 0.62 53 38 manufactureartifact 0.31 0.16 0.21 141 71 movementtransportartifact 0.31 0.12 0.17 242 94 movementtransportperson 0.30 0.34 0.32 447 516 personnelelect 0.52 0.41 0.46 152 122 personnelendposition 0.42 0.43 0.43 186 192 personnelstartposition 0.29 0.19 0.23 116 76 transactiontransaction 0.22 0.22 0.22 51 50 transactiontransfermoney 0.25 0.18 0.21 510 372 transactiontransferownership 0.29 0.31 0.30 261 286 Final Mention Detection Results: Micro Average Macro Average Attributes Prec Rec F1 Prec Rec F1 plain 58.95 56.53 57.72 59.33 58.50 58.91 mention_type 45.21 43.36 44.27 45.92 45.63 45.77 realis_status 43.38 41.60 42.47 43.63 43.27 43.45 mention_type+realis_status 32.64 31.31 31.96 33.00 32.97 32.98 Final Mention Coreference Results: Metric : bcub Score 35.00 Metric : ceafe Score 34.95 Metric : ceafm Score 34.95 * Metric : muc Score 18.66 Metric : blanc Score 16.54 Overall Average CoNLL score 26.29 * Score not included for final CoNLL score. ************************************************************* Run ID: TAMUNLP2 Did the run access the live Web during the evaluation window: No Languages addressed: English Did the run return meaningful confidence values: No ==========English========== Mention Type Results: Type Prec Rec F1 #Gold #Sys conflictattack 0.35 0.34 0.35 410 407 conflictdemonstrate 0.49 0.54 0.51 209 229 contactbroadcast 0.46 0.23 0.31 639 318 contactcontact 0.18 0.39 0.25 238 514 contactcorrespondence 0.28 0.11 0.15 198 75 contactmeet 0.40 0.25 0.31 200 125 justicearrestjail 0.48 0.56 0.52 104 120 lifedie 0.51 0.50 0.50 218 215 lifeinjure 0.77 0.51 0.61 53 35 manufactureartifact 0.27 0.10 0.15 141 51 movementtransportartifact 0.31 0.11 0.16 242 83 movementtransportperson 0.33 0.28 0.31 447 379 personnelelect 0.55 0.39 0.46 152 109 personnelendposition 0.49 0.42 0.45 186 159 personnelstartposition 0.37 0.16 0.22 116 51 transactiontransaction 0.23 0.20 0.21 51 43 transactiontransfermoney 0.28 0.16 0.20 510 294 transactiontransferownership 0.33 0.30 0.31 261 230 Final Mention Detection Results: Micro Average Macro Average Attributes Prec Rec F1 Prec Rec F1 plain 64.22 50.45 56.50 64.43 52.58 57.91 mention_type 50.32 39.53 44.28 50.81 41.82 45.88 realis_status 47.30 37.16 41.62 46.86 38.83 42.47 mention_type+realis_status 36.30 28.52 31.94 35.99 30.14 32.81 Final Mention Coreference Results: Metric : bcub Score 34.34 Metric : ceafe Score 33.63 Metric : ceafm Score 34.74 * Metric : muc Score 22.90 Metric : blanc Score 17.94 Overall Average CoNLL score 27.20 * Score not included for final CoNLL score. ************************************************************* Run ID: TAMUNLP3 Did the run access the live Web during the evaluation window: No Languages addressed: English Did the run return meaningful confidence values: No ==========English========== Mention Type Results: Type Prec Rec F1 #Gold #Sys conflictattack 0.33 0.38 0.36 410 475 conflictdemonstrate 0.52 0.51 0.52 209 205 contactbroadcast 0.44 0.29 0.35 639 414 contactcontact 0.20 0.46 0.28 238 559 contactcorrespondence 0.25 0.13 0.17 198 103 contactmeet 0.34 0.24 0.28 200 143 justicearrestjail 0.47 0.57 0.52 104 125 lifedie 0.47 0.50 0.48 218 231 lifeinjure 0.80 0.53 0.64 53 35 manufactureartifact 0.30 0.22 0.26 141 102 movementtransportartifact 0.33 0.18 0.23 242 131 movementtransportperson 0.25 0.30 0.27 447 534 personnelelect 0.49 0.45 0.47 152 142 personnelendposition 0.45 0.44 0.44 186 182 personnelstartposition 0.32 0.20 0.24 116 73 transactiontransaction 0.14 0.12 0.13 51 42 transactiontransfermoney 0.24 0.18 0.21 510 383 transactiontransferownership 0.28 0.28 0.28 261 267 Final Mention Detection Results: Micro Average Macro Average Attributes Prec Rec F1 Prec Rec F1 plain 57.44 54.44 55.90 58.52 56.50 57.49 mention_type 45.88 43.48 44.65 46.97 45.55 46.25 realis_status 42.07 39.87 40.94 43.06 41.62 42.33 mention_type+realis_status 33.35 31.60 32.45 34.31 33.24 33.77 Final Mention Coreference Results: Metric : bcub Score 35.03 Metric : ceafe Score 34.67 Metric : ceafm Score 34.85 * Metric : muc Score 18.68 Metric : blanc Score 16.47 Overall Average CoNLL score 26.21 * Score not included for final CoNLL score.