====================================================================== TAC KBP 2017 EVENT NUGGET DETECTION AND COREFERENCE EVALUATION RESULTS ====================================================================== Team ID: BUPT_PRIS Organization: Pattern Recognition & Intelligent System Laboratory ************************************************************* Run ID: BUPT_PRIS1 Did the run access the live Web during the evaluation window: No Languages addressed: English Did the run return meaningful confidence values: No ==========English========== Mention Type Results: Type Prec Rec F1 #Gold #Sys conflictattack 0.28 0.28 0.28 410 409 conflictdemonstrate 0.48 0.38 0.42 209 168 contactbroadcast 0.51 0.19 0.28 639 239 contactcontact 0.21 0.11 0.14 238 118 contactcorrespondence 0.18 0.05 0.08 198 54 contactmeet 0.45 0.17 0.25 200 77 justicearrestjail 0.54 0.49 0.52 104 94 lifedie 0.54 0.43 0.48 218 172 lifeinjure 0.79 0.49 0.60 53 33 manufactureartifact 0.30 0.11 0.16 141 50 movementtransportartifact 0.27 0.07 0.12 242 66 movementtransportperson 0.40 0.08 0.14 447 92 personnelelect 0.54 0.39 0.45 152 110 personnelendposition 0.55 0.26 0.35 186 89 personnelstartposition 0.26 0.09 0.13 116 39 transactiontransaction 0.08 0.02 0.03 51 12 transactiontransfermoney 0.31 0.13 0.18 510 213 transactiontransferownership 0.41 0.11 0.18 261 73 Final Mention Detection Results: Micro Average Macro Average Attributes Prec Rec F1 Prec Rec F1 plain 67.95 32.74 44.19 66.76 34.16 45.19 mention_type 58.92 28.39 38.31 57.69 29.61 39.14 realis_status 46.36 22.34 30.15 44.90 23.61 30.95 mention_type+realis_status 39.92 19.24 25.96 38.48 20.41 26.67 Final Mention Coreference Results: Metric : bcub Score 28.66 Metric : ceafe Score 28.64 Metric : ceafm Score 30.17 * Metric : muc Score 19.30 Metric : blanc Score 13.56 Overall Average CoNLL score 22.54 * Score not included for final CoNLL score. ************************************************************* Run ID: BUPT_PRIS2 Did the run access the live Web during the evaluation window: No Languages addressed: English Did the run return meaningful confidence values: No ==========English========== Mention Type Results: Type Prec Rec F1 #Gold #Sys conflictattack 0.30 0.23 0.26 410 317 conflictdemonstrate 0.49 0.37 0.42 209 156 contactbroadcast 0.55 0.16 0.25 639 186 contactcontact 0.34 0.05 0.09 238 35 contactcorrespondence 0.21 0.04 0.07 198 39 contactmeet 0.42 0.18 0.26 200 88 justicearrestjail 0.56 0.46 0.51 104 85 lifedie 0.56 0.41 0.47 218 158 lifeinjure 0.87 0.51 0.64 53 31 manufactureartifact 0.31 0.07 0.12 141 32 movementtransportartifact 0.33 0.06 0.10 242 43 movementtransportperson 0.44 0.05 0.09 447 49 personnelelect 0.54 0.33 0.41 152 93 personnelendposition 0.58 0.27 0.37 186 86 personnelstartposition 0.30 0.07 0.12 116 28 transactiontransaction 0.00 0.00 0.00 51 7 transactiontransfermoney 0.36 0.08 0.13 510 118 transactiontransferownership 0.43 0.09 0.15 261 56 Final Mention Detection Results: Micro Average Macro Average Attributes Prec Rec F1 Prec Rec F1 plain 72.91 26.78 39.17 69.86 27.84 39.82 mention_type 65.44 24.04 35.16 62.39 24.96 35.65 realis_status 49.76 18.28 26.73 47.17 19.39 27.49 mention_type+realis_status 44.48 16.34 23.90 41.98 17.31 24.51 Final Mention Coreference Results: Metric : bcub Score 26.19 Metric : ceafe Score 26.19 Metric : ceafm Score 27.71 * Metric : muc Score 18.00 Metric : blanc Score 11.97 Overall Average CoNLL score 20.59 * Score not included for final CoNLL score. ************************************************************* Run ID: BUPT_PRIS3 Did the run access the live Web during the evaluation window: No Languages addressed: English Did the run return meaningful confidence values: No ==========English========== Mention Type Results: Type Prec Rec F1 #Gold #Sys conflictattack 0.19 0.27 0.23 410 577 conflictdemonstrate 0.49 0.41 0.45 209 175 contactbroadcast 0.54 0.23 0.32 639 268 contactcontact 0.21 0.07 0.10 238 78 contactcorrespondence 0.00 0.00 0.00 198 0 contactmeet 0.42 0.18 0.25 200 85 justicearrestjail 0.56 0.45 0.50 104 84 lifedie 0.55 0.47 0.51 218 188 lifeinjure 0.93 0.47 0.62 53 27 manufactureartifact 0.00 0.00 0.00 141 0 movementtransportartifact 0.00 0.00 0.00 242 0 movementtransportperson 0.32 0.07 0.12 447 98 personnelelect 0.53 0.36 0.43 152 102 personnelendposition 0.52 0.22 0.31 186 80 personnelstartposition 0.00 0.00 0.00 116 1 transactiontransaction 0.00 0.00 0.00 51 0 transactiontransfermoney 0.31 0.13 0.18 510 213 transactiontransferownership 0.55 0.02 0.04 261 11 Final Mention Detection Results: Micro Average Macro Average Attributes Prec Rec F1 Prec Rec F1 plain 65.38 29.69 40.84 63.78 31.08 41.79 mention_type 54.24 24.64 33.88 52.82 26.00 34.84 realis_status 46.15 20.96 28.83 43.95 22.37 29.65 mention_type+realis_status 38.66 17.56 24.15 36.85 18.84 24.93 Final Mention Coreference Results: Metric : bcub Score 25.53 Metric : ceafe Score 24.90 Metric : ceafm Score 26.40 * Metric : muc Score 18.45 Metric : blanc Score 12.47 Overall Average CoNLL score 20.34 * Score not included for final CoNLL score.