====================================================== TAC KBP 2017 EVENT NUGGET DETECTION EVALUATION RESULTS ====================================================== Team ID: wip Organization: Peking University ************************************************************* Run ID: wip1 Did the run access the live Web during the evaluation window: No Languages addressed: English Did the run return meaningful confidence values: No ==========English========== Mention Type Results: Type Prec Rec F1 #Gold #Sys conflictattack 0.39 0.33 0.36 410 350 conflictdemonstrate 0.48 0.45 0.47 209 198 contactbroadcast 0.55 0.29 0.38 639 335 contactcontact 0.26 0.32 0.29 238 300 contactcorrespondence 0.28 0.05 0.08 198 34 contactmeet 0.45 0.21 0.29 200 95 justicearrestjail 0.59 0.62 0.60 104 111 lifedie 0.52 0.53 0.52 218 222 lifeinjure 0.90 0.49 0.63 53 29 manufactureartifact 0.37 0.23 0.28 141 87 movementtransportartifact 0.21 0.05 0.08 242 57 movementtransportperson 0.39 0.25 0.30 447 285 personnelelect 0.58 0.20 0.29 152 52 personnelendposition 0.57 0.34 0.43 186 110 personnelstartposition 0.61 0.13 0.21 116 24 transactiontransaction 0.06 0.02 0.03 51 17 transactiontransfermoney 0.36 0.29 0.32 510 411 transactiontransferownership 0.36 0.27 0.31 261 194 Final Mention Detection Results: Micro Average Macro Average Attributes Prec Rec F1 Prec Rec F1 plain 65.91 43.86 52.67 66.46 44.39 53.23 mention_type 57.84 38.49 46.22 58.70 39.44 47.18 realis_status 48.12 32.02 38.45 48.56 33.27 39.49 mention_type+realis_status 42.21 28.08 33.73 42.56 29.33 34.73 ************************************************************* Run ID: wip2 Did the run access the live Web during the evaluation window: No Languages addressed: English Did the run return meaningful confidence values: No ==========English========== Mention Type Results: Type Prec Rec F1 #Gold #Sys conflictattack 0.40 0.32 0.36 410 333 conflictdemonstrate 0.48 0.43 0.45 209 188 contactbroadcast 0.59 0.22 0.32 639 236 contactcontact 0.27 0.29 0.28 238 251 contactcorrespondence 0.26 0.07 0.11 198 53 contactmeet 0.44 0.21 0.28 200 96 justicearrestjail 0.56 0.62 0.59 104 114 lifedie 0.53 0.52 0.53 218 212 lifeinjure 0.90 0.49 0.63 53 29 manufactureartifact 0.35 0.21 0.27 141 85 movementtransportartifact 0.35 0.14 0.20 242 97 movementtransportperson 0.41 0.26 0.32 447 283 personnelelect 0.62 0.23 0.34 152 56 personnelendposition 0.53 0.31 0.39 186 110 personnelstartposition 0.54 0.15 0.24 116 33 transactiontransaction 0.17 0.02 0.04 51 6 transactiontransfermoney 0.36 0.28 0.32 510 402 transactiontransferownership 0.37 0.27 0.31 261 191 Final Mention Detection Results: Micro Average Macro Average Attributes Prec Rec F1 Prec Rec F1 plain 68.58 43.50 53.24 69.07 43.90 53.68 mention_type 60.98 38.68 47.33 61.65 39.26 47.97 realis_status 48.76 30.93 37.85 49.46 32.04 38.89 mention_type+realis_status 43.01 27.28 33.39 43.66 28.28 34.33 ************************************************************* Run ID: wip3 Did the run access the live Web during the evaluation window: No Languages addressed: English Did the run return meaningful confidence values: No ==========English========== Mention Type Results: Type Prec Rec F1 #Gold #Sys conflictattack 0.40 0.32 0.36 410 333 conflictdemonstrate 0.48 0.43 0.45 209 188 contactbroadcast 0.59 0.22 0.32 639 236 contactcontact 0.27 0.29 0.28 238 251 contactcorrespondence 0.26 0.07 0.11 198 53 contactmeet 0.44 0.21 0.28 200 96 justicearrestjail 0.56 0.62 0.59 104 114 lifedie 0.53 0.52 0.53 218 212 lifeinjure 0.90 0.49 0.63 53 29 manufactureartifact 0.35 0.21 0.27 141 85 movementtransportartifact 0.35 0.14 0.20 242 97 movementtransportperson 0.41 0.26 0.32 447 283 personnelelect 0.62 0.23 0.34 152 56 personnelendposition 0.53 0.31 0.39 186 110 personnelstartposition 0.54 0.15 0.24 116 33 transactiontransaction 0.17 0.02 0.04 51 6 transactiontransfermoney 0.36 0.28 0.32 510 402 transactiontransferownership 0.37 0.27 0.31 261 191 Final Mention Detection Results: Micro Average Macro Average Attributes Prec Rec F1 Prec Rec F1 plain 68.58 43.50 53.24 69.07 43.90 53.68 mention_type 60.98 38.68 47.33 61.65 39.26 47.97 realis_status 48.76 30.93 37.85 49.46 32.04 38.89 mention_type+realis_status 43.01 27.28 33.39 43.66 28.28 34.33