====================================================== TAC KBP 2017 EVENT NUGGET DETECTION EVALUATION RESULTS ====================================================== Team ID: zy Organization: University of Chinese Academy of Sciences (UCAS) ************************************************************* Run ID: zy1 Did the run access the live Web during the evaluation window: Yes Languages addressed: English Did the run return meaningful confidence values: No ==========English========== Mention Type Results: Type Prec Rec F1 #Gold #Sys conflictattack 0.34 0.32 0.33 410 386 conflictdemonstrate 0.45 0.45 0.45 209 213 contactbroadcast 0.41 0.18 0.25 639 282 contactcontact 0.16 0.21 0.18 238 314 contactcorrespondence 0.26 0.21 0.23 198 161 contactmeet 0.35 0.17 0.23 200 99 justicearrestjail 0.51 0.55 0.53 104 111 lifedie 0.59 0.52 0.55 218 190 lifeinjure 0.66 0.51 0.57 53 41 manufactureartifact 0.27 0.21 0.23 141 106 movementtransportartifact 0.35 0.15 0.21 242 102 movementtransportperson 0.33 0.33 0.33 447 440 personnelelect 0.47 0.47 0.47 152 153 personnelendposition 0.45 0.39 0.42 186 160 personnelstartposition 0.33 0.24 0.28 116 82 transactiontransaction 0.17 0.08 0.11 51 24 transactiontransfermoney 0.29 0.21 0.25 510 365 transactiontransferownership 0.31 0.27 0.29 261 231 Final Mention Detection Results: Micro Average Macro Average Attributes Prec Rec F1 Prec Rec F1 plain 58.23 46.05 51.43 59.91 47.24 52.83 mention_type 48.11 38.05 42.49 49.75 39.45 44.01 realis_status 43.51 34.41 38.42 45.48 35.75 40.03 mention_type+realis_status 35.61 28.16 31.45 37.51 29.60 33.09 ************************************************************* Run ID: zy2 Did the run access the live Web during the evaluation window: No Languages addressed: English Did the run return meaningful confidence values: No ==========English========== Mention Type Results: Type Prec Rec F1 #Gold #Sys conflictattack 0.41 0.31 0.35 410 311 conflictdemonstrate 0.54 0.51 0.52 209 198 contactbroadcast 0.56 0.17 0.26 639 193 contactcontact 0.19 0.21 0.20 238 266 contactcorrespondence 0.31 0.19 0.23 198 118 contactmeet 0.45 0.19 0.27 200 85 justicearrestjail 0.58 0.59 0.58 104 105 lifedie 0.65 0.54 0.59 218 180 lifeinjure 0.81 0.49 0.61 53 32 manufactureartifact 0.32 0.18 0.23 141 79 movementtransportartifact 0.43 0.17 0.24 242 96 movementtransportperson 0.37 0.32 0.35 447 390 personnelelect 0.56 0.51 0.53 152 137 personnelendposition 0.51 0.39 0.44 186 140 personnelstartposition 0.43 0.21 0.28 116 57 transactiontransaction 0.25 0.04 0.07 51 8 transactiontransfermoney 0.30 0.19 0.24 510 324 transactiontransferownership 0.33 0.28 0.30 261 217 Final Mention Detection Results: Micro Average Macro Average Attributes Prec Rec F1 Prec Rec F1 plain 64.29 43.14 51.64 65.33 44.21 52.73 mention_type 55.22 37.06 44.35 56.12 38.10 45.38 realis_status 49.28 33.07 39.58 50.45 33.96 40.59 mention_type+realis_status 41.87 28.10 33.63 42.85 28.87 34.50 ************************************************************* Run ID: zy3 Did the run access the live Web during the evaluation window: No Languages addressed: English Did the run return meaningful confidence values: No ==========English========== Mention Type Results: Type Prec Rec F1 #Gold #Sys conflictattack 0.33 0.24 0.28 410 307 conflictdemonstrate 0.48 0.42 0.45 209 181 contactbroadcast 0.43 0.16 0.23 639 241 contactcontact 0.16 0.18 0.17 238 255 contactcorrespondence 0.30 0.17 0.21 198 111 contactmeet 0.35 0.18 0.24 200 102 justicearrestjail 0.53 0.53 0.53 104 104 lifedie 0.55 0.51 0.53 218 202 lifeinjure 0.64 0.47 0.54 53 39 manufactureartifact 0.23 0.13 0.16 141 79 movementtransportartifact 0.27 0.12 0.17 242 109 movementtransportperson 0.32 0.31 0.31 447 433 personnelelect 0.48 0.40 0.44 152 126 personnelendposition 0.39 0.39 0.39 186 185 personnelstartposition 0.23 0.15 0.18 116 77 transactiontransaction 0.00 0.00 0.00 51 14 transactiontransfermoney 0.28 0.20 0.24 510 368 transactiontransferownership 0.23 0.25 0.24 261 274 Final Mention Detection Results: Micro Average Macro Average Attributes Prec Rec F1 Prec Rec F1 plain 55.42 40.63 46.89 57.39 41.68 48.29 mention_type 45.93 33.67 38.85 48.21 35.04 40.58 realis_status 41.33 30.30 34.96 43.67 31.27 36.44 mention_type+realis_status 34.16 25.04 28.90 36.56 26.10 30.46