====================================================== TAC KBP 2017 EVENT NUGGET DETECTION EVALUATION RESULTS ====================================================== Team ID: dsln_nlptt Organization: NEC Corporation and Tokyo Institute of Technology ************************************************************* Run ID: dsln_nlptt1 Did the run access the live Web during the evaluation window: No Languages addressed: English Did the run return meaningful confidence values: No ==========English========== Mention Type Results: Type Prec Rec F1 #Gold #Sys conflictattack 0.39 0.39 0.39 410 411 conflictdemonstrate 0.55 0.54 0.54 209 204 contactbroadcast 0.58 0.21 0.31 639 236 contactcontact 0.24 0.24 0.24 238 232 contactcorrespondence 0.42 0.16 0.23 198 73 contactmeet 0.46 0.23 0.30 200 97 justicearrestjail 0.58 0.70 0.64 104 125 lifedie 0.63 0.62 0.62 218 215 lifeinjure 0.90 0.49 0.63 53 29 manufactureartifact 0.39 0.38 0.38 141 140 movementtransportartifact 0.39 0.24 0.30 242 149 movementtransportperson 0.39 0.29 0.33 447 332 personnelelect 0.54 0.50 0.52 152 141 personnelendposition 0.61 0.40 0.48 186 121 personnelstartposition 0.34 0.24 0.28 116 81 transactiontransaction 0.23 0.12 0.16 51 26 transactiontransfermoney 0.31 0.23 0.26 510 373 transactiontransferownership 0.33 0.33 0.33 261 260 Final Mention Detection Results: Micro Average Macro Average Attributes Prec Rec F1 Prec Rec F1 plain 65.89 48.87 56.12 66.23 49.88 56.90 mention_type 57.02 42.29 48.56 57.87 43.69 49.79 realis_status 49.86 36.98 42.47 51.12 38.39 43.85 mention_type+realis_status 43.22 32.05 36.81 44.63 33.55 38.30 ************************************************************* Run ID: dsln_nlptt2 Did the run access the live Web during the evaluation window: No Languages addressed: English Did the run return meaningful confidence values: No ==========English========== Mention Type Results: Type Prec Rec F1 #Gold #Sys conflictattack 0.43 0.37 0.40 410 349 conflictdemonstrate 0.51 0.45 0.48 209 186 contactbroadcast 0.54 0.23 0.32 639 267 contactcontact 0.24 0.21 0.23 238 214 contactcorrespondence 0.30 0.24 0.27 198 154 contactmeet 0.36 0.23 0.28 200 131 justicearrestjail 0.53 0.66 0.59 104 130 lifedie 0.66 0.57 0.61 218 188 lifeinjure 0.61 0.43 0.51 53 38 manufactureartifact 0.27 0.29 0.28 141 149 movementtransportartifact 0.34 0.29 0.31 242 202 movementtransportperson 0.37 0.30 0.33 447 367 personnelelect 0.50 0.50 0.50 152 151 personnelendposition 0.52 0.38 0.44 186 137 personnelstartposition 0.26 0.31 0.28 116 141 transactiontransaction 0.24 0.14 0.18 51 29 transactiontransfermoney 0.30 0.22 0.25 510 376 transactiontransferownership 0.35 0.34 0.34 261 256 Final Mention Detection Results: Micro Average Macro Average Attributes Prec Rec F1 Prec Rec F1 plain 61.02 48.33 53.94 61.70 49.45 54.90 mention_type 52.70 41.74 46.59 53.71 43.10 47.82 realis_status 46.72 37.00 41.29 48.19 38.32 42.69 mention_type+realis_status 40.06 31.73 35.41 41.62 33.06 36.85 ************************************************************* Run ID: dsln_nlptt3 Did the run access the live Web during the evaluation window: No Languages addressed: English Did the run return meaningful confidence values: Yes ==========English========== Mention Type Results: Type Prec Rec F1 #Gold #Sys conflictattack 0.39 0.39 0.39 410 411 conflictdemonstrate 0.55 0.54 0.54 209 204 contactbroadcast 0.58 0.21 0.31 639 236 contactcontact 0.24 0.24 0.24 238 232 contactcorrespondence 0.42 0.16 0.23 198 73 contactmeet 0.46 0.23 0.30 200 97 justicearrestjail 0.58 0.70 0.64 104 125 lifedie 0.63 0.62 0.62 218 215 lifeinjure 0.90 0.49 0.63 53 29 manufactureartifact 0.39 0.38 0.38 141 140 movementtransportartifact 0.39 0.24 0.30 242 149 movementtransportperson 0.39 0.29 0.33 447 332 personnelelect 0.54 0.50 0.52 152 141 personnelendposition 0.61 0.40 0.48 186 121 personnelstartposition 0.34 0.24 0.28 116 81 transactiontransaction 0.23 0.12 0.16 51 26 transactiontransfermoney 0.31 0.23 0.26 510 373 transactiontransferownership 0.33 0.33 0.33 261 260 Final Mention Detection Results: Micro Average Macro Average Attributes Prec Rec F1 Prec Rec F1 plain 65.89 48.87 56.12 66.23 49.88 56.90 mention_type 57.02 42.29 48.56 57.87 43.69 49.79 realis_status 49.86 36.98 42.47 51.12 38.39 43.85 mention_type+realis_status 43.22 32.05 36.81 44.63 33.55 38.30