====================================================== TAC KBP 2017 EVENT NUGGET DETECTION EVALUATION RESULTS ====================================================== Team ID: lvic_event Organization: CEA LIST ************************************************************* Run ID: lvic_event1 Did the run access the live Web during the evaluation window: No Languages addressed: English Did the run return meaningful confidence values: No ==========English========== Mention Type Results: Type Prec Rec F1 #Gold #Sys conflictattack 0.44 0.45 0.45 410 424 conflictdemonstrate 0.53 0.55 0.54 209 214 contactbroadcast 0.57 0.27 0.36 639 298 contactcontact 0.22 0.42 0.29 238 455 contactcorrespondence 0.28 0.24 0.26 198 169 contactmeet 0.43 0.23 0.30 200 109 justicearrestjail 0.56 0.69 0.62 104 128 lifedie 0.62 0.61 0.62 218 217 lifeinjure 0.60 0.55 0.57 53 48 manufactureartifact 0.43 0.34 0.38 141 112 movementtransportartifact 0.43 0.27 0.33 242 151 movementtransportperson 0.41 0.39 0.40 447 422 personnelelect 0.47 0.55 0.50 152 177 personnelendposition 0.57 0.39 0.46 186 127 personnelstartposition 0.38 0.22 0.28 116 67 transactiontransaction 0.22 0.08 0.12 51 18 transactiontransfermoney 0.35 0.36 0.36 510 529 transactiontransferownership 0.44 0.15 0.23 261 91 Final Mention Detection Results: Micro Average Macro Average Attributes Prec Rec F1 Prec Rec F1 plain 64.89 55.71 59.95 65.52 56.89 60.90 mention_type 54.27 46.59 50.14 55.15 48.06 51.36 realis_status 51.39 44.12 47.48 52.33 45.38 48.61 mention_type+realis_status 42.52 36.50 39.28 43.56 37.80 40.48 ************************************************************* Run ID: lvic_event2 Did the run access the live Web during the evaluation window: No Languages addressed: English Did the run return meaningful confidence values: No ==========English========== Mention Type Results: Type Prec Rec F1 #Gold #Sys conflictattack 0.45 0.44 0.45 410 403 conflictdemonstrate 0.54 0.56 0.55 209 215 contactbroadcast 0.57 0.26 0.35 639 284 contactcontact 0.25 0.39 0.30 238 375 contactcorrespondence 0.25 0.26 0.25 198 201 contactmeet 0.45 0.23 0.30 200 103 justicearrestjail 0.53 0.69 0.60 104 135 lifedie 0.61 0.62 0.62 218 223 lifeinjure 0.64 0.53 0.58 53 44 manufactureartifact 0.39 0.33 0.36 141 117 movementtransportartifact 0.42 0.25 0.31 242 145 movementtransportperson 0.38 0.40 0.39 447 462 personnelelect 0.49 0.53 0.51 152 166 personnelendposition 0.54 0.40 0.46 186 139 personnelstartposition 0.34 0.23 0.27 116 78 transactiontransaction 0.09 0.04 0.05 51 22 transactiontransfermoney 0.35 0.38 0.36 510 565 transactiontransferownership 0.35 0.14 0.20 261 103 Final Mention Detection Results: Micro Average Macro Average Attributes Prec Rec F1 Prec Rec F1 plain 64.26 55.52 59.57 64.63 56.74 60.43 mention_type 53.57 46.28 49.66 54.29 47.74 50.80 realis_status 50.92 44.00 47.21 51.63 45.12 48.16 mention_type+realis_status 42.02 36.31 38.96 42.78 37.44 39.93 ************************************************************* Run ID: lvic_event3 Did the run access the live Web during the evaluation window: No Languages addressed: English Did the run return meaningful confidence values: No ==========English========== Mention Type Results: Type Prec Rec F1 #Gold #Sys conflictattack 0.42 0.44 0.43 410 423 conflictdemonstrate 0.54 0.56 0.55 209 216 contactbroadcast 0.62 0.25 0.35 639 255 contactcontact 0.23 0.31 0.26 238 328 contactcorrespondence 0.28 0.25 0.26 198 177 contactmeet 0.43 0.23 0.30 200 106 justicearrestjail 0.57 0.67 0.62 104 123 lifedie 0.62 0.63 0.62 218 221 lifeinjure 0.70 0.53 0.60 53 40 manufactureartifact 0.37 0.34 0.35 141 130 movementtransportartifact 0.44 0.25 0.32 242 138 movementtransportperson 0.42 0.37 0.39 447 387 personnelelect 0.48 0.51 0.49 152 160 personnelendposition 0.57 0.39 0.47 186 127 personnelstartposition 0.32 0.23 0.27 116 83 transactiontransaction 0.09 0.02 0.03 51 11 transactiontransfermoney 0.35 0.38 0.37 510 558 transactiontransferownership 0.40 0.11 0.17 261 70 Final Mention Detection Results: Micro Average Macro Average Attributes Prec Rec F1 Prec Rec F1 plain 65.07 52.84 58.32 65.31 54.20 59.24 mention_type 54.91 44.59 49.22 55.73 46.21 50.53 realis_status 51.58 41.89 46.23 52.04 43.30 47.27 mention_type+realis_status 43.05 34.96 38.59 43.85 36.38 39.77