====================================================== TAC KBP 2017 EVENT NUGGET DETECTION EVALUATION RESULTS ====================================================== Team ID: UI_CCG Organization: University of Illinois at Urbana Champaign ************************************************************* Run ID: UI_CCG1 Did the run access the live Web during the evaluation window: No Languages addressed: English, Spanish Did the run return meaningful confidence values: No ==========English========== Mention Type Results: Type Prec Rec F1 #Gold #Sys conflictattack 0.16 0.14 0.15 410 349 conflictdemonstrate 0.33 0.10 0.15 209 63 contactbroadcast 0.50 0.12 0.19 639 153 contactcontact 0.04 0.02 0.02 238 92 contactcorrespondence 0.01 0.03 0.02 198 440 contactmeet 0.16 0.10 0.12 200 120 justicearrestjail 0.51 0.38 0.43 104 77 lifedie 0.53 0.37 0.43 218 150 lifeinjure 0.31 0.36 0.33 53 62 manufactureartifact 0.12 0.11 0.11 141 123 movementtransportartifact 0.19 0.04 0.07 242 56 movementtransportperson 0.06 0.05 0.06 447 395 personnelelect 0.33 0.21 0.26 152 96 personnelendposition 0.23 0.15 0.18 186 116 personnelstartposition 0.23 0.14 0.18 116 73 transactiontransaction 0.00 0.00 0.00 51 2 transactiontransfermoney 0.23 0.06 0.10 510 135 transactiontransferownership 0.12 0.11 0.11 261 240 Final Mention Detection Results: Micro Average Macro Average Attributes Prec Rec F1 Prec Rec F1 plain 54.01 33.85 41.61 54.61 34.90 42.59 mention_type 37.16 23.29 28.64 38.19 24.59 29.92 realis_status 29.81 18.68 22.97 30.60 19.75 24.01 mention_type+realis_status 18.44 11.56 14.21 19.43 12.54 15.24 ==========Spanish========== Mention Type Results: Type Prec Rec F1 #Gold #Sys conflictattack 0.22 0.11 0.14 498 242 conflictdemonstrate 0.48 0.08 0.14 166 29 contactbroadcast 0.39 0.07 0.12 495 91 contactcontact 0.02 0.01 0.01 136 53 contactcorrespondence 0.04 0.06 0.05 101 145 contactmeet 0.08 0.04 0.05 115 60 justicearrestjail 0.30 0.08 0.13 124 34 lifedie 0.38 0.24 0.30 207 133 lifeinjure 0.29 0.07 0.12 93 24 manufactureartifact 0.02 0.01 0.02 67 43 movementtransportartifact 0.07 0.03 0.04 59 27 movementtransportperson 0.05 0.05 0.05 341 331 personnelelect 0.28 0.09 0.14 141 48 personnelendposition 0.02 0.02 0.02 97 88 personnelstartposition 0.04 0.01 0.02 106 23 transactiontransaction 0.00 0.00 0.00 65 0 transactiontransfermoney 0.10 0.02 0.04 426 103 transactiontransferownership 0.04 0.03 0.03 191 134 Final Mention Detection Results: Micro Average Macro Average Attributes Prec Rec F1 Prec Rec F1 plain 32.36 15.18 20.67 32.28 14.76 20.26 mention_type 24.93 11.69 15.92 25.08 10.99 15.28 realis_status 19.13 8.97 12.22 19.18 9.01 12.26 mention_type+realis_status 14.47 6.79 9.24 15.01 6.62 9.18 ************************************************************* Run ID: UI_CCG2 Did the run access the live Web during the evaluation window: No Languages addressed: English, Spanish Did the run return meaningful confidence values: No ==========English========== Mention Type Results: Type Prec Rec F1 #Gold #Sys conflictattack 0.17 0.15 0.16 410 364 conflictdemonstrate 0.33 0.10 0.15 209 64 contactbroadcast 0.50 0.12 0.19 639 153 contactcontact 0.04 0.02 0.02 238 92 contactcorrespondence 0.01 0.03 0.02 198 463 contactmeet 0.16 0.10 0.12 200 122 justicearrestjail 0.50 0.38 0.43 104 80 lifedie 0.54 0.38 0.45 218 155 lifeinjure 0.32 0.38 0.34 53 63 manufactureartifact 0.12 0.11 0.11 141 123 movementtransportartifact 0.10 0.06 0.08 242 145 movementtransportperson 0.06 0.05 0.06 447 395 personnelelect 0.35 0.23 0.28 152 100 personnelendposition 0.23 0.15 0.18 186 119 personnelstartposition 0.18 0.14 0.16 116 93 transactiontransaction 0.00 0.00 0.00 51 2 transactiontransfermoney 0.18 0.07 0.10 510 186 transactiontransferownership 0.12 0.11 0.11 261 242 Final Mention Detection Results: Micro Average Macro Average Attributes Prec Rec F1 Prec Rec F1 plain 52.24 35.36 42.17 52.87 36.33 43.07 mention_type 35.84 24.25 28.93 37.02 25.55 30.23 realis_status 28.44 19.25 22.96 29.37 20.35 24.04 mention_type+realis_status 17.74 12.01 14.32 18.90 13.03 15.43 ==========Spanish========== Mention Type Results: Type Prec Rec F1 #Gold #Sys conflictattack 0.22 0.15 0.18 498 350 conflictdemonstrate 0.39 0.08 0.14 166 36 contactbroadcast 0.35 0.09 0.14 495 128 contactcontact 0.02 0.04 0.03 136 243 contactcorrespondence 0.05 0.07 0.05 101 154 contactmeet 0.16 0.10 0.12 115 75 justicearrestjail 0.32 0.13 0.19 124 51 lifedie 0.41 0.31 0.35 207 155 lifeinjure 0.28 0.08 0.13 93 28 manufactureartifact 0.05 0.04 0.05 67 54 movementtransportartifact 0.03 0.03 0.03 59 70 movementtransportperson 0.05 0.06 0.06 341 351 personnelelect 0.33 0.16 0.21 141 68 personnelendposition 0.03 0.03 0.03 97 114 personnelstartposition 0.03 0.01 0.01 106 33 transactiontransaction 0.00 0.00 0.00 65 32 transactiontransfermoney 0.07 0.02 0.03 426 147 transactiontransferownership 0.03 0.03 0.03 191 148 Final Mention Detection Results: Micro Average Macro Average Attributes Prec Rec F1 Prec Rec F1 plain 34.45 22.48 27.21 33.19 22.19 26.60 mention_type 24.87 16.23 19.64 23.90 15.46 18.78 realis_status 19.96 13.02 15.76 19.43 13.21 15.72 mention_type+realis_status 13.92 9.08 10.99 13.83 9.00 10.90 ************************************************************* Run ID: UI_CCG3 Did the run access the live Web during the evaluation window: No Languages addressed: English, Spanish Did the run return meaningful confidence values: No ==========English========== Mention Type Results: Type Prec Rec F1 #Gold #Sys conflictattack 0.21 0.21 0.21 410 410 conflictdemonstrate 0.44 0.31 0.36 209 145 contactbroadcast 0.50 0.16 0.24 639 203 contactcontact 0.09 0.05 0.07 238 147 contactcorrespondence 0.02 0.05 0.03 198 481 contactmeet 0.16 0.10 0.12 200 126 justicearrestjail 0.47 0.42 0.44 104 94 lifedie 0.53 0.39 0.45 218 161 lifeinjure 0.32 0.38 0.34 53 63 manufactureartifact 0.14 0.13 0.14 141 133 movementtransportartifact 0.10 0.06 0.08 242 143 movementtransportperson 0.08 0.07 0.07 447 442 personnelelect 0.34 0.23 0.28 152 104 personnelendposition 0.25 0.19 0.21 186 142 personnelstartposition 0.19 0.18 0.18 116 109 transactiontransaction 0.00 0.00 0.00 51 16 transactiontransfermoney 0.19 0.09 0.12 510 236 transactiontransferownership 0.11 0.11 0.11 261 260 Final Mention Detection Results: Micro Average Macro Average Attributes Prec Rec F1 Prec Rec F1 plain 53.44 41.72 46.86 54.14 42.94 47.90 mention_type 37.46 29.24 32.85 38.50 30.93 34.30 realis_status 30.30 23.65 26.57 31.32 24.99 27.80 mention_type+realis_status 19.80 15.46 17.36 20.86 16.72 18.56 ==========Spanish========== Mention Type Results: Type Prec Rec F1 #Gold #Sys conflictattack 0.25 0.19 0.22 498 377 conflictdemonstrate 0.34 0.17 0.22 166 83 contactbroadcast 0.35 0.09 0.14 495 128 contactcontact 0.02 0.04 0.03 136 243 contactcorrespondence 0.05 0.08 0.06 101 165 contactmeet 0.09 0.05 0.06 115 64 justicearrestjail 0.27 0.15 0.19 124 67 lifedie 0.42 0.33 0.37 207 162 lifeinjure 0.28 0.08 0.13 93 28 manufactureartifact 0.05 0.04 0.05 67 54 movementtransportartifact 0.03 0.03 0.03 59 72 movementtransportperson 0.05 0.06 0.06 341 351 personnelelect 0.41 0.31 0.35 141 105 personnelendposition 0.02 0.03 0.03 97 121 personnelstartposition 0.15 0.06 0.08 106 41 transactiontransaction 0.00 0.00 0.00 65 32 transactiontransfermoney 0.07 0.03 0.04 426 172 transactiontransferownership 0.03 0.03 0.03 191 175 Final Mention Detection Results: Micro Average Macro Average Attributes Prec Rec F1 Prec Rec F1 plain 37.40 26.62 31.10 36.33 26.64 30.74 mention_type 27.96 19.90 23.25 27.22 19.32 22.60 realis_status 21.17 15.07 17.60 21.35 15.83 18.18 mention_type+realis_status 15.26 10.86 12.69 15.74 11.24 13.12