====================================================== TAC KBP 2017 EVENT NUGGET DETECTION EVALUATION RESULTS ====================================================== Team ID: CLUZH Organization: Institute of Computational Linguistics, University of Zurich ************************************************************* Run ID: CLUZH1 Did the run access the live Web during the evaluation window: No Languages addressed: Chinese, English, Spanish Did the run return meaningful confidence values: No ==========English========== Mention Type Results: Type Prec Rec F1 #Gold #Sys conflictattack 0.40 0.58 0.47 410 587 conflictdemonstrate 0.52 0.56 0.54 209 221 contactbroadcast 0.45 0.31 0.37 639 441 contactcontact 0.21 0.44 0.28 238 495 contactcorrespondence 0.29 0.17 0.22 198 117 contactmeet 0.35 0.24 0.29 200 140 justicearrestjail 0.48 0.68 0.56 104 149 lifedie 0.53 0.62 0.57 218 255 lifeinjure 0.82 0.51 0.63 53 33 manufactureartifact 0.34 0.33 0.33 141 134 movementtransportartifact 0.48 0.19 0.27 242 93 movementtransportperson 0.32 0.46 0.38 447 638 personnelelect 0.49 0.60 0.54 152 185 personnelendposition 0.44 0.49 0.46 186 207 personnelstartposition 0.37 0.21 0.27 116 66 transactiontransaction 0.00 0.00 0.00 51 0 transactiontransfermoney 0.36 0.40 0.38 510 568 transactiontransferownership 0.36 0.46 0.40 261 332 Final Mention Detection Results: Micro Average Macro Average Attributes Prec Rec F1 Prec Rec F1 plain 57.34 61.09 59.16 57.33 61.84 59.50 mention_type 47.10 50.18 48.60 47.45 51.15 49.23 realis_status 46.85 49.91 48.33 46.96 50.92 48.86 mention_type+realis_status 38.51 41.03 39.73 38.93 42.00 40.41 ==========Chinese========== Mention Type Results: Type Prec Rec F1 #Gold #Sys conflictattack 0.49 0.33 0.39 820 549 conflictdemonstrate 0.63 0.55 0.59 101 88 contactbroadcast 0.55 0.49 0.52 775 691 contactcontact 0.25 0.05 0.08 62 12 contactcorrespondence 0.29 0.06 0.10 25 5 contactmeet 0.26 0.33 0.29 126 161 justicearrestjail 0.53 0.39 0.45 109 81 lifedie 0.62 0.47 0.53 298 227 lifeinjure 0.83 0.47 0.60 93 52 manufactureartifact 0.32 0.16 0.21 131 65 movementtransportartifact 0.33 0.01 0.02 79 3 movementtransportperson 0.37 0.30 0.33 360 290 personnelelect 0.56 0.41 0.47 176 127 personnelendposition 0.64 0.41 0.50 108 68 personnelstartposition 0.55 0.28 0.37 78 40 transactiontransaction 0.00 0.00 0.00 107 0 transactiontransfermoney 0.23 0.10 0.14 264 110 transactiontransferownership 0.33 0.12 0.18 172 63 Final Mention Detection Results: Micro Average Macro Average Attributes Prec Rec F1 Prec Rec F1 plain 67.76 45.92 54.74 66.60 46.40 54.69 mention_type 62.69 42.48 50.64 60.80 42.88 50.30 realis_status 52.64 35.67 42.52 50.08 35.90 41.83 mention_type+realis_status 49.09 33.27 39.66 46.18 33.53 38.85 ==========Spanish========== Mention Type Results: Type Prec Rec F1 #Gold #Sys conflictattack 0.49 0.34 0.40 498 348 conflictdemonstrate 0.58 0.32 0.41 166 91 contactbroadcast 0.33 0.36 0.34 495 540 contactcontact 0.08 0.03 0.04 136 53 contactcorrespondence 0.19 0.06 0.09 101 32 contactmeet 0.21 0.22 0.22 115 123 justicearrestjail 0.54 0.49 0.51 124 112 lifedie 0.61 0.57 0.59 207 195 lifeinjure 0.90 0.33 0.48 93 34 manufactureartifact 0.24 0.06 0.10 67 17 movementtransportartifact 0.00 0.00 0.00 59 16 movementtransportperson 0.31 0.20 0.24 341 222 personnelelect 0.56 0.36 0.44 141 91 personnelendposition 0.67 0.14 0.24 97 21 personnelstartposition 0.00 0.00 0.00 106 0 transactiontransaction 0.00 0.00 0.00 65 0 transactiontransfermoney 0.25 0.18 0.21 426 305 transactiontransferownership 0.27 0.17 0.21 191 118 Final Mention Detection Results: Micro Average Macro Average Attributes Prec Rec F1 Prec Rec F1 plain 60.74 41.07 49.00 58.95 41.50 48.71 mention_type 51.03 34.51 41.17 48.57 34.21 40.15 realis_status 45.63 30.85 36.81 42.83 31.62 36.38 mention_type+realis_status 38.36 25.94 30.95 35.50 26.40 30.28 ************************************************************* Run ID: CLUZH2 Did the run access the live Web during the evaluation window: No Languages addressed: Chinese, English, Spanish Did the run return meaningful confidence values: No ==========English========== Mention Type Results: Type Prec Rec F1 #Gold #Sys conflictattack 0.42 0.50 0.46 410 483 conflictdemonstrate 0.54 0.53 0.54 209 205 contactbroadcast 0.51 0.23 0.31 639 285 contactcontact 0.25 0.33 0.28 238 316 contactcorrespondence 0.35 0.09 0.14 198 50 contactmeet 0.40 0.19 0.26 200 94 justicearrestjail 0.53 0.63 0.58 104 125 lifedie 0.55 0.57 0.56 218 225 lifeinjure 0.81 0.50 0.62 53 33 manufactureartifact 0.35 0.17 0.23 141 69 movementtransportartifact 0.48 0.04 0.08 242 21 movementtransportperson 0.39 0.33 0.36 447 382 personnelelect 0.51 0.41 0.46 152 123 personnelendposition 0.50 0.41 0.45 186 154 personnelstartposition 0.44 0.13 0.20 116 33 transactiontransaction 0.00 0.00 0.00 51 0 transactiontransfermoney 0.40 0.31 0.35 510 396 transactiontransferownership 0.38 0.36 0.37 261 245 Final Mention Detection Results: Micro Average Macro Average Attributes Prec Rec F1 Prec Rec F1 plain 68.63 50.81 58.39 68.05 51.48 58.62 mention_type 56.76 42.02 48.29 56.85 43.35 49.19 realis_status 52.08 38.56 44.31 51.95 39.85 45.10 mention_type+realis_status 43.12 31.92 36.69 43.58 33.54 37.90 ==========Chinese========== Mention Type Results: Type Prec Rec F1 #Gold #Sys conflictattack 0.45 0.31 0.37 820 571 conflictdemonstrate 0.69 0.53 0.60 101 78 contactbroadcast 0.59 0.46 0.51 775 603 contactcontact 0.00 0.00 0.00 62 3 contactcorrespondence 0.57 0.09 0.16 25 4 contactmeet 0.27 0.30 0.28 126 141 justicearrestjail 0.62 0.32 0.42 109 56 lifedie 0.52 0.43 0.47 298 246 lifeinjure 0.88 0.41 0.55 93 43 manufactureartifact 0.30 0.11 0.16 131 49 movementtransportartifact 0.00 0.00 0.00 79 5 movementtransportperson 0.43 0.25 0.32 360 211 personnelelect 0.58 0.43 0.49 176 129 personnelendposition 0.60 0.39 0.47 108 70 personnelstartposition 0.65 0.22 0.33 78 26 transactiontransaction 0.00 0.00 0.00 107 0 transactiontransfermoney 0.25 0.12 0.17 264 129 transactiontransferownership 0.26 0.10 0.14 172 63 Final Mention Detection Results: Micro Average Macro Average Attributes Prec Rec F1 Prec Rec F1 plain 69.02 43.13 53.09 67.56 43.28 52.76 mention_type 64.05 40.02 49.26 62.50 40.32 49.02 realis_status 52.76 32.97 40.58 51.30 33.35 40.42 mention_type+realis_status 49.25 30.77 37.88 48.12 31.37 37.98 ==========Spanish========== Mention Type Results: Type Prec Rec F1 #Gold #Sys conflictattack 0.09 0.06 0.07 498 374 conflictdemonstrate 0.11 0.05 0.07 166 84 contactbroadcast 0.01 0.01 0.01 495 565 contactcontact 0.00 0.00 0.00 136 9 contactcorrespondence 0.00 0.00 0.00 101 35 contactmeet 0.02 0.02 0.02 115 119 justicearrestjail 0.11 0.10 0.11 124 122 lifedie 0.07 0.07 0.07 207 198 lifeinjure 0.00 0.00 0.00 93 33 manufactureartifact 0.00 0.00 0.00 67 18 movementtransportartifact 0.00 0.00 0.00 59 17 movementtransportperson 0.06 0.04 0.05 341 269 personnelelect 0.11 0.08 0.09 141 101 personnelendposition 0.05 0.01 0.02 97 22 personnelstartposition 0.00 0.00 0.00 106 0 transactiontransaction 0.00 0.00 0.00 65 0 transactiontransfermoney 0.21 0.14 0.17 426 289 transactiontransferownership 0.17 0.13 0.15 191 144 Final Mention Detection Results: Micro Average Macro Average Attributes Prec Rec F1 Prec Rec F1 plain 60.93 42.64 50.17 58.75 42.39 49.25 mention_type 51.99 36.38 42.81 49.30 35.40 41.21 realis_status 9.33 6.53 7.68 9.63 6.59 7.83 mention_type+realis_status 7.85 5.49 6.46 7.96 5.09 6.21 ************************************************************* Run ID: CLUZH3 Did the run access the live Web during the evaluation window: No Languages addressed: Chinese, English, Spanish Did the run return meaningful confidence values: No ==========English========== Mention Type Results: Type Prec Rec F1 #Gold #Sys conflictattack 0.39 0.56 0.46 410 587 conflictdemonstrate 0.51 0.58 0.54 209 237 contactbroadcast 0.47 0.30 0.37 639 411 contactcontact 0.23 0.37 0.28 238 394 contactcorrespondence 0.30 0.17 0.22 198 115 contactmeet 0.38 0.23 0.29 200 124 justicearrestjail 0.53 0.68 0.60 104 133 lifedie 0.52 0.62 0.56 218 260 lifeinjure 0.78 0.53 0.63 53 36 manufactureartifact 0.38 0.21 0.27 141 79 movementtransportartifact 0.33 0.07 0.12 242 51 movementtransportperson 0.36 0.42 0.39 447 510 personnelelect 0.47 0.51 0.49 152 165 personnelendposition 0.45 0.46 0.46 186 191 personnelstartposition 0.41 0.20 0.27 116 56 transactiontransaction 0.00 0.00 0.00 51 0 transactiontransfermoney 0.39 0.36 0.37 510 477 transactiontransferownership 0.32 0.44 0.37 261 357 Final Mention Detection Results: Micro Average Macro Average Attributes Prec Rec F1 Prec Rec F1 plain 58.58 56.01 57.27 57.95 56.47 57.20 mention_type 48.75 46.61 47.66 48.73 47.91 48.32 realis_status 47.75 45.65 46.67 46.96 46.53 46.75 mention_type+realis_status 39.86 38.11 38.96 39.57 39.38 39.47 ==========Chinese========== Mention Type Results: Type Prec Rec F1 #Gold #Sys conflictattack 0.43 0.36 0.40 820 689 conflictdemonstrate 0.64 0.57 0.60 101 90 contactbroadcast 0.54 0.52 0.53 775 752 contactcontact 0.15 0.03 0.05 62 13 contactcorrespondence 0.16 0.03 0.05 25 5 contactmeet 0.26 0.33 0.29 126 160 justicearrestjail 0.55 0.37 0.44 109 73 lifedie 0.53 0.47 0.50 298 269 lifeinjure 0.84 0.44 0.58 93 49 manufactureartifact 0.29 0.13 0.18 131 59 movementtransportartifact 0.00 0.00 0.00 79 3 movementtransportperson 0.35 0.32 0.33 360 330 personnelelect 0.55 0.51 0.53 176 162 personnelendposition 0.59 0.40 0.47 108 73 personnelstartposition 0.61 0.40 0.48 78 51 transactiontransaction 0.00 0.00 0.00 107 1 transactiontransfermoney 0.24 0.14 0.17 264 152 transactiontransferownership 0.26 0.12 0.17 172 80 Final Mention Detection Results: Micro Average Macro Average Attributes Prec Rec F1 Prec Rec F1 plain 61.50 47.67 53.71 60.51 48.44 53.81 mention_type 56.07 43.47 48.97 54.43 44.19 48.78 realis_status 49.66 38.50 43.37 47.57 39.01 42.87 mention_type+realis_status 45.76 35.48 39.97 43.35 36.04 39.36 ==========Spanish========== Mention Type Results: Type Prec Rec F1 #Gold #Sys conflictattack 0.46 0.32 0.38 498 352 conflictdemonstrate 0.54 0.23 0.33 166 72 contactbroadcast 0.32 0.41 0.36 495 628 contactcontact 0.12 0.01 0.01 136 8 contactcorrespondence 0.12 0.03 0.05 101 25 contactmeet 0.16 0.14 0.15 115 99 justicearrestjail 0.54 0.47 0.50 124 108 lifedie 0.55 0.56 0.55 207 209 lifeinjure 0.94 0.26 0.41 93 26 manufactureartifact 0.00 0.00 0.00 67 4 movementtransportartifact 0.00 0.00 0.00 59 10 movementtransportperson 0.29 0.20 0.24 341 240 personnelelect 0.55 0.33 0.41 141 83 personnelendposition 0.64 0.07 0.13 97 11 personnelstartposition 0.00 0.00 0.00 106 0 transactiontransaction 0.00 0.00 0.00 65 0 transactiontransfermoney 0.19 0.13 0.15 426 296 transactiontransferownership 0.23 0.14 0.18 191 122 Final Mention Detection Results: Micro Average Macro Average Attributes Prec Rec F1 Prec Rec F1 plain 59.75 39.97 47.89 58.21 40.06 47.46 mention_type 49.65 33.21 39.80 47.52 32.55 38.64 realis_status 42.68 28.55 34.21 40.21 29.16 33.81 mention_type+realis_status 35.91 24.02 28.79 33.21 24.26 28.04