====================================================================== TAC KBP 2017 EVENT NUGGET DETECTION AND COREFERENCE EVALUATION RESULTS ====================================================================== Team ID: srcb Organization: Ricoh Software Research Center (Beijing) Co.,Ltd. ************************************************************* Run ID: srcb1 Did the run access the live Web during the evaluation window: No Languages addressed: Chinese, English Did the run return meaningful confidence values: No ==========English========== Mention Type Results: Type Prec Rec F1 #Gold #Sys conflictattack 0.43 0.50 0.46 410 471 conflictdemonstrate 0.62 0.58 0.60 209 196 contactbroadcast 0.41 0.34 0.37 639 529 contactcontact 0.19 0.41 0.26 238 517 contactcorrespondence 0.30 0.27 0.29 198 180 contactmeet 0.38 0.28 0.32 200 143 justicearrestjail 0.61 0.61 0.61 104 103 lifedie 0.61 0.56 0.58 218 200 lifeinjure 0.65 0.53 0.58 53 43 manufactureartifact 0.37 0.35 0.36 141 133 movementtransportartifact 0.38 0.23 0.29 242 150 movementtransportperson 0.36 0.37 0.36 447 462 personnelelect 0.47 0.52 0.49 152 169 personnelendposition 0.55 0.43 0.48 186 144 personnelstartposition 0.39 0.31 0.34 116 91 transactiontransaction 0.07 0.03 0.04 51 24 transactiontransfermoney 0.41 0.34 0.37 510 429 transactiontransferownership 0.33 0.37 0.35 261 294 Final Mention Detection Results: Micro Average Macro Average Attributes Prec Rec F1 Prec Rec F1 plain 68.04 66.53 67.27 68.07 68.04 68.06 mention_type 56.83 55.57 56.19 57.02 56.82 56.92 realis_status 47.95 46.89 47.42 48.77 48.73 48.75 mention_type+realis_status 39.69 38.81 39.24 40.47 40.17 40.32 Final Mention Coreference Results: Metric : bcub Score 45.18 Metric : ceafe Score 42.57 Metric : ceafm Score 43.06 * Metric : muc Score 25.84 Metric : blanc Score 24.59 Overall Average CoNLL score 34.55 * Score not included for final CoNLL score. ==========Chinese========== Mention Type Results: Type Prec Rec F1 #Gold #Sys conflictattack 0.40 0.28 0.33 820 574 conflictdemonstrate 0.60 0.40 0.48 101 67 contactbroadcast 0.40 0.46 0.43 775 889 contactcontact 0.01 0.02 0.01 62 113 contactcorrespondence 0.60 0.12 0.20 25 5 contactmeet 0.17 0.29 0.22 126 216 justicearrestjail 0.48 0.40 0.44 109 92 lifedie 0.52 0.45 0.49 298 258 lifeinjure 0.75 0.59 0.66 93 73 manufactureartifact 0.16 0.18 0.17 131 149 movementtransportartifact 0.15 0.12 0.13 79 67 movementtransportperson 0.15 0.24 0.19 360 571 personnelelect 0.42 0.30 0.35 176 128 personnelendposition 0.40 0.29 0.33 108 78 personnelstartposition 0.19 0.39 0.26 78 157 transactiontransaction 0.00 0.00 0.00 107 0 transactiontransfermoney 0.05 0.05 0.05 264 225 transactiontransferownership 0.16 0.15 0.15 172 159 Final Mention Detection Results: Micro Average Macro Average Attributes Prec Rec F1 Prec Rec F1 plain 47.18 46.41 46.79 46.58 47.21 46.89 mention_type 42.28 41.59 41.93 41.98 42.52 42.25 realis_status 33.92 33.37 33.64 32.86 33.54 33.20 mention_type+realis_status 30.81 30.31 30.55 29.95 30.64 30.29 Final Mention Coreference Results: Metric : bcub Score 31.26 Metric : ceafe Score 31.19 Metric : ceafm Score 32.03 * Metric : muc Score 19.68 Metric : blanc Score 13.44 Overall Average CoNLL score 23.89 * Score not included for final CoNLL score. ************************************************************* Run ID: srcb2 Did the run access the live Web during the evaluation window: No Languages addressed: Chinese, English Did the run return meaningful confidence values: No ==========English========== Mention Type Results: Type Prec Rec F1 #Gold #Sys conflictattack 0.43 0.50 0.46 410 471 conflictdemonstrate 0.62 0.58 0.60 209 196 contactbroadcast 0.41 0.34 0.37 639 529 contactcontact 0.19 0.41 0.26 238 517 contactcorrespondence 0.30 0.27 0.29 198 180 contactmeet 0.38 0.28 0.32 200 143 justicearrestjail 0.61 0.61 0.61 104 103 lifedie 0.61 0.56 0.58 218 200 lifeinjure 0.65 0.53 0.58 53 43 manufactureartifact 0.37 0.35 0.36 141 133 movementtransportartifact 0.38 0.23 0.29 242 150 movementtransportperson 0.36 0.37 0.36 447 462 personnelelect 0.47 0.52 0.49 152 169 personnelendposition 0.55 0.43 0.48 186 144 personnelstartposition 0.39 0.31 0.34 116 91 transactiontransaction 0.07 0.03 0.04 51 24 transactiontransfermoney 0.41 0.34 0.37 510 429 transactiontransferownership 0.33 0.37 0.35 261 294 Final Mention Detection Results: Micro Average Macro Average Attributes Prec Rec F1 Prec Rec F1 plain 68.04 66.53 67.27 68.07 68.04 68.06 mention_type 56.83 55.57 56.19 57.02 56.82 56.92 realis_status 47.95 46.89 47.42 48.77 48.73 48.75 mention_type+realis_status 39.69 38.81 39.24 40.47 40.17 40.32 Final Mention Coreference Results: Metric : bcub Score 43.84 Metric : ceafe Score 39.86 Metric : ceafm Score 41.62 * Metric : muc Score 30.63 Metric : blanc Score 26.97 Overall Average CoNLL score 35.33 * Score not included for final CoNLL score. ==========Chinese========== Mention Type Results: Type Prec Rec F1 #Gold #Sys conflictattack 0.40 0.27 0.32 820 548 conflictdemonstrate 0.59 0.43 0.49 101 73 contactbroadcast 0.41 0.47 0.43 775 884 contactcontact 0.01 0.02 0.01 62 116 contactcorrespondence 0.50 0.12 0.19 25 6 contactmeet 0.17 0.28 0.21 126 206 justicearrestjail 0.47 0.40 0.43 109 91 lifedie 0.50 0.45 0.47 298 267 lifeinjure 0.75 0.61 0.67 93 76 manufactureartifact 0.16 0.19 0.17 131 151 movementtransportartifact 0.14 0.12 0.13 79 68 movementtransportperson 0.15 0.24 0.18 360 586 personnelelect 0.40 0.30 0.34 176 132 personnelendposition 0.42 0.31 0.35 108 79 personnelstartposition 0.19 0.37 0.25 78 156 transactiontransaction 0.00 0.00 0.00 107 0 transactiontransfermoney 0.06 0.05 0.06 264 233 transactiontransferownership 0.18 0.16 0.17 172 153 Final Mention Detection Results: Micro Average Macro Average Attributes Prec Rec F1 Prec Rec F1 plain 47.48 46.76 47.12 47.05 47.86 47.45 mention_type 42.47 41.82 42.14 42.28 42.99 42.63 realis_status 33.80 33.29 33.54 33.07 33.70 33.38 mention_type+realis_status 30.66 30.19 30.42 30.08 30.74 30.40 Final Mention Coreference Results: Metric : bcub Score 31.58 Metric : ceafe Score 31.49 Metric : ceafm Score 32.27 * Metric : muc Score 20.01 Metric : blanc Score 13.52 Overall Average CoNLL score 24.15 * Score not included for final CoNLL score. ************************************************************* Run ID: srcb3 Did the run access the live Web during the evaluation window: No Languages addressed: Chinese, English Did the run return meaningful confidence values: No ==========English========== Mention Type Results: Type Prec Rec F1 #Gold #Sys conflictattack 0.38 0.50 0.43 410 539 conflictdemonstrate 0.52 0.58 0.55 209 235 contactbroadcast 0.38 0.34 0.36 639 578 contactcontact 0.17 0.41 0.24 238 554 contactcorrespondence 0.23 0.27 0.25 198 237 contactmeet 0.34 0.28 0.30 200 161 justicearrestjail 0.50 0.61 0.55 104 126 lifedie 0.54 0.56 0.55 218 225 lifeinjure 0.62 0.53 0.57 53 45 manufactureartifact 0.33 0.34 0.34 141 147 movementtransportartifact 0.33 0.23 0.27 242 172 movementtransportperson 0.31 0.37 0.34 447 529 personnelelect 0.39 0.52 0.45 152 203 personnelendposition 0.49 0.43 0.46 186 163 personnelstartposition 0.33 0.31 0.32 116 108 transactiontransaction 0.06 0.03 0.04 51 25 transactiontransfermoney 0.34 0.34 0.34 510 523 transactiontransferownership 0.29 0.37 0.32 261 334 Final Mention Detection Results: Micro Average Macro Average Attributes Prec Rec F1 Prec Rec F1 plain 59.35 66.53 62.73 61.06 68.04 64.36 mention_type 49.58 55.57 52.40 51.16 56.82 53.84 realis_status 41.85 46.91 44.23 44.17 48.76 46.35 mention_type+realis_status 34.64 38.83 36.61 36.68 40.20 38.36 Final Mention Coreference Results: Metric : bcub Score 41.82 Metric : ceafe Score 40.66 Metric : ceafm Score 40.65 * Metric : muc Score 22.25 Metric : blanc Score 21.53 Overall Average CoNLL score 31.56 * Score not included for final CoNLL score. ==========Chinese========== Mention Type Results: Type Prec Rec F1 #Gold #Sys conflictattack 0.41 0.29 0.34 820 574 conflictdemonstrate 0.70 0.47 0.56 101 67 contactbroadcast 0.39 0.44 0.41 775 889 contactcontact 0.02 0.03 0.02 62 113 contactcorrespondence 0.60 0.12 0.20 25 5 contactmeet 0.19 0.33 0.24 126 216 justicearrestjail 0.51 0.43 0.46 109 92 lifedie 0.55 0.48 0.51 298 258 lifeinjure 0.75 0.59 0.66 93 73 manufactureartifact 0.15 0.17 0.15 131 149 movementtransportartifact 0.21 0.17 0.19 79 67 movementtransportperson 0.18 0.29 0.23 360 571 personnelelect 0.45 0.32 0.37 176 128 personnelendposition 0.42 0.31 0.35 108 78 personnelstartposition 0.18 0.37 0.24 78 157 transactiontransaction 0.00 0.00 0.00 107 0 transactiontransfermoney 0.05 0.04 0.04 264 225 transactiontransferownership 0.16 0.15 0.16 172 159 Final Mention Detection Results: Micro Average Macro Average Attributes Prec Rec F1 Prec Rec F1 plain 47.11 46.35 46.73 46.54 47.12 46.83 mention_type 42.27 41.58 41.92 41.97 42.50 42.23 realis_status 34.87 34.30 34.58 34.13 34.39 34.26 mention_type+realis_status 31.77 31.25 31.51 31.26 31.54 31.40 Final Mention Coreference Results: Metric : bcub Score 31.26 Metric : ceafe Score 31.19 Metric : ceafm Score 32.03 * Metric : muc Score 19.68 Metric : blanc Score 13.44 Overall Average CoNLL score 23.89 * Score not included for final CoNLL score.