================================================================================================= TAC KBP 2017 KB CONSTRUCTION: COMPONENT EVENT NUGGET DETECTION AND COREFERENCE EVALUATION RESULTS ================================================================================================= Team ID: ICTCAS_OKN Organization: Institute of Computing Technology, Chinese Academy of Sciences ************************************************************* Run ID: ICTCAS_OKN_KB_ENG_1 See full details of run under Cold Start KB: Composite KB evaluation results ==========English========== Mention Type Results: Type Prec Rec F1 #Gold #Sys conflictattack 0.22 0.23 0.23 410 437 conflictdemonstrate 0.33 0.18 0.24 209 118 contactbroadcast 0.24 0.02 0.04 639 59 contactcontact 0.04 0.01 0.01 238 52 contactcorrespondence 0.00 0.00 0.00 198 53 contactmeet 0.17 0.08 0.11 200 95 justicearrestjail 0.45 0.36 0.40 104 82 lifedie 0.26 0.17 0.21 218 147 lifeinjure 0.18 0.13 0.15 53 39 manufactureartifact 0.00 0.00 0.00 141 1 movementtransportartifact 0.00 0.00 0.00 242 0 movementtransportperson 0.25 0.03 0.05 447 48 personnelelect 0.17 0.03 0.05 152 30 personnelendposition 0.59 0.08 0.14 186 25 personnelstartposition 0.00 0.00 0.00 116 0 transactiontransaction 0.00 0.00 0.00 51 4 transactiontransfermoney 0.00 0.00 0.00 510 3 transactiontransferownership 0.00 0.00 0.00 261 7 Final Mention Detection Results: Micro Average Macro Average Attributes Prec Rec F1 Prec Rec F1 plain 50.56 13.87 21.76 44.42 13.87 21.14 mention_type 32.83 9.01 14.13 27.18 8.71 13.19 realis_status 35.51 9.74 15.29 31.15 9.95 15.09 mention_type+realis_status 23.38 6.41 10.07 18.35 6.31 9.40 Final Mention Coreference Results: Metric : bcub Score 9.40 Metric : ceafe Score 7.29 Metric : ceafm Score 10.54 * Metric : muc Score 10.92 Metric : blanc Score 3.96 Overall Average CoNLL score 7.89 * Score not included for final CoNLL score. ************************************************************* Run ID: ICTCAS_OKN_KB_ENG_2 See full details of run under Cold Start KB: Composite KB evaluation results ==========English========== Mention Type Results: Type Prec Rec F1 #Gold #Sys conflictattack 0.22 0.23 0.23 410 437 conflictdemonstrate 0.33 0.18 0.24 209 118 contactbroadcast 0.24 0.02 0.04 639 59 contactcontact 0.04 0.01 0.01 238 52 contactcorrespondence 0.00 0.00 0.00 198 53 contactmeet 0.17 0.08 0.11 200 95 justicearrestjail 0.45 0.36 0.40 104 82 lifedie 0.26 0.17 0.21 218 147 lifeinjure 0.18 0.13 0.15 53 39 manufactureartifact 0.00 0.00 0.00 141 1 movementtransportartifact 0.00 0.00 0.00 242 0 movementtransportperson 0.25 0.03 0.05 447 48 personnelelect 0.17 0.03 0.05 152 30 personnelendposition 0.59 0.08 0.14 186 25 personnelstartposition 0.00 0.00 0.00 116 0 transactiontransaction 0.00 0.00 0.00 51 4 transactiontransfermoney 0.00 0.00 0.00 510 3 transactiontransferownership 0.00 0.00 0.00 261 7 Final Mention Detection Results: Micro Average Macro Average Attributes Prec Rec F1 Prec Rec F1 plain 50.56 13.87 21.76 44.42 13.87 21.14 mention_type 32.83 9.01 14.13 27.18 8.71 13.19 realis_status 35.51 9.74 15.29 31.15 9.95 15.09 mention_type+realis_status 23.38 6.41 10.07 18.35 6.31 9.40 Final Mention Coreference Results: Metric : bcub Score 9.40 Metric : ceafe Score 7.29 Metric : ceafm Score 10.54 * Metric : muc Score 10.92 Metric : blanc Score 3.96 Overall Average CoNLL score 7.89 * Score not included for final CoNLL score. ************************************************************* Run ID: ICTCAS_OKN_KB_ENG_3 See full details of run under Cold Start KB: Composite KB evaluation results ==========English========== Mention Type Results: Type Prec Rec F1 #Gold #Sys conflictattack 0.30 0.21 0.24 410 279 conflictdemonstrate 0.43 0.12 0.19 209 60 contactbroadcast 0.25 0.01 0.02 639 28 contactcontact 0.03 0.00 0.01 238 33 contactcorrespondence 0.00 0.00 0.00 198 33 contactmeet 0.15 0.04 0.07 200 60 justicearrestjail 0.50 0.31 0.38 104 64 lifedie 0.32 0.16 0.21 218 109 lifeinjure 0.32 0.13 0.19 53 22 manufactureartifact 0.00 0.00 0.00 141 1 movementtransportartifact 0.00 0.00 0.00 242 0 movementtransportperson 0.24 0.02 0.04 447 41 personnelelect 0.17 0.01 0.02 152 12 personnelendposition 0.62 0.05 0.10 186 16 personnelstartposition 0.00 0.00 0.00 116 0 transactiontransaction 0.00 0.00 0.00 51 3 transactiontransfermoney 0.00 0.00 0.00 510 3 transactiontransferownership 0.00 0.00 0.00 261 5 Final Mention Detection Results: Micro Average Macro Average Attributes Prec Rec F1 Prec Rec F1 plain 64.32 11.31 19.23 50.19 11.58 18.82 mention_type 42.05 7.39 12.57 31.40 7.29 11.83 realis_status 45.36 7.97 13.56 35.44 8.39 13.56 mention_type+realis_status 29.04 5.10 8.68 21.16 5.18 8.32 Final Mention Coreference Results: Metric : bcub Score 8.91 Metric : ceafe Score 7.57 Metric : ceafm Score 10.03 * Metric : muc Score 9.48 Metric : blanc Score 4.58 Overall Average CoNLL score 7.63 * Score not included for final CoNLL score.