================================================================================================= TAC KBP 2017 KB CONSTRUCTION: COMPONENT EVENT NUGGET DETECTION AND COREFERENCE EVALUATION RESULTS ================================================================================================= Team ID: ISCAS_Sogou Organization: Institute of Software, Chinese Academy of Sciences & Sogou, Inc. ************************************************************* Run ID: ISCAS_Sogou_KB_CMN_1 See full details of run under Cold Start KB: Composite KB evaluation results ==========Chinese========== Mention Type Results: Type Prec Rec F1 #Gold #Sys conflictattack 0.52 0.24 0.33 820 382 conflictdemonstrate 0.73 0.46 0.56 101 63 contactbroadcast 0.59 0.38 0.46 775 494 contactcontact 0.00 0.00 0.00 62 24 contactcorrespondence 0.13 0.03 0.05 25 6 contactmeet 0.26 0.25 0.26 126 121 justicearrestjail 0.44 0.25 0.32 109 62 lifedie 0.77 0.31 0.44 298 119 lifeinjure 0.89 0.42 0.57 93 44 manufactureartifact 0.16 0.03 0.05 131 25 movementtransportartifact 0.25 0.01 0.02 79 4 movementtransportperson 0.36 0.20 0.26 360 197 personnelelect 0.64 0.27 0.38 176 75 personnelendposition 0.53 0.23 0.32 108 47 personnelstartposition 0.55 0.27 0.36 78 38 transactiontransaction 0.00 0.00 0.00 107 2 transactiontransfermoney 0.19 0.03 0.05 264 42 transactiontransferownership 0.35 0.10 0.16 172 50 Final Mention Detection Results: Micro Average Macro Average Attributes Prec Rec F1 Prec Rec F1 plain 69.59 32.16 43.99 67.33 32.93 44.23 mention_type 63.29 29.25 40.01 61.16 30.03 40.28 realis_status 55.93 25.85 35.35 53.30 26.34 35.26 mention_type+realis_status 51.30 23.71 32.43 48.88 24.31 32.47 Final Mention Coreference Results: Metric : bcub Score 29.44 Metric : ceafe Score 31.39 Metric : ceafm Score 30.60 * Metric : muc Score 12.48 Metric : blanc Score 9.87 Overall Average CoNLL score 20.80 * Score not included for final CoNLL score. ************************************************************* Run ID: ISCAS_Sogou_KB_CMN_2 See full details of run under Cold Start KB: Composite KB evaluation results ==========Chinese========== Mention Type Results: Type Prec Rec F1 #Gold #Sys conflictattack 0.50 0.23 0.31 820 372 conflictdemonstrate 0.73 0.46 0.56 101 63 contactbroadcast 0.59 0.38 0.46 775 498 contactcontact 0.00 0.00 0.00 62 24 contactcorrespondence 0.13 0.03 0.05 25 6 contactmeet 0.26 0.25 0.26 126 122 justicearrestjail 0.44 0.26 0.33 109 63 lifedie 0.76 0.31 0.44 298 121 lifeinjure 0.91 0.42 0.57 93 43 manufactureartifact 0.16 0.03 0.05 131 25 movementtransportartifact 0.25 0.01 0.02 79 4 movementtransportperson 0.36 0.20 0.26 360 203 personnelelect 0.63 0.27 0.38 176 76 personnelendposition 0.53 0.23 0.32 108 47 personnelstartposition 0.55 0.27 0.36 78 38 transactiontransaction 0.00 0.00 0.00 107 2 transactiontransfermoney 0.19 0.03 0.05 264 43 transactiontransferownership 0.34 0.10 0.16 172 52 Final Mention Detection Results: Micro Average Macro Average Attributes Prec Rec F1 Prec Rec F1 plain 68.99 32.01 43.73 67.01 32.92 44.15 mention_type 62.72 29.10 39.76 60.80 29.97 40.15 realis_status 55.26 25.64 35.03 53.00 26.25 35.11 mention_type+realis_status 50.72 23.53 32.15 48.61 24.23 32.34 Final Mention Coreference Results: Metric : bcub Score 29.00 Metric : ceafe Score 31.18 Metric : ceafm Score 30.14 * Metric : muc Score 11.69 Metric : blanc Score 9.42 Overall Average CoNLL score 20.32 * Score not included for final CoNLL score. ************************************************************* Run ID: ISCAS_Sogou_KB_CMN_3 See full details of run under Cold Start KB: Composite KB evaluation results ==========Chinese========== Mention Type Results: Type Prec Rec F1 #Gold #Sys conflictattack 0.52 0.24 0.33 820 382 conflictdemonstrate 0.73 0.46 0.56 101 63 contactbroadcast 0.59 0.38 0.46 775 494 contactcontact 0.00 0.00 0.00 62 24 contactcorrespondence 0.13 0.03 0.05 25 6 contactmeet 0.26 0.25 0.26 126 121 justicearrestjail 0.44 0.25 0.32 109 62 lifedie 0.77 0.31 0.44 298 119 lifeinjure 0.89 0.42 0.57 93 44 manufactureartifact 0.16 0.03 0.05 131 25 movementtransportartifact 0.25 0.01 0.02 79 4 movementtransportperson 0.36 0.20 0.26 360 197 personnelelect 0.64 0.27 0.38 176 75 personnelendposition 0.53 0.23 0.32 108 47 personnelstartposition 0.55 0.27 0.36 78 38 transactiontransaction 0.00 0.00 0.00 107 2 transactiontransfermoney 0.19 0.03 0.05 264 42 transactiontransferownership 0.35 0.10 0.16 172 50 Final Mention Detection Results: Micro Average Macro Average Attributes Prec Rec F1 Prec Rec F1 plain 69.59 32.16 43.99 67.33 32.93 44.23 mention_type 63.29 29.25 40.01 61.16 30.03 40.28 realis_status 55.93 25.85 35.35 53.30 26.34 35.26 mention_type+realis_status 51.30 23.71 32.43 48.88 24.31 32.47 Final Mention Coreference Results: Metric : bcub Score 29.44 Metric : ceafe Score 31.39 Metric : ceafm Score 30.60 * Metric : muc Score 12.48 Metric : blanc Score 9.87 Overall Average CoNLL score 20.80 * Score not included for final CoNLL score. ************************************************************* Run ID: ISCAS_Sogou_KB_CMN_4 See full details of run under Cold Start KB: Composite KB evaluation results ==========Chinese========== Mention Type Results: Type Prec Rec F1 #Gold #Sys conflictattack 0.52 0.24 0.33 820 382 conflictdemonstrate 0.73 0.46 0.56 101 63 contactbroadcast 0.59 0.38 0.46 775 494 contactcontact 0.00 0.00 0.00 62 24 contactcorrespondence 0.13 0.03 0.05 25 6 contactmeet 0.26 0.25 0.26 126 121 justicearrestjail 0.44 0.25 0.32 109 62 lifedie 0.77 0.31 0.44 298 119 lifeinjure 0.89 0.42 0.57 93 44 manufactureartifact 0.16 0.03 0.05 131 25 movementtransportartifact 0.25 0.01 0.02 79 4 movementtransportperson 0.36 0.20 0.26 360 197 personnelelect 0.64 0.27 0.38 176 75 personnelendposition 0.53 0.23 0.32 108 47 personnelstartposition 0.55 0.27 0.36 78 38 transactiontransaction 0.00 0.00 0.00 107 2 transactiontransfermoney 0.19 0.03 0.05 264 42 transactiontransferownership 0.35 0.10 0.16 172 50 Final Mention Detection Results: Micro Average Macro Average Attributes Prec Rec F1 Prec Rec F1 plain 69.59 32.16 43.99 67.33 32.93 44.23 mention_type 63.29 29.25 40.01 61.16 30.03 40.28 realis_status 55.93 25.85 35.35 53.30 26.34 35.26 mention_type+realis_status 51.30 23.71 32.43 48.88 24.31 32.47 Final Mention Coreference Results: Metric : bcub Score 29.44 Metric : ceafe Score 31.39 Metric : ceafm Score 30.60 * Metric : muc Score 12.48 Metric : blanc Score 9.87 Overall Average CoNLL score 20.80 * Score not included for final CoNLL score. ************************************************************* Run ID: ISCAS_Sogou_KB_CMN_5 See full details of run under Cold Start KB: Composite KB evaluation results ==========Chinese========== Mention Type Results: Type Prec Rec F1 #Gold #Sys conflictattack 0.52 0.24 0.33 820 382 conflictdemonstrate 0.73 0.46 0.56 101 63 contactbroadcast 0.59 0.38 0.46 775 494 contactcontact 0.00 0.00 0.00 62 24 contactcorrespondence 0.13 0.03 0.05 25 6 contactmeet 0.26 0.25 0.26 126 121 justicearrestjail 0.44 0.25 0.32 109 62 lifedie 0.77 0.31 0.44 298 119 lifeinjure 0.89 0.42 0.57 93 44 manufactureartifact 0.16 0.03 0.05 131 25 movementtransportartifact 0.25 0.01 0.02 79 4 movementtransportperson 0.36 0.20 0.26 360 197 personnelelect 0.64 0.27 0.38 176 75 personnelendposition 0.53 0.23 0.32 108 47 personnelstartposition 0.55 0.27 0.36 78 38 transactiontransaction 0.00 0.00 0.00 107 2 transactiontransfermoney 0.19 0.03 0.05 264 42 transactiontransferownership 0.35 0.10 0.16 172 50 Final Mention Detection Results: Micro Average Macro Average Attributes Prec Rec F1 Prec Rec F1 plain 69.59 32.16 43.99 67.33 32.93 44.23 mention_type 63.29 29.25 40.01 61.16 30.03 40.28 realis_status 55.93 25.85 35.35 53.30 26.34 35.26 mention_type+realis_status 51.30 23.71 32.43 48.88 24.31 32.47 Final Mention Coreference Results: Metric : bcub Score 29.44 Metric : ceafe Score 31.39 Metric : ceafm Score 30.60 * Metric : muc Score 12.48 Metric : blanc Score 9.87 Overall Average CoNLL score 20.80 * Score not included for final CoNLL score.