================================================================================================= TAC KBP 2017 KB CONSTRUCTION: COMPONENT EVENT NUGGET DETECTION AND COREFERENCE EVALUATION RESULTS ================================================================================================= Team ID: A2KD_Adept Organization: Raytheon BBN Technologies ************************************************************* Run ID: A2KD_Adept_KB_CMN_1 See full details of run under Cold Start KB: Composite KB evaluation results ==========Chinese========== Mention Type Results: Type Prec Rec F1 #Gold #Sys conflictattack 0.51 0.24 0.33 820 394 conflictdemonstrate 0.40 0.45 0.42 101 115 contactbroadcast 0.49 0.40 0.44 775 624 contactcontact 0.03 0.03 0.03 62 72 contactcorrespondence 0.13 0.05 0.07 25 10 contactmeet 0.21 0.18 0.20 126 108 justicearrestjail 0.38 0.37 0.37 109 104 lifedie 0.52 0.39 0.45 298 224 lifeinjure 0.68 0.46 0.55 93 63 manufactureartifact 0.06 0.06 0.06 131 130 movementtransportartifact 0.06 0.11 0.08 79 149 movementtransportperson 0.17 0.19 0.18 360 417 personnelelect 0.25 0.25 0.25 176 178 personnelendposition 0.34 0.11 0.17 108 35 personnelstartposition 0.47 0.33 0.39 78 55 transactiontransaction 0.00 0.00 0.00 107 1 transactiontransfermoney 0.05 0.02 0.03 264 92 transactiontransferownership 0.28 0.08 0.12 172 48 Final Mention Detection Results: Micro Average Macro Average Attributes Prec Rec F1 Prec Rec F1 plain 45.91 33.32 38.62 52.88 34.12 41.48 mention_type 43.35 31.46 36.46 49.87 32.26 39.18 realis_status 36.06 26.17 30.33 40.25 26.44 31.92 mention_type+realis_status 34.24 24.85 28.80 38.43 25.21 30.45 Final Mention Coreference Results: Metric : bcub Score 25.71 Metric : ceafe Score 30.54 Metric : ceafm Score 26.97 * Metric : muc Score 8.02 Metric : blanc Score 6.98 Overall Average CoNLL score 17.81 * Score not included for final CoNLL score. ************************************************************* Run ID: A2KD_Adept_KB_CMN_2 See full details of run under Cold Start KB: Composite KB evaluation results ==========Chinese========== Mention Type Results: Type Prec Rec F1 #Gold #Sys conflictattack 0.51 0.24 0.33 820 394 conflictdemonstrate 0.40 0.45 0.42 101 115 contactbroadcast 0.49 0.40 0.44 775 624 contactcontact 0.03 0.03 0.03 62 72 contactcorrespondence 0.13 0.05 0.07 25 10 contactmeet 0.21 0.18 0.20 126 108 justicearrestjail 0.38 0.37 0.37 109 104 lifedie 0.52 0.39 0.45 298 224 lifeinjure 0.68 0.46 0.55 93 63 manufactureartifact 0.06 0.06 0.06 131 130 movementtransportartifact 0.06 0.11 0.08 79 149 movementtransportperson 0.17 0.19 0.18 360 417 personnelelect 0.25 0.25 0.25 176 178 personnelendposition 0.34 0.11 0.17 108 35 personnelstartposition 0.47 0.33 0.39 78 55 transactiontransaction 0.00 0.00 0.00 107 1 transactiontransfermoney 0.05 0.02 0.03 264 92 transactiontransferownership 0.28 0.08 0.12 172 48 Final Mention Detection Results: Micro Average Macro Average Attributes Prec Rec F1 Prec Rec F1 plain 45.91 33.32 38.62 52.88 34.12 41.48 mention_type 43.35 31.46 36.46 49.87 32.26 39.18 realis_status 36.06 26.17 30.33 40.25 26.44 31.92 mention_type+realis_status 34.24 24.85 28.80 38.43 25.21 30.45 Final Mention Coreference Results: Metric : bcub Score 25.63 Metric : ceafe Score 30.52 Metric : ceafm Score 26.97 * Metric : muc Score 7.85 Metric : blanc Score 6.86 Overall Average CoNLL score 17.71 * Score not included for final CoNLL score. ************************************************************* Run ID: A2KD_Adept_KB_CMN_3 See full details of run under Cold Start KB: Composite KB evaluation results ==========Chinese========== Mention Type Results: Type Prec Rec F1 #Gold #Sys conflictattack 0.51 0.24 0.33 820 394 conflictdemonstrate 0.40 0.45 0.42 101 115 contactbroadcast 0.49 0.40 0.44 775 624 contactcontact 0.03 0.03 0.03 62 72 contactcorrespondence 0.13 0.05 0.07 25 10 contactmeet 0.21 0.18 0.20 126 108 justicearrestjail 0.38 0.37 0.37 109 104 lifedie 0.52 0.39 0.45 298 224 lifeinjure 0.68 0.46 0.55 93 63 manufactureartifact 0.06 0.06 0.06 131 130 movementtransportartifact 0.06 0.11 0.08 79 149 movementtransportperson 0.17 0.19 0.18 360 417 personnelelect 0.25 0.25 0.25 176 178 personnelendposition 0.34 0.11 0.17 108 35 personnelstartposition 0.47 0.33 0.39 78 55 transactiontransaction 0.00 0.00 0.00 107 1 transactiontransfermoney 0.05 0.02 0.03 264 92 transactiontransferownership 0.28 0.08 0.12 172 48 Final Mention Detection Results: Micro Average Macro Average Attributes Prec Rec F1 Prec Rec F1 plain 45.91 33.32 38.62 52.88 34.12 41.48 mention_type 43.35 31.46 36.46 49.87 32.26 39.18 realis_status 36.06 26.17 30.33 40.25 26.44 31.92 mention_type+realis_status 34.24 24.85 28.80 38.43 25.21 30.45 Final Mention Coreference Results: Metric : bcub Score 25.72 Metric : ceafe Score 30.59 Metric : ceafm Score 27.18 * Metric : muc Score 8.18 Metric : blanc Score 6.97 Overall Average CoNLL score 17.87 * Score not included for final CoNLL score. ************************************************************* Run ID: A2KD_Adept_KB_CMN_4 See full details of run under Cold Start KB: Composite KB evaluation results ==========Chinese========== Mention Type Results: Type Prec Rec F1 #Gold #Sys conflictattack 0.51 0.24 0.33 820 394 conflictdemonstrate 0.40 0.45 0.42 101 115 contactbroadcast 0.49 0.40 0.44 775 624 contactcontact 0.03 0.03 0.03 62 72 contactcorrespondence 0.13 0.05 0.07 25 10 contactmeet 0.21 0.18 0.20 126 108 justicearrestjail 0.38 0.37 0.37 109 104 lifedie 0.52 0.39 0.45 298 224 lifeinjure 0.68 0.46 0.55 93 63 manufactureartifact 0.06 0.06 0.06 131 130 movementtransportartifact 0.06 0.11 0.08 79 149 movementtransportperson 0.17 0.19 0.18 360 417 personnelelect 0.25 0.25 0.25 176 178 personnelendposition 0.34 0.11 0.17 108 35 personnelstartposition 0.47 0.33 0.39 78 55 transactiontransaction 0.00 0.00 0.00 107 1 transactiontransfermoney 0.05 0.02 0.03 264 92 transactiontransferownership 0.28 0.08 0.12 172 48 Final Mention Detection Results: Micro Average Macro Average Attributes Prec Rec F1 Prec Rec F1 plain 45.91 33.32 38.62 52.88 34.12 41.48 mention_type 43.35 31.46 36.46 49.87 32.26 39.18 realis_status 36.06 26.17 30.33 40.25 26.44 31.92 mention_type+realis_status 34.24 24.85 28.80 38.43 25.21 30.45 Final Mention Coreference Results: Metric : bcub Score 25.73 Metric : ceafe Score 30.80 Metric : ceafm Score 27.15 * Metric : muc Score 7.85 Metric : blanc Score 6.89 Overall Average CoNLL score 17.82 * Score not included for final CoNLL score. ************************************************************* Run ID: A2KD_Adept_KB_ENG_1 See full details of run under Cold Start KB: Composite KB evaluation results ==========English========== Mention Type Results: Type Prec Rec F1 #Gold #Sys conflictattack 0.45 0.30 0.36 410 273 conflictdemonstrate 0.22 0.40 0.28 209 389 contactbroadcast 0.26 0.32 0.29 639 809 contactcontact 0.13 0.32 0.18 238 592 contactcorrespondence 0.35 0.13 0.19 198 75 contactmeet 0.15 0.19 0.17 200 253 justicearrestjail 0.34 0.38 0.36 104 116 lifedie 0.34 0.39 0.36 218 246 lifeinjure 0.27 0.49 0.35 53 95 manufactureartifact 0.07 0.11 0.09 141 210 movementtransportartifact 0.03 0.07 0.05 242 453 movementtransportperson 0.11 0.18 0.13 447 737 personnelelect 0.22 0.32 0.26 152 220 personnelendposition 0.39 0.11 0.17 186 51 personnelstartposition 0.23 0.20 0.21 116 100 transactiontransaction 0.25 0.02 0.04 51 4 transactiontransfermoney 0.11 0.13 0.12 510 587 transactiontransferownership 0.18 0.14 0.16 261 200 Final Mention Detection Results: Micro Average Macro Average Attributes Prec Rec F1 Prec Rec F1 plain 29.20 36.11 32.29 37.08 37.54 37.31 mention_type 25.97 32.12 28.72 32.74 33.52 33.12 realis_status 20.87 25.81 23.08 25.25 27.58 26.36 mention_type+realis_status 18.65 23.06 20.62 22.57 24.76 23.61 Final Mention Coreference Results: Metric : bcub Score 20.43 Metric : ceafe Score 24.97 Metric : ceafm Score 22.09 * Metric : muc Score 6.59 Metric : blanc Score 4.99 Overall Average CoNLL score 14.24 * Score not included for final CoNLL score. ************************************************************* Run ID: A2KD_Adept_KB_ENG_2 See full details of run under Cold Start KB: Composite KB evaluation results ==========English========== Mention Type Results: Type Prec Rec F1 #Gold #Sys conflictattack 0.45 0.30 0.36 410 273 conflictdemonstrate 0.22 0.40 0.28 209 389 contactbroadcast 0.26 0.32 0.29 639 809 contactcontact 0.13 0.32 0.18 238 592 contactcorrespondence 0.35 0.13 0.19 198 75 contactmeet 0.15 0.19 0.17 200 253 justicearrestjail 0.34 0.38 0.36 104 116 lifedie 0.34 0.39 0.36 218 246 lifeinjure 0.27 0.49 0.35 53 95 manufactureartifact 0.07 0.11 0.09 141 210 movementtransportartifact 0.03 0.07 0.05 242 453 movementtransportperson 0.11 0.18 0.13 447 737 personnelelect 0.22 0.32 0.26 152 220 personnelendposition 0.39 0.11 0.17 186 51 personnelstartposition 0.23 0.20 0.21 116 100 transactiontransaction 0.25 0.02 0.04 51 4 transactiontransfermoney 0.11 0.13 0.12 510 587 transactiontransferownership 0.18 0.14 0.16 261 200 Final Mention Detection Results: Micro Average Macro Average Attributes Prec Rec F1 Prec Rec F1 plain 29.20 36.11 32.29 37.08 37.54 37.31 mention_type 25.97 32.12 28.72 32.74 33.52 33.12 realis_status 20.87 25.81 23.08 25.25 27.58 26.36 mention_type+realis_status 18.65 23.06 20.62 22.57 24.76 23.61 Final Mention Coreference Results: Metric : bcub Score 20.43 Metric : ceafe Score 24.97 Metric : ceafm Score 22.09 * Metric : muc Score 6.59 Metric : blanc Score 4.99 Overall Average CoNLL score 14.24 * Score not included for final CoNLL score. ************************************************************* Run ID: A2KD_Adept_KB_ENG_3 See full details of run under Cold Start KB: Composite KB evaluation results ==========English========== Mention Type Results: Type Prec Rec F1 #Gold #Sys conflictattack 0.45 0.30 0.36 410 273 conflictdemonstrate 0.22 0.40 0.28 209 389 contactbroadcast 0.26 0.32 0.29 639 809 contactcontact 0.13 0.32 0.18 238 592 contactcorrespondence 0.35 0.13 0.19 198 75 contactmeet 0.15 0.19 0.17 200 253 justicearrestjail 0.34 0.38 0.36 104 116 lifedie 0.34 0.39 0.36 218 246 lifeinjure 0.27 0.49 0.35 53 95 manufactureartifact 0.07 0.11 0.09 141 210 movementtransportartifact 0.03 0.07 0.05 242 453 movementtransportperson 0.11 0.18 0.13 447 737 personnelelect 0.22 0.32 0.26 152 220 personnelendposition 0.39 0.11 0.17 186 51 personnelstartposition 0.23 0.20 0.21 116 100 transactiontransaction 0.25 0.02 0.04 51 4 transactiontransfermoney 0.11 0.13 0.12 510 587 transactiontransferownership 0.18 0.14 0.16 261 200 Final Mention Detection Results: Micro Average Macro Average Attributes Prec Rec F1 Prec Rec F1 plain 29.20 36.11 32.29 37.08 37.54 37.31 mention_type 25.97 32.12 28.72 32.74 33.52 33.12 realis_status 20.87 25.81 23.08 25.25 27.58 26.36 mention_type+realis_status 18.65 23.06 20.62 22.57 24.76 23.61 Final Mention Coreference Results: Metric : bcub Score 20.35 Metric : ceafe Score 24.86 Metric : ceafm Score 22.07 * Metric : muc Score 6.54 Metric : blanc Score 5.02 Overall Average CoNLL score 14.19 * Score not included for final CoNLL score. ************************************************************* Run ID: A2KD_Adept_KB_ENG_4 See full details of run under Cold Start KB: Composite KB evaluation results ==========English========== Mention Type Results: Type Prec Rec F1 #Gold #Sys conflictattack 0.45 0.30 0.36 410 273 conflictdemonstrate 0.22 0.40 0.28 209 389 contactbroadcast 0.26 0.32 0.29 639 809 contactcontact 0.13 0.32 0.18 238 592 contactcorrespondence 0.35 0.13 0.19 198 75 contactmeet 0.15 0.19 0.17 200 253 justicearrestjail 0.34 0.38 0.36 104 116 lifedie 0.34 0.39 0.36 218 246 lifeinjure 0.27 0.49 0.35 53 95 manufactureartifact 0.07 0.11 0.09 141 210 movementtransportartifact 0.03 0.07 0.05 242 453 movementtransportperson 0.11 0.18 0.13 447 737 personnelelect 0.22 0.32 0.26 152 220 personnelendposition 0.39 0.11 0.17 186 51 personnelstartposition 0.23 0.20 0.21 116 100 transactiontransaction 0.25 0.02 0.04 51 4 transactiontransfermoney 0.11 0.13 0.12 510 587 transactiontransferownership 0.18 0.14 0.16 261 200 Final Mention Detection Results: Micro Average Macro Average Attributes Prec Rec F1 Prec Rec F1 plain 29.20 36.11 32.29 37.08 37.54 37.31 mention_type 25.97 32.12 28.72 32.74 33.52 33.12 realis_status 20.87 25.81 23.08 25.25 27.58 26.36 mention_type+realis_status 18.65 23.06 20.62 22.57 24.76 23.61 Final Mention Coreference Results: Metric : bcub Score 20.35 Metric : ceafe Score 24.86 Metric : ceafm Score 22.07 * Metric : muc Score 6.54 Metric : blanc Score 5.02 Overall Average CoNLL score 14.19 * Score not included for final CoNLL score. ************************************************************* Run ID: A2KD_Adept_KB_XLING_1 See full details of run under Cold Start KB: Composite KB evaluation results ==========English========== Mention Type Results: Type Prec Rec F1 #Gold #Sys conflictattack 0.45 0.30 0.36 410 273 conflictdemonstrate 0.22 0.40 0.28 209 389 contactbroadcast 0.26 0.32 0.29 639 809 contactcontact 0.13 0.32 0.18 238 592 contactcorrespondence 0.35 0.13 0.19 198 75 contactmeet 0.15 0.19 0.17 200 253 justicearrestjail 0.34 0.38 0.36 104 116 lifedie 0.34 0.39 0.36 218 246 lifeinjure 0.27 0.49 0.35 53 95 manufactureartifact 0.07 0.11 0.09 141 210 movementtransportartifact 0.03 0.07 0.05 242 453 movementtransportperson 0.11 0.18 0.13 447 737 personnelelect 0.22 0.32 0.26 152 220 personnelendposition 0.39 0.11 0.17 186 51 personnelstartposition 0.23 0.20 0.21 116 100 transactiontransaction 0.25 0.02 0.04 51 4 transactiontransfermoney 0.11 0.13 0.12 510 587 transactiontransferownership 0.18 0.14 0.16 261 200 Final Mention Detection Results: Micro Average Macro Average Attributes Prec Rec F1 Prec Rec F1 plain 29.20 36.11 32.29 37.08 37.54 37.31 mention_type 25.97 32.12 28.72 32.74 33.52 33.12 realis_status 20.87 25.81 23.08 25.25 27.58 26.36 mention_type+realis_status 18.65 23.06 20.62 22.57 24.76 23.61 Final Mention Coreference Results: Metric : bcub Score 20.19 Metric : ceafe Score 24.79 Metric : ceafm Score 21.74 * Metric : muc Score 5.94 Metric : blanc Score 4.78 Overall Average CoNLL score 13.93 * Score not included for final CoNLL score. ==========Chinese========== Mention Type Results: Type Prec Rec F1 #Gold #Sys conflictattack 0.51 0.24 0.33 820 394 conflictdemonstrate 0.40 0.45 0.42 101 115 contactbroadcast 0.49 0.40 0.44 775 624 contactcontact 0.03 0.03 0.03 62 72 contactcorrespondence 0.13 0.05 0.07 25 10 contactmeet 0.21 0.18 0.20 126 108 justicearrestjail 0.38 0.37 0.37 109 104 lifedie 0.52 0.39 0.45 298 224 lifeinjure 0.68 0.46 0.55 93 63 manufactureartifact 0.06 0.06 0.06 131 130 movementtransportartifact 0.06 0.11 0.08 79 149 movementtransportperson 0.17 0.19 0.18 360 417 personnelelect 0.25 0.25 0.25 176 178 personnelendposition 0.34 0.11 0.17 108 35 personnelstartposition 0.47 0.33 0.39 78 55 transactiontransaction 0.00 0.00 0.00 107 1 transactiontransfermoney 0.05 0.02 0.03 264 92 transactiontransferownership 0.28 0.08 0.12 172 48 Final Mention Detection Results: Micro Average Macro Average Attributes Prec Rec F1 Prec Rec F1 plain 45.91 33.32 38.62 52.88 34.12 41.48 mention_type 43.35 31.46 36.46 49.87 32.26 39.18 realis_status 36.06 26.17 30.33 40.25 26.44 31.92 mention_type+realis_status 34.24 24.85 28.80 38.43 25.21 30.45 Final Mention Coreference Results: Metric : bcub Score 25.62 Metric : ceafe Score 30.61 Metric : ceafm Score 26.94 * Metric : muc Score 7.69 Metric : blanc Score 6.76 Overall Average CoNLL score 17.67 * Score not included for final CoNLL score. ************************************************************* Run ID: A2KD_Adept_KB_XLING_2 See full details of run under Cold Start KB: Composite KB evaluation results ==========English========== Mention Type Results: Type Prec Rec F1 #Gold #Sys conflictattack 0.45 0.30 0.36 410 273 conflictdemonstrate 0.22 0.40 0.28 209 389 contactbroadcast 0.26 0.32 0.29 639 809 contactcontact 0.13 0.32 0.18 238 592 contactcorrespondence 0.35 0.13 0.19 198 75 contactmeet 0.15 0.19 0.17 200 253 justicearrestjail 0.34 0.38 0.36 104 116 lifedie 0.34 0.39 0.36 218 246 lifeinjure 0.27 0.49 0.35 53 95 manufactureartifact 0.07 0.11 0.09 141 210 movementtransportartifact 0.03 0.07 0.05 242 453 movementtransportperson 0.11 0.18 0.13 447 737 personnelelect 0.22 0.32 0.26 152 220 personnelendposition 0.39 0.11 0.17 186 51 personnelstartposition 0.23 0.20 0.21 116 100 transactiontransaction 0.25 0.02 0.04 51 4 transactiontransfermoney 0.11 0.13 0.12 510 587 transactiontransferownership 0.18 0.14 0.16 261 200 Final Mention Detection Results: Micro Average Macro Average Attributes Prec Rec F1 Prec Rec F1 plain 29.20 36.11 32.29 37.08 37.54 37.31 mention_type 25.97 32.12 28.72 32.74 33.52 33.12 realis_status 20.87 25.81 23.08 25.25 27.58 26.36 mention_type+realis_status 18.65 23.06 20.62 22.57 24.76 23.61 Final Mention Coreference Results: Metric : bcub Score 20.38 Metric : ceafe Score 25.07 Metric : ceafm Score 22.01 * Metric : muc Score 6.40 Metric : blanc Score 4.82 Overall Average CoNLL score 14.17 * Score not included for final CoNLL score. ==========Chinese========== Mention Type Results: Type Prec Rec F1 #Gold #Sys conflictattack 0.51 0.24 0.33 820 394 conflictdemonstrate 0.40 0.45 0.42 101 115 contactbroadcast 0.49 0.40 0.44 775 624 contactcontact 0.03 0.03 0.03 62 72 contactcorrespondence 0.13 0.05 0.07 25 10 contactmeet 0.21 0.18 0.20 126 108 justicearrestjail 0.38 0.37 0.37 109 104 lifedie 0.52 0.39 0.45 298 224 lifeinjure 0.68 0.46 0.55 93 63 manufactureartifact 0.06 0.06 0.06 131 130 movementtransportartifact 0.06 0.11 0.08 79 149 movementtransportperson 0.17 0.19 0.18 360 417 personnelelect 0.25 0.25 0.25 176 178 personnelendposition 0.34 0.11 0.17 108 35 personnelstartposition 0.47 0.33 0.39 78 55 transactiontransaction 0.00 0.00 0.00 107 1 transactiontransfermoney 0.05 0.02 0.03 264 92 transactiontransferownership 0.28 0.08 0.12 172 48 Final Mention Detection Results: Micro Average Macro Average Attributes Prec Rec F1 Prec Rec F1 plain 45.91 33.32 38.62 52.88 34.12 41.48 mention_type 43.35 31.46 36.46 49.87 32.26 39.18 realis_status 36.06 26.17 30.33 40.25 26.44 31.92 mention_type+realis_status 34.24 24.85 28.80 38.43 25.21 30.45 Final Mention Coreference Results: Metric : bcub Score 25.68 Metric : ceafe Score 30.87 Metric : ceafm Score 27.12 * Metric : muc Score 7.77 Metric : blanc Score 6.77 Overall Average CoNLL score 17.77 * Score not included for final CoNLL score.