================================================================================================= TAC KBP 2017 KB CONSTRUCTION: COMPONENT EVENT NUGGET DETECTION AND COREFERENCE EVALUATION RESULTS ================================================================================================= Team ID: TinkerBell Organization: RPI, UIUC, Stanford, Columbia, Cornell, JHU, UPenn ************************************************************* Run ID: TinkerBell_KB_CMN_1 See full details of run under Cold Start KB: Composite KB evaluation results ==========Chinese========== Mention Type Results: Type Prec Rec F1 #Gold #Sys conflictattack 0.33 0.26 0.29 820 646 conflictdemonstrate 0.48 0.13 0.20 101 27 contactbroadcast 0.67 0.33 0.44 775 387 contactcontact 0.00 0.00 0.00 62 0 contactcorrespondence 0.22 0.08 0.12 25 9 contactmeet 0.24 0.20 0.22 126 104 justicearrestjail 0.44 0.34 0.38 109 85 lifedie 0.60 0.48 0.53 298 236 lifeinjure 0.67 0.69 0.68 93 96 manufactureartifact 0.21 0.06 0.09 131 36 movementtransportartifact 0.00 0.00 0.00 79 5 movementtransportperson 0.25 0.16 0.20 360 225 personnelelect 0.37 0.19 0.25 176 89 personnelendposition 0.50 0.12 0.19 108 26 personnelstartposition 0.09 0.01 0.02 78 11 transactiontransaction 0.00 0.00 0.00 107 12 transactiontransfermoney 0.12 0.03 0.04 264 54 transactiontransferownership 0.20 0.08 0.11 172 66 Final Mention Detection Results: Micro Average Macro Average Attributes Prec Rec F1 Prec Rec F1 plain 70.60 38.42 49.76 67.36 37.95 48.55 mention_type 65.89 35.86 46.45 61.83 35.28 44.92 realis_status 44.40 24.16 31.29 41.34 24.55 30.80 mention_type+realis_status 41.86 22.78 29.50 38.71 23.11 28.94 Final Mention Coreference Results: Metric : bcub Score 31.35 Metric : ceafe Score 31.14 Metric : ceafm Score 30.87 * Metric : muc Score 15.42 Metric : blanc Score 12.94 Overall Average CoNLL score 22.71 * Score not included for final CoNLL score. ************************************************************* Run ID: TinkerBell_KB_CMN_2 See full details of run under Cold Start KB: Composite KB evaluation results ==========Chinese========== Mention Type Results: Type Prec Rec F1 #Gold #Sys conflictattack 0.33 0.26 0.29 820 646 conflictdemonstrate 0.48 0.13 0.20 101 27 contactbroadcast 0.67 0.33 0.44 775 387 contactcontact 0.00 0.00 0.00 62 0 contactcorrespondence 0.22 0.08 0.12 25 9 contactmeet 0.24 0.20 0.22 126 104 justicearrestjail 0.44 0.34 0.38 109 85 lifedie 0.60 0.48 0.53 298 236 lifeinjure 0.67 0.69 0.68 93 96 manufactureartifact 0.21 0.06 0.09 131 36 movementtransportartifact 0.00 0.00 0.00 79 5 movementtransportperson 0.25 0.16 0.20 360 225 personnelelect 0.37 0.19 0.25 176 89 personnelendposition 0.50 0.12 0.19 108 26 personnelstartposition 0.09 0.01 0.02 78 11 transactiontransaction 0.00 0.00 0.00 107 12 transactiontransfermoney 0.12 0.03 0.04 264 54 transactiontransferownership 0.20 0.08 0.11 172 66 Final Mention Detection Results: Micro Average Macro Average Attributes Prec Rec F1 Prec Rec F1 plain 70.60 38.42 49.76 67.36 37.95 48.55 mention_type 65.89 35.86 46.45 61.83 35.28 44.92 realis_status 44.40 24.16 31.29 41.34 24.55 30.80 mention_type+realis_status 41.86 22.78 29.50 38.71 23.11 28.94 Final Mention Coreference Results: Metric : bcub Score 31.35 Metric : ceafe Score 31.14 Metric : ceafm Score 30.87 * Metric : muc Score 15.42 Metric : blanc Score 12.94 Overall Average CoNLL score 22.71 * Score not included for final CoNLL score. ************************************************************* Run ID: TinkerBell_KB_CMN_3 See full details of run under Cold Start KB: Composite KB evaluation results ==========Chinese========== Mention Type Results: Type Prec Rec F1 #Gold #Sys conflictattack 0.33 0.26 0.29 820 646 conflictdemonstrate 0.48 0.13 0.20 101 27 contactbroadcast 0.67 0.33 0.44 775 387 contactcontact 0.00 0.00 0.00 62 0 contactcorrespondence 0.22 0.08 0.12 25 9 contactmeet 0.24 0.20 0.22 126 104 justicearrestjail 0.44 0.34 0.38 109 85 lifedie 0.60 0.48 0.53 298 236 lifeinjure 0.67 0.69 0.68 93 96 manufactureartifact 0.21 0.06 0.09 131 36 movementtransportartifact 0.00 0.00 0.00 79 5 movementtransportperson 0.25 0.16 0.20 360 225 personnelelect 0.37 0.19 0.25 176 89 personnelendposition 0.50 0.12 0.19 108 26 personnelstartposition 0.09 0.01 0.02 78 11 transactiontransaction 0.00 0.00 0.00 107 12 transactiontransfermoney 0.12 0.03 0.04 264 54 transactiontransferownership 0.20 0.08 0.11 172 66 Final Mention Detection Results: Micro Average Macro Average Attributes Prec Rec F1 Prec Rec F1 plain 70.60 38.42 49.76 67.36 37.95 48.55 mention_type 65.89 35.86 46.45 61.83 35.28 44.92 realis_status 44.40 24.16 31.29 41.34 24.55 30.80 mention_type+realis_status 41.86 22.78 29.50 38.71 23.11 28.94 Final Mention Coreference Results: Metric : bcub Score 31.35 Metric : ceafe Score 31.14 Metric : ceafm Score 30.87 * Metric : muc Score 15.42 Metric : blanc Score 12.94 Overall Average CoNLL score 22.71 * Score not included for final CoNLL score. ************************************************************* Run ID: TinkerBell_KB_CMN_4 See full details of run under Cold Start KB: Composite KB evaluation results ==========Chinese========== Mention Type Results: Type Prec Rec F1 #Gold #Sys conflictattack 0.33 0.26 0.29 820 646 conflictdemonstrate 0.48 0.13 0.20 101 27 contactbroadcast 0.67 0.33 0.44 775 387 contactcontact 0.00 0.00 0.00 62 0 contactcorrespondence 0.22 0.08 0.12 25 9 contactmeet 0.24 0.20 0.22 126 104 justicearrestjail 0.44 0.34 0.38 109 85 lifedie 0.60 0.48 0.53 298 236 lifeinjure 0.67 0.69 0.68 93 96 manufactureartifact 0.21 0.06 0.09 131 36 movementtransportartifact 0.00 0.00 0.00 79 5 movementtransportperson 0.25 0.16 0.20 360 225 personnelelect 0.37 0.19 0.25 176 89 personnelendposition 0.50 0.12 0.19 108 26 personnelstartposition 0.09 0.01 0.02 78 11 transactiontransaction 0.00 0.00 0.00 107 12 transactiontransfermoney 0.12 0.03 0.04 264 54 transactiontransferownership 0.20 0.08 0.11 172 66 Final Mention Detection Results: Micro Average Macro Average Attributes Prec Rec F1 Prec Rec F1 plain 70.60 38.42 49.76 67.36 37.95 48.55 mention_type 65.89 35.86 46.45 61.83 35.28 44.92 realis_status 44.40 24.16 31.29 41.34 24.55 30.80 mention_type+realis_status 41.86 22.78 29.50 38.71 23.11 28.94 Final Mention Coreference Results: Metric : bcub Score 31.35 Metric : ceafe Score 31.14 Metric : ceafm Score 30.87 * Metric : muc Score 15.42 Metric : blanc Score 12.94 Overall Average CoNLL score 22.71 * Score not included for final CoNLL score. ************************************************************* Run ID: TinkerBell_KB_CMN_5 See full details of run under Cold Start KB: Composite KB evaluation results ==========Chinese========== Mention Type Results: Type Prec Rec F1 #Gold #Sys conflictattack 0.33 0.26 0.29 820 646 conflictdemonstrate 0.48 0.13 0.20 101 27 contactbroadcast 0.67 0.33 0.44 775 387 contactcontact 0.00 0.00 0.00 62 0 contactcorrespondence 0.22 0.08 0.12 25 9 contactmeet 0.24 0.20 0.22 126 104 justicearrestjail 0.44 0.34 0.38 109 85 lifedie 0.60 0.48 0.53 298 236 lifeinjure 0.67 0.69 0.68 93 96 manufactureartifact 0.21 0.06 0.09 131 36 movementtransportartifact 0.00 0.00 0.00 79 5 movementtransportperson 0.25 0.16 0.20 360 225 personnelelect 0.37 0.19 0.25 176 89 personnelendposition 0.50 0.12 0.19 108 26 personnelstartposition 0.09 0.01 0.02 78 11 transactiontransaction 0.00 0.00 0.00 107 12 transactiontransfermoney 0.12 0.03 0.04 264 54 transactiontransferownership 0.20 0.08 0.11 172 66 Final Mention Detection Results: Micro Average Macro Average Attributes Prec Rec F1 Prec Rec F1 plain 70.60 38.42 49.76 67.36 37.95 48.55 mention_type 65.89 35.86 46.45 61.83 35.28 44.92 realis_status 44.40 24.16 31.29 41.34 24.55 30.80 mention_type+realis_status 41.86 22.78 29.50 38.71 23.11 28.94 Final Mention Coreference Results: Metric : bcub Score 31.35 Metric : ceafe Score 31.14 Metric : ceafm Score 30.87 * Metric : muc Score 15.42 Metric : blanc Score 12.94 Overall Average CoNLL score 22.71 * Score not included for final CoNLL score. ************************************************************* Run ID: TinkerBell_KB_ENG_1 See full details of run under Cold Start KB: Composite KB evaluation results ==========English========== Mention Type Results: Type Prec Rec F1 #Gold #Sys conflictattack 0.35 0.11 0.17 410 135 conflictdemonstrate 0.32 0.07 0.11 209 44 contactbroadcast 0.30 0.03 0.05 639 54 contactcontact 0.07 0.01 0.01 238 30 contactcorrespondence 0.25 0.01 0.01 198 4 contactmeet 0.38 0.14 0.21 200 76 justicearrestjail 0.67 0.34 0.45 104 52 lifedie 0.59 0.39 0.47 218 143 lifeinjure 0.83 0.38 0.52 53 24 manufactureartifact 0.29 0.01 0.03 141 7 movementtransportartifact 0.14 0.02 0.03 242 28 movementtransportperson 0.18 0.02 0.03 447 44 personnelelect 0.48 0.28 0.36 152 89 personnelendposition 0.52 0.08 0.14 186 29 personnelstartposition 0.37 0.09 0.15 116 29 transactiontransaction 0.00 0.00 0.00 51 3 transactiontransfermoney 0.41 0.07 0.12 510 90 transactiontransferownership 0.18 0.03 0.05 261 45 Final Mention Detection Results: Micro Average Macro Average Attributes Prec Rec F1 Prec Rec F1 plain 65.33 13.83 22.83 64.19 14.98 24.30 mention_type 56.93 12.05 19.89 57.68 13.23 21.53 realis_status 46.70 9.89 16.32 48.50 10.84 17.73 mention_type+realis_status 40.68 8.61 14.21 44.01 9.59 15.75 Final Mention Coreference Results: Metric : bcub Score 14.80 Metric : ceafe Score 15.38 Metric : ceafm Score 15.69 * Metric : muc Score 7.98 Metric : blanc Score 3.88 Overall Average CoNLL score 10.51 * Score not included for final CoNLL score. ************************************************************* Run ID: TinkerBell_KB_ENG_2 See full details of run under Cold Start KB: Composite KB evaluation results ==========English========== Mention Type Results: Type Prec Rec F1 #Gold #Sys conflictattack 0.35 0.11 0.17 410 135 conflictdemonstrate 0.32 0.07 0.11 209 44 contactbroadcast 0.30 0.03 0.05 639 54 contactcontact 0.07 0.01 0.01 238 30 contactcorrespondence 0.25 0.01 0.01 198 4 contactmeet 0.38 0.14 0.21 200 76 justicearrestjail 0.67 0.34 0.45 104 52 lifedie 0.59 0.39 0.47 218 143 lifeinjure 0.83 0.38 0.52 53 24 manufactureartifact 0.29 0.01 0.03 141 7 movementtransportartifact 0.14 0.02 0.03 242 28 movementtransportperson 0.18 0.02 0.03 447 44 personnelelect 0.48 0.28 0.36 152 89 personnelendposition 0.52 0.08 0.14 186 29 personnelstartposition 0.37 0.09 0.15 116 29 transactiontransaction 0.00 0.00 0.00 51 3 transactiontransfermoney 0.41 0.07 0.12 510 90 transactiontransferownership 0.18 0.03 0.05 261 45 Final Mention Detection Results: Micro Average Macro Average Attributes Prec Rec F1 Prec Rec F1 plain 65.33 13.83 22.83 64.19 14.98 24.30 mention_type 56.93 12.05 19.89 57.68 13.23 21.53 realis_status 46.70 9.89 16.32 48.50 10.84 17.73 mention_type+realis_status 40.68 8.61 14.21 44.01 9.59 15.75 Final Mention Coreference Results: Metric : bcub Score 14.80 Metric : ceafe Score 15.38 Metric : ceafm Score 15.69 * Metric : muc Score 7.98 Metric : blanc Score 3.88 Overall Average CoNLL score 10.51 * Score not included for final CoNLL score. ************************************************************* Run ID: TinkerBell_KB_ENG_3 See full details of run under Cold Start KB: Composite KB evaluation results ==========English========== Mention Type Results: Type Prec Rec F1 #Gold #Sys conflictattack 0.35 0.11 0.17 410 135 conflictdemonstrate 0.32 0.07 0.11 209 44 contactbroadcast 0.30 0.03 0.05 639 54 contactcontact 0.07 0.01 0.01 238 30 contactcorrespondence 0.25 0.01 0.01 198 4 contactmeet 0.38 0.14 0.21 200 76 justicearrestjail 0.67 0.34 0.45 104 52 lifedie 0.59 0.39 0.47 218 143 lifeinjure 0.83 0.38 0.52 53 24 manufactureartifact 0.29 0.01 0.03 141 7 movementtransportartifact 0.14 0.02 0.03 242 28 movementtransportperson 0.18 0.02 0.03 447 44 personnelelect 0.48 0.28 0.36 152 89 personnelendposition 0.52 0.08 0.14 186 29 personnelstartposition 0.37 0.09 0.15 116 29 transactiontransaction 0.00 0.00 0.00 51 3 transactiontransfermoney 0.41 0.07 0.12 510 90 transactiontransferownership 0.18 0.03 0.05 261 45 Final Mention Detection Results: Micro Average Macro Average Attributes Prec Rec F1 Prec Rec F1 plain 65.33 13.83 22.83 64.19 14.98 24.30 mention_type 56.93 12.05 19.89 57.68 13.23 21.53 realis_status 46.70 9.89 16.32 48.50 10.84 17.73 mention_type+realis_status 40.68 8.61 14.21 44.01 9.59 15.75 Final Mention Coreference Results: Metric : bcub Score 14.80 Metric : ceafe Score 15.38 Metric : ceafm Score 15.69 * Metric : muc Score 7.98 Metric : blanc Score 3.88 Overall Average CoNLL score 10.51 * Score not included for final CoNLL score. ************************************************************* Run ID: TinkerBell_KB_ENG_4 See full details of run under Cold Start KB: Composite KB evaluation results ==========English========== Mention Type Results: Type Prec Rec F1 #Gold #Sys conflictattack 0.35 0.11 0.17 410 135 conflictdemonstrate 0.32 0.07 0.11 209 44 contactbroadcast 0.30 0.03 0.05 639 54 contactcontact 0.07 0.01 0.01 238 30 contactcorrespondence 0.25 0.01 0.01 198 4 contactmeet 0.38 0.14 0.21 200 76 justicearrestjail 0.67 0.34 0.45 104 52 lifedie 0.59 0.39 0.47 218 143 lifeinjure 0.83 0.38 0.52 53 24 manufactureartifact 0.29 0.01 0.03 141 7 movementtransportartifact 0.14 0.02 0.03 242 28 movementtransportperson 0.18 0.02 0.03 447 44 personnelelect 0.48 0.28 0.36 152 89 personnelendposition 0.52 0.08 0.14 186 29 personnelstartposition 0.37 0.09 0.15 116 29 transactiontransaction 0.00 0.00 0.00 51 3 transactiontransfermoney 0.41 0.07 0.12 510 90 transactiontransferownership 0.18 0.03 0.05 261 45 Final Mention Detection Results: Micro Average Macro Average Attributes Prec Rec F1 Prec Rec F1 plain 65.33 13.83 22.83 64.19 14.98 24.30 mention_type 56.93 12.05 19.89 57.68 13.23 21.53 realis_status 46.70 9.89 16.32 48.50 10.84 17.73 mention_type+realis_status 40.68 8.61 14.21 44.01 9.59 15.75 Final Mention Coreference Results: Metric : bcub Score 14.80 Metric : ceafe Score 15.38 Metric : ceafm Score 15.69 * Metric : muc Score 7.98 Metric : blanc Score 3.88 Overall Average CoNLL score 10.51 * Score not included for final CoNLL score. ************************************************************* Run ID: TinkerBell_KB_ENG_5 See full details of run under Cold Start KB: Composite KB evaluation results ==========English========== Mention Type Results: Type Prec Rec F1 #Gold #Sys conflictattack 0.35 0.11 0.17 410 135 conflictdemonstrate 0.32 0.07 0.11 209 44 contactbroadcast 0.30 0.03 0.05 639 54 contactcontact 0.07 0.01 0.01 238 30 contactcorrespondence 0.25 0.01 0.01 198 4 contactmeet 0.38 0.14 0.21 200 76 justicearrestjail 0.67 0.34 0.45 104 52 lifedie 0.59 0.39 0.47 218 143 lifeinjure 0.83 0.38 0.52 53 24 manufactureartifact 0.29 0.01 0.03 141 7 movementtransportartifact 0.14 0.02 0.03 242 28 movementtransportperson 0.18 0.02 0.03 447 44 personnelelect 0.48 0.28 0.36 152 89 personnelendposition 0.52 0.08 0.14 186 29 personnelstartposition 0.37 0.09 0.15 116 29 transactiontransaction 0.00 0.00 0.00 51 3 transactiontransfermoney 0.41 0.07 0.12 510 90 transactiontransferownership 0.18 0.03 0.05 261 45 Final Mention Detection Results: Micro Average Macro Average Attributes Prec Rec F1 Prec Rec F1 plain 65.33 13.83 22.83 64.19 14.98 24.30 mention_type 56.93 12.05 19.89 57.68 13.23 21.53 realis_status 46.70 9.89 16.32 48.50 10.84 17.73 mention_type+realis_status 40.68 8.61 14.21 44.01 9.59 15.75 Final Mention Coreference Results: Metric : bcub Score 14.80 Metric : ceafe Score 15.38 Metric : ceafm Score 15.69 * Metric : muc Score 7.98 Metric : blanc Score 3.88 Overall Average CoNLL score 10.51 * Score not included for final CoNLL score. ************************************************************* Run ID: TinkerBell_KB_SPA_1 See full details of run under Cold Start KB: Composite KB evaluation results ==========Spanish========== Mention Type Results: Type Prec Rec F1 #Gold #Sys conflictattack 0.25 0.05 0.08 498 91 conflictdemonstrate 0.50 0.07 0.12 166 22 contactbroadcast 0.25 0.01 0.02 495 24 contactcontact 0.00 0.00 0.00 136 6 contactcorrespondence 0.00 0.00 0.00 101 0 contactmeet 0.24 0.06 0.10 115 29 justicearrestjail 0.25 0.04 0.07 124 20 lifedie 0.42 0.15 0.23 207 77 lifeinjure 0.40 0.09 0.14 93 20 manufactureartifact 0.00 0.00 0.00 67 1 movementtransportartifact 0.00 0.00 0.00 59 1 movementtransportperson 0.23 0.01 0.03 341 22 personnelelect 0.39 0.11 0.18 141 41 personnelendposition 0.16 0.03 0.05 97 19 personnelstartposition 0.29 0.02 0.04 106 7 transactiontransaction 0.00 0.00 0.00 65 1 transactiontransfermoney 0.15 0.01 0.02 426 27 transactiontransferownership 0.11 0.01 0.01 191 9 Final Mention Detection Results: Micro Average Macro Average Attributes Prec Rec F1 Prec Rec F1 plain 58.60 7.13 12.71 50.72 7.98 13.80 mention_type 51.08 6.21 11.08 44.55 6.64 11.55 realis_status 34.29 4.17 7.44 28.48 4.87 8.31 mention_type+realis_status 29.50 3.59 6.40 24.85 4.10 7.04 Final Mention Coreference Results: Metric : bcub Score 7.98 Metric : ceafe Score 10.23 Metric : ceafm Score 10.14 * Metric : muc Score 0.42 Metric : blanc Score 0.44 Overall Average CoNLL score 4.77 * Score not included for final CoNLL score. ************************************************************* Run ID: TinkerBell_KB_SPA_2 See full details of run under Cold Start KB: Composite KB evaluation results ==========Spanish========== Mention Type Results: Type Prec Rec F1 #Gold #Sys conflictattack 0.25 0.05 0.08 498 91 conflictdemonstrate 0.50 0.07 0.12 166 22 contactbroadcast 0.25 0.01 0.02 495 24 contactcontact 0.00 0.00 0.00 136 6 contactcorrespondence 0.00 0.00 0.00 101 0 contactmeet 0.24 0.06 0.10 115 29 justicearrestjail 0.25 0.04 0.07 124 20 lifedie 0.42 0.15 0.23 207 77 lifeinjure 0.40 0.09 0.14 93 20 manufactureartifact 0.00 0.00 0.00 67 1 movementtransportartifact 0.00 0.00 0.00 59 1 movementtransportperson 0.23 0.01 0.03 341 22 personnelelect 0.39 0.11 0.18 141 41 personnelendposition 0.16 0.03 0.05 97 19 personnelstartposition 0.29 0.02 0.04 106 7 transactiontransaction 0.00 0.00 0.00 65 1 transactiontransfermoney 0.15 0.01 0.02 426 27 transactiontransferownership 0.11 0.01 0.01 191 9 Final Mention Detection Results: Micro Average Macro Average Attributes Prec Rec F1 Prec Rec F1 plain 58.60 7.13 12.71 50.72 7.98 13.80 mention_type 51.08 6.21 11.08 44.55 6.64 11.55 realis_status 34.29 4.17 7.44 28.48 4.87 8.31 mention_type+realis_status 29.50 3.59 6.40 24.85 4.10 7.04 Final Mention Coreference Results: Metric : bcub Score 7.98 Metric : ceafe Score 10.23 Metric : ceafm Score 10.14 * Metric : muc Score 0.42 Metric : blanc Score 0.44 Overall Average CoNLL score 4.77 * Score not included for final CoNLL score. ************************************************************* Run ID: TinkerBell_KB_SPA_3 See full details of run under Cold Start KB: Composite KB evaluation results ==========Spanish========== Mention Type Results: Type Prec Rec F1 #Gold #Sys conflictattack 0.25 0.05 0.08 498 91 conflictdemonstrate 0.50 0.07 0.12 166 22 contactbroadcast 0.25 0.01 0.02 495 24 contactcontact 0.00 0.00 0.00 136 6 contactcorrespondence 0.00 0.00 0.00 101 0 contactmeet 0.24 0.06 0.10 115 29 justicearrestjail 0.25 0.04 0.07 124 20 lifedie 0.42 0.15 0.23 207 77 lifeinjure 0.40 0.09 0.14 93 20 manufactureartifact 0.00 0.00 0.00 67 1 movementtransportartifact 0.00 0.00 0.00 59 1 movementtransportperson 0.23 0.01 0.03 341 22 personnelelect 0.39 0.11 0.18 141 41 personnelendposition 0.16 0.03 0.05 97 19 personnelstartposition 0.29 0.02 0.04 106 7 transactiontransaction 0.00 0.00 0.00 65 1 transactiontransfermoney 0.15 0.01 0.02 426 27 transactiontransferownership 0.11 0.01 0.01 191 9 Final Mention Detection Results: Micro Average Macro Average Attributes Prec Rec F1 Prec Rec F1 plain 58.60 7.13 12.71 50.72 7.98 13.80 mention_type 51.08 6.21 11.08 44.55 6.64 11.55 realis_status 34.29 4.17 7.44 28.48 4.87 8.31 mention_type+realis_status 29.50 3.59 6.40 24.85 4.10 7.04 Final Mention Coreference Results: Metric : bcub Score 7.98 Metric : ceafe Score 10.23 Metric : ceafm Score 10.14 * Metric : muc Score 0.42 Metric : blanc Score 0.44 Overall Average CoNLL score 4.77 * Score not included for final CoNLL score. ************************************************************* Run ID: TinkerBell_KB_SPA_4 See full details of run under Cold Start KB: Composite KB evaluation results ==========Spanish========== Mention Type Results: Type Prec Rec F1 #Gold #Sys conflictattack 0.25 0.05 0.08 498 91 conflictdemonstrate 0.50 0.07 0.12 166 22 contactbroadcast 0.25 0.01 0.02 495 24 contactcontact 0.00 0.00 0.00 136 6 contactcorrespondence 0.00 0.00 0.00 101 0 contactmeet 0.24 0.06 0.10 115 29 justicearrestjail 0.25 0.04 0.07 124 20 lifedie 0.42 0.15 0.23 207 77 lifeinjure 0.40 0.09 0.14 93 20 manufactureartifact 0.00 0.00 0.00 67 1 movementtransportartifact 0.00 0.00 0.00 59 1 movementtransportperson 0.23 0.01 0.03 341 22 personnelelect 0.39 0.11 0.18 141 41 personnelendposition 0.16 0.03 0.05 97 19 personnelstartposition 0.29 0.02 0.04 106 7 transactiontransaction 0.00 0.00 0.00 65 1 transactiontransfermoney 0.15 0.01 0.02 426 27 transactiontransferownership 0.11 0.01 0.01 191 9 Final Mention Detection Results: Micro Average Macro Average Attributes Prec Rec F1 Prec Rec F1 plain 58.60 7.13 12.71 50.72 7.98 13.80 mention_type 51.08 6.21 11.08 44.55 6.64 11.55 realis_status 34.29 4.17 7.44 28.48 4.87 8.31 mention_type+realis_status 29.50 3.59 6.40 24.85 4.10 7.04 Final Mention Coreference Results: Metric : bcub Score 7.98 Metric : ceafe Score 10.23 Metric : ceafm Score 10.14 * Metric : muc Score 0.42 Metric : blanc Score 0.44 Overall Average CoNLL score 4.77 * Score not included for final CoNLL score. ************************************************************* Run ID: TinkerBell_KB_SPA_5 See full details of run under Cold Start KB: Composite KB evaluation results ==========Spanish========== Mention Type Results: Type Prec Rec F1 #Gold #Sys conflictattack 0.25 0.05 0.08 498 91 conflictdemonstrate 0.50 0.07 0.12 166 22 contactbroadcast 0.25 0.01 0.02 495 24 contactcontact 0.00 0.00 0.00 136 6 contactcorrespondence 0.00 0.00 0.00 101 0 contactmeet 0.24 0.06 0.10 115 29 justicearrestjail 0.25 0.04 0.07 124 20 lifedie 0.42 0.15 0.23 207 77 lifeinjure 0.40 0.09 0.14 93 20 manufactureartifact 0.00 0.00 0.00 67 1 movementtransportartifact 0.00 0.00 0.00 59 1 movementtransportperson 0.23 0.01 0.03 341 22 personnelelect 0.39 0.11 0.18 141 41 personnelendposition 0.16 0.03 0.05 97 19 personnelstartposition 0.29 0.02 0.04 106 7 transactiontransaction 0.00 0.00 0.00 65 1 transactiontransfermoney 0.15 0.01 0.02 426 27 transactiontransferownership 0.11 0.01 0.01 191 9 Final Mention Detection Results: Micro Average Macro Average Attributes Prec Rec F1 Prec Rec F1 plain 58.60 7.13 12.71 50.72 7.98 13.80 mention_type 51.08 6.21 11.08 44.55 6.64 11.55 realis_status 34.29 4.17 7.44 28.48 4.87 8.31 mention_type+realis_status 29.50 3.59 6.40 24.85 4.10 7.04 Final Mention Coreference Results: Metric : bcub Score 7.98 Metric : ceafe Score 10.23 Metric : ceafm Score 10.14 * Metric : muc Score 0.42 Metric : blanc Score 0.44 Overall Average CoNLL score 4.77 * Score not included for final CoNLL score. ************************************************************* Run ID: TinkerBell_KB_XLING_1 See full details of run under Cold Start KB: Composite KB evaluation results ==========English========== Mention Type Results: Type Prec Rec F1 #Gold #Sys conflictattack 0.35 0.11 0.17 410 135 conflictdemonstrate 0.32 0.07 0.11 209 44 contactbroadcast 0.30 0.03 0.05 639 54 contactcontact 0.07 0.01 0.01 238 30 contactcorrespondence 0.25 0.01 0.01 198 4 contactmeet 0.38 0.14 0.21 200 76 justicearrestjail 0.67 0.34 0.45 104 52 lifedie 0.59 0.39 0.47 218 143 lifeinjure 0.83 0.38 0.52 53 24 manufactureartifact 0.29 0.01 0.03 141 7 movementtransportartifact 0.14 0.02 0.03 242 28 movementtransportperson 0.18 0.02 0.03 447 44 personnelelect 0.48 0.28 0.36 152 89 personnelendposition 0.52 0.08 0.14 186 29 personnelstartposition 0.37 0.09 0.15 116 29 transactiontransaction 0.00 0.00 0.00 51 3 transactiontransfermoney 0.41 0.07 0.12 510 90 transactiontransferownership 0.18 0.03 0.05 261 45 Final Mention Detection Results: Micro Average Macro Average Attributes Prec Rec F1 Prec Rec F1 plain 65.33 13.83 22.83 64.19 14.98 24.30 mention_type 56.93 12.05 19.89 57.68 13.23 21.53 realis_status 46.70 9.89 16.32 48.50 10.84 17.73 mention_type+realis_status 40.68 8.61 14.21 44.01 9.59 15.75 Final Mention Coreference Results: Metric : bcub Score 14.80 Metric : ceafe Score 15.38 Metric : ceafm Score 15.69 * Metric : muc Score 7.98 Metric : blanc Score 3.88 Overall Average CoNLL score 10.51 * Score not included for final CoNLL score. ==========Chinese========== Mention Type Results: Type Prec Rec F1 #Gold #Sys conflictattack 0.33 0.26 0.29 820 646 conflictdemonstrate 0.48 0.13 0.20 101 27 contactbroadcast 0.67 0.33 0.44 775 387 contactcontact 0.00 0.00 0.00 62 0 contactcorrespondence 0.22 0.08 0.12 25 9 contactmeet 0.24 0.20 0.22 126 104 justicearrestjail 0.44 0.34 0.38 109 85 lifedie 0.60 0.48 0.53 298 236 lifeinjure 0.67 0.69 0.68 93 96 manufactureartifact 0.21 0.06 0.09 131 36 movementtransportartifact 0.00 0.00 0.00 79 5 movementtransportperson 0.25 0.16 0.20 360 225 personnelelect 0.37 0.19 0.25 176 89 personnelendposition 0.50 0.12 0.19 108 26 personnelstartposition 0.09 0.01 0.02 78 11 transactiontransaction 0.00 0.00 0.00 107 12 transactiontransfermoney 0.12 0.03 0.04 264 54 transactiontransferownership 0.20 0.08 0.11 172 66 Final Mention Detection Results: Micro Average Macro Average Attributes Prec Rec F1 Prec Rec F1 plain 70.60 38.42 49.76 67.36 37.95 48.55 mention_type 65.89 35.86 46.45 61.83 35.28 44.92 realis_status 44.40 24.16 31.29 41.34 24.55 30.80 mention_type+realis_status 41.86 22.78 29.50 38.71 23.11 28.94 Final Mention Coreference Results: Metric : bcub Score 31.35 Metric : ceafe Score 31.14 Metric : ceafm Score 30.87 * Metric : muc Score 15.42 Metric : blanc Score 12.94 Overall Average CoNLL score 22.71 * Score not included for final CoNLL score. ==========Spanish========== Mention Type Results: Type Prec Rec F1 #Gold #Sys conflictattack 0.25 0.05 0.08 498 91 conflictdemonstrate 0.50 0.07 0.12 166 22 contactbroadcast 0.25 0.01 0.02 495 24 contactcontact 0.00 0.00 0.00 136 6 contactcorrespondence 0.00 0.00 0.00 101 0 contactmeet 0.24 0.06 0.10 115 29 justicearrestjail 0.25 0.04 0.07 124 20 lifedie 0.42 0.15 0.23 207 77 lifeinjure 0.40 0.09 0.14 93 20 manufactureartifact 0.00 0.00 0.00 67 1 movementtransportartifact 0.00 0.00 0.00 59 1 movementtransportperson 0.23 0.01 0.03 341 22 personnelelect 0.39 0.11 0.18 141 41 personnelendposition 0.16 0.03 0.05 97 19 personnelstartposition 0.29 0.02 0.04 106 7 transactiontransaction 0.00 0.00 0.00 65 1 transactiontransfermoney 0.15 0.01 0.02 426 27 transactiontransferownership 0.11 0.01 0.01 191 9 Final Mention Detection Results: Micro Average Macro Average Attributes Prec Rec F1 Prec Rec F1 plain 58.60 7.13 12.71 50.72 7.98 13.80 mention_type 51.08 6.21 11.08 44.55 6.64 11.55 realis_status 34.29 4.17 7.44 28.48 4.87 8.31 mention_type+realis_status 29.50 3.59 6.40 24.85 4.10 7.04 Final Mention Coreference Results: Metric : bcub Score 7.98 Metric : ceafe Score 10.23 Metric : ceafm Score 10.14 * Metric : muc Score 0.42 Metric : blanc Score 0.44 Overall Average CoNLL score 4.77 * Score not included for final CoNLL score. ************************************************************* Run ID: TinkerBell_KB_XLING_2 See full details of run under Cold Start KB: Composite KB evaluation results ==========English========== Mention Type Results: Type Prec Rec F1 #Gold #Sys conflictattack 0.35 0.11 0.17 410 135 conflictdemonstrate 0.32 0.07 0.11 209 44 contactbroadcast 0.30 0.03 0.05 639 54 contactcontact 0.07 0.01 0.01 238 30 contactcorrespondence 0.25 0.01 0.01 198 4 contactmeet 0.38 0.14 0.21 200 76 justicearrestjail 0.67 0.34 0.45 104 52 lifedie 0.59 0.39 0.47 218 143 lifeinjure 0.83 0.38 0.52 53 24 manufactureartifact 0.29 0.01 0.03 141 7 movementtransportartifact 0.14 0.02 0.03 242 28 movementtransportperson 0.18 0.02 0.03 447 44 personnelelect 0.48 0.28 0.36 152 89 personnelendposition 0.52 0.08 0.14 186 29 personnelstartposition 0.37 0.09 0.15 116 29 transactiontransaction 0.00 0.00 0.00 51 3 transactiontransfermoney 0.41 0.07 0.12 510 90 transactiontransferownership 0.18 0.03 0.05 261 45 Final Mention Detection Results: Micro Average Macro Average Attributes Prec Rec F1 Prec Rec F1 plain 65.33 13.83 22.83 64.19 14.98 24.30 mention_type 56.93 12.05 19.89 57.68 13.23 21.53 realis_status 46.70 9.89 16.32 48.50 10.84 17.73 mention_type+realis_status 40.68 8.61 14.21 44.01 9.59 15.75 Final Mention Coreference Results: Metric : bcub Score 14.80 Metric : ceafe Score 15.38 Metric : ceafm Score 15.69 * Metric : muc Score 7.98 Metric : blanc Score 3.88 Overall Average CoNLL score 10.51 * Score not included for final CoNLL score. ==========Chinese========== Mention Type Results: Type Prec Rec F1 #Gold #Sys conflictattack 0.33 0.26 0.29 820 646 conflictdemonstrate 0.48 0.13 0.20 101 27 contactbroadcast 0.67 0.33 0.44 775 387 contactcontact 0.00 0.00 0.00 62 0 contactcorrespondence 0.22 0.08 0.12 25 9 contactmeet 0.24 0.20 0.22 126 104 justicearrestjail 0.44 0.34 0.38 109 85 lifedie 0.60 0.48 0.53 298 236 lifeinjure 0.67 0.69 0.68 93 96 manufactureartifact 0.21 0.06 0.09 131 36 movementtransportartifact 0.00 0.00 0.00 79 5 movementtransportperson 0.25 0.16 0.20 360 225 personnelelect 0.37 0.19 0.25 176 89 personnelendposition 0.50 0.12 0.19 108 26 personnelstartposition 0.09 0.01 0.02 78 11 transactiontransaction 0.00 0.00 0.00 107 12 transactiontransfermoney 0.12 0.03 0.04 264 54 transactiontransferownership 0.20 0.08 0.11 172 66 Final Mention Detection Results: Micro Average Macro Average Attributes Prec Rec F1 Prec Rec F1 plain 70.60 38.42 49.76 67.36 37.95 48.55 mention_type 65.89 35.86 46.45 61.83 35.28 44.92 realis_status 44.40 24.16 31.29 41.34 24.55 30.80 mention_type+realis_status 41.86 22.78 29.50 38.71 23.11 28.94 Final Mention Coreference Results: Metric : bcub Score 31.35 Metric : ceafe Score 31.14 Metric : ceafm Score 30.87 * Metric : muc Score 15.42 Metric : blanc Score 12.94 Overall Average CoNLL score 22.71 * Score not included for final CoNLL score. ==========Spanish========== Mention Type Results: Type Prec Rec F1 #Gold #Sys conflictattack 0.25 0.05 0.08 498 91 conflictdemonstrate 0.50 0.07 0.12 166 22 contactbroadcast 0.25 0.01 0.02 495 24 contactcontact 0.00 0.00 0.00 136 6 contactcorrespondence 0.00 0.00 0.00 101 0 contactmeet 0.24 0.06 0.10 115 29 justicearrestjail 0.25 0.04 0.07 124 20 lifedie 0.42 0.15 0.23 207 77 lifeinjure 0.40 0.09 0.14 93 20 manufactureartifact 0.00 0.00 0.00 67 1 movementtransportartifact 0.00 0.00 0.00 59 1 movementtransportperson 0.23 0.01 0.03 341 22 personnelelect 0.39 0.11 0.18 141 41 personnelendposition 0.16 0.03 0.05 97 19 personnelstartposition 0.29 0.02 0.04 106 7 transactiontransaction 0.00 0.00 0.00 65 1 transactiontransfermoney 0.15 0.01 0.02 426 27 transactiontransferownership 0.11 0.01 0.01 191 9 Final Mention Detection Results: Micro Average Macro Average Attributes Prec Rec F1 Prec Rec F1 plain 58.60 7.13 12.71 50.72 7.98 13.80 mention_type 51.08 6.21 11.08 44.55 6.64 11.55 realis_status 34.29 4.17 7.44 28.48 4.87 8.31 mention_type+realis_status 29.50 3.59 6.40 24.85 4.10 7.04 Final Mention Coreference Results: Metric : bcub Score 7.98 Metric : ceafe Score 10.23 Metric : ceafm Score 10.14 * Metric : muc Score 0.42 Metric : blanc Score 0.44 Overall Average CoNLL score 4.77 * Score not included for final CoNLL score. ************************************************************* Run ID: TinkerBell_KB_XLING_3 See full details of run under Cold Start KB: Composite KB evaluation results ==========English========== Mention Type Results: Type Prec Rec F1 #Gold #Sys conflictattack 0.35 0.11 0.17 410 135 conflictdemonstrate 0.32 0.07 0.11 209 44 contactbroadcast 0.30 0.03 0.05 639 54 contactcontact 0.07 0.01 0.01 238 30 contactcorrespondence 0.25 0.01 0.01 198 4 contactmeet 0.38 0.14 0.21 200 76 justicearrestjail 0.67 0.34 0.45 104 52 lifedie 0.59 0.39 0.47 218 143 lifeinjure 0.83 0.38 0.52 53 24 manufactureartifact 0.29 0.01 0.03 141 7 movementtransportartifact 0.14 0.02 0.03 242 28 movementtransportperson 0.18 0.02 0.03 447 44 personnelelect 0.48 0.28 0.36 152 89 personnelendposition 0.52 0.08 0.14 186 29 personnelstartposition 0.37 0.09 0.15 116 29 transactiontransaction 0.00 0.00 0.00 51 3 transactiontransfermoney 0.41 0.07 0.12 510 90 transactiontransferownership 0.18 0.03 0.05 261 45 Final Mention Detection Results: Micro Average Macro Average Attributes Prec Rec F1 Prec Rec F1 plain 65.33 13.83 22.83 64.19 14.98 24.30 mention_type 56.93 12.05 19.89 57.68 13.23 21.53 realis_status 46.70 9.89 16.32 48.50 10.84 17.73 mention_type+realis_status 40.68 8.61 14.21 44.01 9.59 15.75 Final Mention Coreference Results: Metric : bcub Score 14.80 Metric : ceafe Score 15.38 Metric : ceafm Score 15.69 * Metric : muc Score 7.98 Metric : blanc Score 3.88 Overall Average CoNLL score 10.51 * Score not included for final CoNLL score. ==========Chinese========== Mention Type Results: Type Prec Rec F1 #Gold #Sys conflictattack 0.33 0.26 0.29 820 646 conflictdemonstrate 0.48 0.13 0.20 101 27 contactbroadcast 0.67 0.33 0.44 775 387 contactcontact 0.00 0.00 0.00 62 0 contactcorrespondence 0.22 0.08 0.12 25 9 contactmeet 0.24 0.20 0.22 126 104 justicearrestjail 0.44 0.34 0.38 109 85 lifedie 0.60 0.48 0.53 298 236 lifeinjure 0.67 0.69 0.68 93 96 manufactureartifact 0.21 0.06 0.09 131 36 movementtransportartifact 0.00 0.00 0.00 79 5 movementtransportperson 0.25 0.16 0.20 360 225 personnelelect 0.37 0.19 0.25 176 89 personnelendposition 0.50 0.12 0.19 108 26 personnelstartposition 0.09 0.01 0.02 78 11 transactiontransaction 0.00 0.00 0.00 107 12 transactiontransfermoney 0.12 0.03 0.04 264 54 transactiontransferownership 0.20 0.08 0.11 172 66 Final Mention Detection Results: Micro Average Macro Average Attributes Prec Rec F1 Prec Rec F1 plain 70.60 38.42 49.76 67.36 37.95 48.55 mention_type 65.89 35.86 46.45 61.83 35.28 44.92 realis_status 44.40 24.16 31.29 41.34 24.55 30.80 mention_type+realis_status 41.86 22.78 29.50 38.71 23.11 28.94 Final Mention Coreference Results: Metric : bcub Score 31.35 Metric : ceafe Score 31.14 Metric : ceafm Score 30.87 * Metric : muc Score 15.42 Metric : blanc Score 12.94 Overall Average CoNLL score 22.71 * Score not included for final CoNLL score. ==========Spanish========== Mention Type Results: Type Prec Rec F1 #Gold #Sys conflictattack 0.25 0.05 0.08 498 91 conflictdemonstrate 0.50 0.07 0.12 166 22 contactbroadcast 0.25 0.01 0.02 495 24 contactcontact 0.00 0.00 0.00 136 6 contactcorrespondence 0.00 0.00 0.00 101 0 contactmeet 0.24 0.06 0.10 115 29 justicearrestjail 0.25 0.04 0.07 124 20 lifedie 0.42 0.15 0.23 207 77 lifeinjure 0.40 0.09 0.14 93 20 manufactureartifact 0.00 0.00 0.00 67 1 movementtransportartifact 0.00 0.00 0.00 59 1 movementtransportperson 0.23 0.01 0.03 341 22 personnelelect 0.39 0.11 0.18 141 41 personnelendposition 0.16 0.03 0.05 97 19 personnelstartposition 0.29 0.02 0.04 106 7 transactiontransaction 0.00 0.00 0.00 65 1 transactiontransfermoney 0.15 0.01 0.02 426 27 transactiontransferownership 0.11 0.01 0.01 191 9 Final Mention Detection Results: Micro Average Macro Average Attributes Prec Rec F1 Prec Rec F1 plain 58.60 7.13 12.71 50.72 7.98 13.80 mention_type 51.08 6.21 11.08 44.55 6.64 11.55 realis_status 34.29 4.17 7.44 28.48 4.87 8.31 mention_type+realis_status 29.50 3.59 6.40 24.85 4.10 7.04 Final Mention Coreference Results: Metric : bcub Score 7.98 Metric : ceafe Score 10.23 Metric : ceafm Score 10.14 * Metric : muc Score 0.42 Metric : blanc Score 0.44 Overall Average CoNLL score 4.77 * Score not included for final CoNLL score. ************************************************************* Run ID: TinkerBell_KB_XLING_4 See full details of run under Cold Start KB: Composite KB evaluation results ==========English========== Mention Type Results: Type Prec Rec F1 #Gold #Sys conflictattack 0.35 0.11 0.17 410 135 conflictdemonstrate 0.32 0.07 0.11 209 44 contactbroadcast 0.30 0.03 0.05 639 54 contactcontact 0.07 0.01 0.01 238 30 contactcorrespondence 0.25 0.01 0.01 198 4 contactmeet 0.38 0.14 0.21 200 76 justicearrestjail 0.67 0.34 0.45 104 52 lifedie 0.59 0.39 0.47 218 143 lifeinjure 0.83 0.38 0.52 53 24 manufactureartifact 0.29 0.01 0.03 141 7 movementtransportartifact 0.14 0.02 0.03 242 28 movementtransportperson 0.18 0.02 0.03 447 44 personnelelect 0.48 0.28 0.36 152 89 personnelendposition 0.52 0.08 0.14 186 29 personnelstartposition 0.37 0.09 0.15 116 29 transactiontransaction 0.00 0.00 0.00 51 3 transactiontransfermoney 0.41 0.07 0.12 510 90 transactiontransferownership 0.18 0.03 0.05 261 45 Final Mention Detection Results: Micro Average Macro Average Attributes Prec Rec F1 Prec Rec F1 plain 65.33 13.83 22.83 64.19 14.98 24.30 mention_type 56.93 12.05 19.89 57.68 13.23 21.53 realis_status 46.70 9.89 16.32 48.50 10.84 17.73 mention_type+realis_status 40.68 8.61 14.21 44.01 9.59 15.75 Final Mention Coreference Results: Metric : bcub Score 14.80 Metric : ceafe Score 15.38 Metric : ceafm Score 15.69 * Metric : muc Score 7.98 Metric : blanc Score 3.88 Overall Average CoNLL score 10.51 * Score not included for final CoNLL score. ==========Chinese========== Mention Type Results: Type Prec Rec F1 #Gold #Sys conflictattack 0.33 0.26 0.29 820 646 conflictdemonstrate 0.48 0.13 0.20 101 27 contactbroadcast 0.67 0.33 0.44 775 387 contactcontact 0.00 0.00 0.00 62 0 contactcorrespondence 0.22 0.08 0.12 25 9 contactmeet 0.24 0.20 0.22 126 104 justicearrestjail 0.44 0.34 0.38 109 85 lifedie 0.60 0.48 0.53 298 236 lifeinjure 0.67 0.69 0.68 93 96 manufactureartifact 0.21 0.06 0.09 131 36 movementtransportartifact 0.00 0.00 0.00 79 5 movementtransportperson 0.25 0.16 0.20 360 225 personnelelect 0.37 0.19 0.25 176 89 personnelendposition 0.50 0.12 0.19 108 26 personnelstartposition 0.09 0.01 0.02 78 11 transactiontransaction 0.00 0.00 0.00 107 12 transactiontransfermoney 0.12 0.03 0.04 264 54 transactiontransferownership 0.20 0.08 0.11 172 66 Final Mention Detection Results: Micro Average Macro Average Attributes Prec Rec F1 Prec Rec F1 plain 70.60 38.42 49.76 67.36 37.95 48.55 mention_type 65.89 35.86 46.45 61.83 35.28 44.92 realis_status 44.40 24.16 31.29 41.34 24.55 30.80 mention_type+realis_status 41.86 22.78 29.50 38.71 23.11 28.94 Final Mention Coreference Results: Metric : bcub Score 31.35 Metric : ceafe Score 31.14 Metric : ceafm Score 30.87 * Metric : muc Score 15.42 Metric : blanc Score 12.94 Overall Average CoNLL score 22.71 * Score not included for final CoNLL score. ==========Spanish========== Mention Type Results: Type Prec Rec F1 #Gold #Sys conflictattack 0.25 0.05 0.08 498 91 conflictdemonstrate 0.50 0.07 0.12 166 22 contactbroadcast 0.25 0.01 0.02 495 24 contactcontact 0.00 0.00 0.00 136 6 contactcorrespondence 0.00 0.00 0.00 101 0 contactmeet 0.24 0.06 0.10 115 29 justicearrestjail 0.25 0.04 0.07 124 20 lifedie 0.42 0.15 0.23 207 77 lifeinjure 0.40 0.09 0.14 93 20 manufactureartifact 0.00 0.00 0.00 67 1 movementtransportartifact 0.00 0.00 0.00 59 1 movementtransportperson 0.23 0.01 0.03 341 22 personnelelect 0.39 0.11 0.18 141 41 personnelendposition 0.16 0.03 0.05 97 19 personnelstartposition 0.29 0.02 0.04 106 7 transactiontransaction 0.00 0.00 0.00 65 1 transactiontransfermoney 0.15 0.01 0.02 426 27 transactiontransferownership 0.11 0.01 0.01 191 9 Final Mention Detection Results: Micro Average Macro Average Attributes Prec Rec F1 Prec Rec F1 plain 58.60 7.13 12.71 50.72 7.98 13.80 mention_type 51.08 6.21 11.08 44.55 6.64 11.55 realis_status 34.29 4.17 7.44 28.48 4.87 8.31 mention_type+realis_status 29.50 3.59 6.40 24.85 4.10 7.04 Final Mention Coreference Results: Metric : bcub Score 7.98 Metric : ceafe Score 10.23 Metric : ceafm Score 10.14 * Metric : muc Score 0.42 Metric : blanc Score 0.44 Overall Average CoNLL score 4.77 * Score not included for final CoNLL score. ************************************************************* Run ID: TinkerBell_KB_XLING_5 See full details of run under Cold Start KB: Composite KB evaluation results ==========English========== Mention Type Results: Type Prec Rec F1 #Gold #Sys conflictattack 0.35 0.11 0.17 410 135 conflictdemonstrate 0.32 0.07 0.11 209 44 contactbroadcast 0.30 0.03 0.05 639 54 contactcontact 0.07 0.01 0.01 238 30 contactcorrespondence 0.25 0.01 0.01 198 4 contactmeet 0.38 0.14 0.21 200 76 justicearrestjail 0.67 0.34 0.45 104 52 lifedie 0.59 0.39 0.47 218 143 lifeinjure 0.83 0.38 0.52 53 24 manufactureartifact 0.29 0.01 0.03 141 7 movementtransportartifact 0.14 0.02 0.03 242 28 movementtransportperson 0.18 0.02 0.03 447 44 personnelelect 0.48 0.28 0.36 152 89 personnelendposition 0.52 0.08 0.14 186 29 personnelstartposition 0.37 0.09 0.15 116 29 transactiontransaction 0.00 0.00 0.00 51 3 transactiontransfermoney 0.41 0.07 0.12 510 90 transactiontransferownership 0.18 0.03 0.05 261 45 Final Mention Detection Results: Micro Average Macro Average Attributes Prec Rec F1 Prec Rec F1 plain 65.33 13.83 22.83 64.19 14.98 24.30 mention_type 56.93 12.05 19.89 57.68 13.23 21.53 realis_status 46.70 9.89 16.32 48.50 10.84 17.73 mention_type+realis_status 40.68 8.61 14.21 44.01 9.59 15.75 Final Mention Coreference Results: Metric : bcub Score 14.80 Metric : ceafe Score 15.38 Metric : ceafm Score 15.69 * Metric : muc Score 7.98 Metric : blanc Score 3.88 Overall Average CoNLL score 10.51 * Score not included for final CoNLL score. ==========Chinese========== Mention Type Results: Type Prec Rec F1 #Gold #Sys conflictattack 0.33 0.26 0.29 820 646 conflictdemonstrate 0.48 0.13 0.20 101 27 contactbroadcast 0.67 0.33 0.44 775 387 contactcontact 0.00 0.00 0.00 62 0 contactcorrespondence 0.22 0.08 0.12 25 9 contactmeet 0.24 0.20 0.22 126 104 justicearrestjail 0.44 0.34 0.38 109 85 lifedie 0.60 0.48 0.53 298 236 lifeinjure 0.67 0.69 0.68 93 96 manufactureartifact 0.21 0.06 0.09 131 36 movementtransportartifact 0.00 0.00 0.00 79 5 movementtransportperson 0.25 0.16 0.20 360 225 personnelelect 0.37 0.19 0.25 176 89 personnelendposition 0.50 0.12 0.19 108 26 personnelstartposition 0.09 0.01 0.02 78 11 transactiontransaction 0.00 0.00 0.00 107 12 transactiontransfermoney 0.12 0.03 0.04 264 54 transactiontransferownership 0.20 0.08 0.11 172 66 Final Mention Detection Results: Micro Average Macro Average Attributes Prec Rec F1 Prec Rec F1 plain 70.60 38.42 49.76 67.36 37.95 48.55 mention_type 65.89 35.86 46.45 61.83 35.28 44.92 realis_status 44.40 24.16 31.29 41.34 24.55 30.80 mention_type+realis_status 41.86 22.78 29.50 38.71 23.11 28.94 Final Mention Coreference Results: Metric : bcub Score 31.35 Metric : ceafe Score 31.14 Metric : ceafm Score 30.87 * Metric : muc Score 15.42 Metric : blanc Score 12.94 Overall Average CoNLL score 22.71 * Score not included for final CoNLL score. ==========Spanish========== Mention Type Results: Type Prec Rec F1 #Gold #Sys conflictattack 0.25 0.05 0.08 498 91 conflictdemonstrate 0.50 0.07 0.12 166 22 contactbroadcast 0.25 0.01 0.02 495 24 contactcontact 0.00 0.00 0.00 136 6 contactcorrespondence 0.00 0.00 0.00 101 0 contactmeet 0.24 0.06 0.10 115 29 justicearrestjail 0.25 0.04 0.07 124 20 lifedie 0.42 0.15 0.23 207 77 lifeinjure 0.40 0.09 0.14 93 20 manufactureartifact 0.00 0.00 0.00 67 1 movementtransportartifact 0.00 0.00 0.00 59 1 movementtransportperson 0.23 0.01 0.03 341 22 personnelelect 0.39 0.11 0.18 141 41 personnelendposition 0.16 0.03 0.05 97 19 personnelstartposition 0.29 0.02 0.04 106 7 transactiontransaction 0.00 0.00 0.00 65 1 transactiontransfermoney 0.15 0.01 0.02 426 27 transactiontransferownership 0.11 0.01 0.01 191 9 Final Mention Detection Results: Micro Average Macro Average Attributes Prec Rec F1 Prec Rec F1 plain 58.60 7.13 12.71 50.72 7.98 13.80 mention_type 51.08 6.21 11.08 44.55 6.64 11.55 realis_status 34.29 4.17 7.44 28.48 4.87 8.31 mention_type+realis_status 29.50 3.59 6.40 24.85 4.10 7.04 Final Mention Coreference Results: Metric : bcub Score 7.98 Metric : ceafe Score 10.23 Metric : ceafm Score 10.14 * Metric : muc Score 0.42 Metric : blanc Score 0.44 Overall Average CoNLL score 4.77 * Score not included for final CoNLL score.