================================================================================================= TAC KBP 2017 KB CONSTRUCTION: COMPONENT EVENT NUGGET DETECTION AND COREFERENCE EVALUATION RESULTS ================================================================================================= Team ID: SAFT_ISI Organization: USC Information Sciences Institute ************************************************************* Run ID: SAFT_ISI_KB_CMN_1 See full details of run under Cold Start KB: Composite KB evaluation results ==========Chinese========== Mention Type Results: Type Prec Rec F1 #Gold #Sys conflictattack 0.31 0.11 0.16 820 288 conflictdemonstrate 0.48 0.44 0.46 101 93 contactbroadcast 0.26 0.19 0.22 775 553 contactcontact 0.00 0.01 0.00 62 143 contactcorrespondence 0.06 0.08 0.07 25 31 contactmeet 0.15 0.15 0.15 126 127 justicearrestjail 0.36 0.38 0.37 109 116 lifedie 0.36 0.34 0.35 298 281 lifeinjure 0.49 0.36 0.41 93 68 manufactureartifact 0.00 0.00 0.00 131 0 movementtransportartifact 0.17 0.01 0.02 79 4 movementtransportperson 0.24 0.16 0.19 360 234 personnelelect 0.33 0.11 0.16 176 57 personnelendposition 0.30 0.21 0.24 108 73 personnelstartposition 0.17 0.06 0.09 78 27 transactiontransaction 0.00 0.00 0.00 107 15 transactiontransfermoney 0.07 0.05 0.05 264 177 transactiontransferownership 0.07 0.01 0.02 172 30 Final Mention Detection Results: Micro Average Macro Average Attributes Prec Rec F1 Prec Rec F1 plain 38.88 23.20 29.06 37.62 23.32 28.79 mention_type 31.69 18.90 23.68 30.27 18.99 23.34 realis_status 30.74 18.34 22.98 28.79 18.16 22.27 mention_type+realis_status 25.53 15.23 19.08 23.49 15.13 18.40 Final Mention Coreference Results: Metric : bcub Score 13.05 Metric : ceafe Score 13.31 Metric : ceafm Score 13.48 * Metric : muc Score 5.45 Metric : blanc Score 3.01 Overall Average CoNLL score 8.71 * Score not included for final CoNLL score. ************************************************************* Run ID: SAFT_ISI_KB_ENG_1 See full details of run under Cold Start KB: Composite KB evaluation results ==========English========== Mention Type Results: Type Prec Rec F1 #Gold #Sys conflictattack 0.22 0.32 0.26 410 606 conflictdemonstrate 0.36 0.45 0.40 209 262 contactbroadcast 0.19 0.26 0.22 639 860 contactcontact 0.11 0.23 0.15 238 506 contactcorrespondence 0.10 0.16 0.12 198 333 contactmeet 0.14 0.21 0.17 200 293 justicearrestjail 0.23 0.59 0.33 104 268 lifedie 0.31 0.61 0.41 218 432 lifeinjure 0.28 0.60 0.38 53 115 manufactureartifact 0.09 0.18 0.12 141 277 movementtransportartifact 0.06 0.11 0.08 242 475 movementtransportperson 0.10 0.24 0.14 447 1029 personnelelect 0.23 0.36 0.28 152 235 personnelendposition 0.25 0.36 0.29 186 272 personnelstartposition 0.09 0.15 0.11 116 200 transactiontransaction 0.01 0.08 0.02 51 322 transactiontransfermoney 0.13 0.19 0.15 510 740 transactiontransferownership 0.22 0.32 0.26 261 380 Final Mention Detection Results: Micro Average Macro Average Attributes Prec Rec F1 Prec Rec F1 plain 28.23 49.08 35.85 29.16 50.37 36.94 mention_type 22.43 38.99 28.48 23.46 40.91 29.82 realis_status 19.80 34.42 25.14 21.10 36.34 26.70 mention_type+realis_status 16.13 28.03 20.47 17.40 30.16 22.07 Final Mention Coreference Results: Metric : bcub Score 20.09 Metric : ceafe Score 20.16 Metric : ceafm Score 21.53 * Metric : muc Score 8.16 Metric : blanc Score 5.84 Overall Average CoNLL score 13.56 * Score not included for final CoNLL score. ************************************************************* Run ID: SAFT_ISI_KB_SPA_1 See full details of run under Cold Start KB: Composite KB evaluation results ==========Spanish========== Mention Type Results: Type Prec Rec F1 #Gold #Sys conflictattack 0.20 0.03 0.06 498 84 conflictdemonstrate 0.00 0.00 0.00 166 0 contactbroadcast 0.16 0.27 0.20 495 830 contactcontact 0.00 0.00 0.00 136 3 contactcorrespondence 0.00 0.00 0.00 101 2 contactmeet 0.00 0.00 0.00 115 5 justicearrestjail 0.17 0.01 0.02 124 6 lifedie 0.27 0.29 0.28 207 217 lifeinjure 0.00 0.00 0.00 93 0 manufactureartifact 0.00 0.00 0.00 67 0 movementtransportartifact 0.00 0.00 0.00 59 15 movementtransportperson 0.12 0.01 0.02 341 39 personnelelect 0.08 0.12 0.10 141 205 personnelendposition 0.00 0.00 0.00 97 1 personnelstartposition 0.14 0.01 0.02 106 7 transactiontransaction 0.00 0.00 0.00 65 0 transactiontransfermoney 0.02 0.00 0.00 426 41 transactiontransferownership 0.09 0.01 0.01 191 11 Final Mention Detection Results: Micro Average Macro Average Attributes Prec Rec F1 Prec Rec F1 plain 28.24 12.08 16.92 28.97 12.69 17.65 mention_type 19.47 8.33 11.66 20.28 9.12 12.58 realis_status 20.75 8.87 12.43 21.00 9.82 13.39 mention_type+realis_status 15.93 6.81 9.54 16.12 7.72 10.44 Final Mention Coreference Results: Metric : bcub Score 9.37 Metric : ceafe Score 11.22 Metric : ceafm Score 10.29 * Metric : muc Score 0.00 Metric : blanc Score 0.67 Overall Average CoNLL score 5.32 * Score not included for final CoNLL score.