======================================================= TAC KBP 2016 SPANISH KB CONSTRUCTION EVALUATION RESULTS ======================================================= Team ID: hltcoe Organization: Human Language Technology Center of Excellence ************************************************************* Run ID: hltcoe_KB_SPA_1 Did the run access the live Web during the evaluation window: No Did the run extract relations from the Cold Start source corpus: Yes Did the run generate meaningful confidence values: No Entity Discovery Evaluation: ONLY Spanish documents: Prec Recall F1 Metric 0.771 0.467 0.581 strong_mention_match 0.716 0.433 0.540 strong_typed_mention_match 0.000 0.000 0.000 entity_match 0.653 0.289 0.401 b_cubed 0.662 0.401 0.499 mention_ceaf 0.632 0.383 0.477 typed_mention_ceaf Slot Filling Evaluation: Metric RunID Hop Prec Recall F1 SF-ALL-Micro hltcoe_KB_SPA_1 0 0.3077 0.0120 0.0232 SF-ALL-Micro hltcoe_KB_SPA_1 1 0.0000 0.0000 0.0000 SF-ALL-Micro hltcoe_KB_SPA_1 ALL 0.3077 0.0073 0.0143 SF-ALL-Macro hltcoe_KB_SPA_1 0 0.0064 0.0155 0.0089 SF-ALL-Macro hltcoe_KB_SPA_1 1 0.0000 0.0000 0.0000 SF-ALL-Macro hltcoe_KB_SPA_1 ALL 0.0043 0.0103 0.0060 LDC-MAX-ALL-Micro hltcoe_KB_SPA_1 0 0.2727 0.0120 0.0231 LDC-MAX-ALL-Micro hltcoe_KB_SPA_1 1 0.0000 0.0000 0.0000 LDC-MAX-ALL-Micro hltcoe_KB_SPA_1 ALL 0.2727 0.0080 0.0155 LDC-MAX-ALL-Macro hltcoe_KB_SPA_1 0 0.0054 0.0144 0.0077 LDC-MAX-ALL-Macro hltcoe_KB_SPA_1 1 0.0000 0.0000 0.0000 LDC-MAX-ALL-Macro hltcoe_KB_SPA_1 ALL 0.0036 0.0097 0.0052 LDC-MEAN-ALL-Macro hltcoe_KB_SPA_1 0 0.0054 0.0144 0.0077 LDC-MEAN-ALL-Macro hltcoe_KB_SPA_1 1 0.0000 0.0000 0.0000 LDC-MEAN-ALL-Macro hltcoe_KB_SPA_1 ALL 0.0036 0.0097 0.0052 *ALL-Macro Prec, Recall and F1 refer to mean-precision, mean-recall and mean-F1. NIL-DETECTION P/R/F1: 0.3138 1.0000 0.4777 ************************************************************* Run ID: hltcoe_KB_SPA_2 Did the run access the live Web during the evaluation window: No Did the run extract relations from the Cold Start source corpus: Yes Did the run generate meaningful confidence values: No Entity Discovery Evaluation: ONLY Spanish documents: Prec Recall F1 Metric 0.771 0.467 0.581 strong_mention_match 0.716 0.433 0.540 strong_typed_mention_match 0.000 0.000 0.000 entity_match 0.653 0.288 0.400 b_cubed 0.658 0.398 0.496 mention_ceaf 0.628 0.380 0.474 typed_mention_ceaf Slot Filling Evaluation: Metric RunID Hop Prec Recall F1 SF-ALL-Micro hltcoe_KB_SPA_2 0 0.3077 0.0120 0.0232 SF-ALL-Micro hltcoe_KB_SPA_2 1 0.0000 0.0000 0.0000 SF-ALL-Micro hltcoe_KB_SPA_2 ALL 0.3077 0.0073 0.0143 SF-ALL-Macro hltcoe_KB_SPA_2 0 0.0064 0.0155 0.0089 SF-ALL-Macro hltcoe_KB_SPA_2 1 0.0000 0.0000 0.0000 SF-ALL-Macro hltcoe_KB_SPA_2 ALL 0.0043 0.0103 0.0060 LDC-MAX-ALL-Micro hltcoe_KB_SPA_2 0 0.2727 0.0120 0.0231 LDC-MAX-ALL-Micro hltcoe_KB_SPA_2 1 0.0000 0.0000 0.0000 LDC-MAX-ALL-Micro hltcoe_KB_SPA_2 ALL 0.2727 0.0080 0.0155 LDC-MAX-ALL-Macro hltcoe_KB_SPA_2 0 0.0054 0.0144 0.0077 LDC-MAX-ALL-Macro hltcoe_KB_SPA_2 1 0.0000 0.0000 0.0000 LDC-MAX-ALL-Macro hltcoe_KB_SPA_2 ALL 0.0036 0.0097 0.0052 LDC-MEAN-ALL-Macro hltcoe_KB_SPA_2 0 0.0054 0.0144 0.0077 LDC-MEAN-ALL-Macro hltcoe_KB_SPA_2 1 0.0000 0.0000 0.0000 LDC-MEAN-ALL-Macro hltcoe_KB_SPA_2 ALL 0.0036 0.0097 0.0052 *ALL-Macro Prec, Recall and F1 refer to mean-precision, mean-recall and mean-F1. NIL-DETECTION P/R/F1: 0.3138 1.0000 0.4777 ************************************************************* Run ID: hltcoe_KB_SPA_3 Did the run access the live Web during the evaluation window: No Did the run extract relations from the Cold Start source corpus: Yes Did the run generate meaningful confidence values: No Entity Discovery Evaluation: ONLY Spanish documents: Prec Recall F1 Metric 0.771 0.467 0.581 strong_mention_match 0.716 0.433 0.540 strong_typed_mention_match 0.000 0.000 0.000 entity_match 0.668 0.244 0.358 b_cubed 0.551 0.334 0.416 mention_ceaf 0.525 0.318 0.396 typed_mention_ceaf Slot Filling Evaluation: Metric RunID Hop Prec Recall F1 SF-ALL-Micro hltcoe_KB_SPA_3 0 0.5000 0.0090 0.0178 SF-ALL-Micro hltcoe_KB_SPA_3 1 0.0000 0.0000 0.0000 SF-ALL-Micro hltcoe_KB_SPA_3 ALL 0.5000 0.0055 0.0109 SF-ALL-Macro hltcoe_KB_SPA_3 0 0.0103 0.0129 0.0112 SF-ALL-Macro hltcoe_KB_SPA_3 1 0.0000 0.0000 0.0000 SF-ALL-Macro hltcoe_KB_SPA_3 ALL 0.0069 0.0086 0.0074 LDC-MAX-ALL-Micro hltcoe_KB_SPA_3 0 0.5000 0.0080 0.0158 LDC-MAX-ALL-Micro hltcoe_KB_SPA_3 1 0.0000 0.0000 0.0000 LDC-MAX-ALL-Micro hltcoe_KB_SPA_3 ALL 0.5000 0.0053 0.0106 LDC-MAX-ALL-Macro hltcoe_KB_SPA_3 0 0.0108 0.0108 0.0108 LDC-MAX-ALL-Macro hltcoe_KB_SPA_3 1 0.0000 0.0000 0.0000 LDC-MAX-ALL-Macro hltcoe_KB_SPA_3 ALL 0.0072 0.0072 0.0072 LDC-MEAN-ALL-Macro hltcoe_KB_SPA_3 0 0.0090 0.0108 0.0096 LDC-MEAN-ALL-Macro hltcoe_KB_SPA_3 1 0.0000 0.0000 0.0000 LDC-MEAN-ALL-Macro hltcoe_KB_SPA_3 ALL 0.0060 0.0072 0.0064 *ALL-Macro Prec, Recall and F1 refer to mean-precision, mean-recall and mean-F1. NIL-DETECTION P/R/F1: 0.3138 1.0000 0.4777 ************************************************************* Run ID: hltcoe_KB_SPA_4 Did the run access the live Web during the evaluation window: No Did the run extract relations from the Cold Start source corpus: Yes Did the run generate meaningful confidence values: No Entity Discovery Evaluation: ONLY Spanish documents: Prec Recall F1 Metric 0.776 0.461 0.579 strong_mention_match 0.721 0.428 0.537 strong_typed_mention_match 0.000 0.000 0.000 entity_match 0.656 0.287 0.400 b_cubed 0.668 0.397 0.498 mention_ceaf 0.638 0.379 0.476 typed_mention_ceaf Slot Filling Evaluation: Metric RunID Hop Prec Recall F1 SF-ALL-Micro hltcoe_KB_SPA_4 0 0.3077 0.0120 0.0232 SF-ALL-Micro hltcoe_KB_SPA_4 1 0.0000 0.0000 0.0000 SF-ALL-Micro hltcoe_KB_SPA_4 ALL 0.3077 0.0073 0.0143 SF-ALL-Macro hltcoe_KB_SPA_4 0 0.0089 0.0155 0.0105 SF-ALL-Macro hltcoe_KB_SPA_4 1 0.0000 0.0000 0.0000 SF-ALL-Macro hltcoe_KB_SPA_4 ALL 0.0059 0.0103 0.0070 LDC-MAX-ALL-Micro hltcoe_KB_SPA_4 0 0.2727 0.0120 0.0231 LDC-MAX-ALL-Micro hltcoe_KB_SPA_4 1 0.0000 0.0000 0.0000 LDC-MAX-ALL-Micro hltcoe_KB_SPA_4 ALL 0.2727 0.0080 0.0155 LDC-MAX-ALL-Macro hltcoe_KB_SPA_4 0 0.0088 0.0144 0.0098 LDC-MAX-ALL-Macro hltcoe_KB_SPA_4 1 0.0000 0.0000 0.0000 LDC-MAX-ALL-Macro hltcoe_KB_SPA_4 ALL 0.0059 0.0097 0.0066 LDC-MEAN-ALL-Macro hltcoe_KB_SPA_4 0 0.0070 0.0144 0.0086 LDC-MEAN-ALL-Macro hltcoe_KB_SPA_4 1 0.0000 0.0000 0.0000 LDC-MEAN-ALL-Macro hltcoe_KB_SPA_4 ALL 0.0047 0.0097 0.0058 *ALL-Macro Prec, Recall and F1 refer to mean-precision, mean-recall and mean-F1. NIL-DETECTION P/R/F1: 0.3138 1.0000 0.4777