=========================================================== TAC KBP 2015 COLD START KB CONSTRUCTION EVALUATION RESULTS =========================================================== Team ID: ICTCAS_OKN Organization: Institute of Computing Technology, Chinese Academy of Sciences ************************************************************* Run ID: ICTCAS_OKN1 Did the run access the live Web during the evaluation window: No Run Number of most similar Cold Start Slot Filling task submission: NA Did the run extract relations from the Cold Start source corpus: Yes Did the run generate meaningful confidence values: No Entity Discovery Evaluation: Prec Recall F1 Metric 0.229 0.175 0.198 strong_mention_match 0.210 0.161 0.182 strong_typed_mention_match 0.000 0.000 0.000 entity_match 0.176 0.103 0.130 b_cubed 0.196 0.150 0.170 mention_ceaf 0.184 0.141 0.160 typed_mention_ceaf 0.184 0.141 0.160 typed_mention_ceaf_plus Slot Filling Evaluation: Metric Hop GoldT Submit Correct Incorrect Inexact PIncorrect Dup Right Wrong Prec Recall F1 CSSF micro 0 4840 69 26 38 5 0 0 26 43 0.3768 0.0054 0.0106 CSSF micro 1 3954 16 0 0 0 16 0 0 16 0.0000 0.0000 0.0000 CSSF micro ALL 8794 85 26 38 5 16 0 26 59 0.3059 0.0030 0.0059 LDC-MEAN macro 0 0.0096 LDC-MEAN macro 1 0.0000 LDC-MEAN macro ALL 0.0061 LDC-MAX micro 0 1268 31 12 17 2 0 0 12 19 0.3871 0.0095 0.0185 LDC-MAX micro 1 900 12 0 0 0 12 0 0 12 0.0000 0.0000 0.0000 LDC-MAX micro ALL 2168 43 12 17 2 12 0 12 31 0.2791 0.0055 0.0109 LDC-MAX macro 0 0.0162 LDC-MAX macro 1 0.0000 LDC-MAX macro ALL 0.0104 ************************************************************* Run ID: ICTCAS_OKN2 Did the run access the live Web during the evaluation window: No Run Number of most similar Cold Start Slot Filling task submission: NA Did the run extract relations from the Cold Start source corpus: Yes Did the run generate meaningful confidence values: No Entity Discovery Evaluation: Prec Recall F1 Metric 0.229 0.175 0.198 strong_mention_match 0.211 0.162 0.183 strong_typed_mention_match 0.000 0.000 0.000 entity_match 0.219 0.094 0.131 b_cubed 0.173 0.133 0.150 mention_ceaf 0.165 0.126 0.143 typed_mention_ceaf 0.165 0.126 0.143 typed_mention_ceaf_plus Slot Filling Evaluation: Metric Hop GoldT Submit Correct Incorrect Inexact PIncorrect Dup Right Wrong Prec Recall F1 CSSF micro 0 4840 4 3 0 1 0 0 3 1 0.7500 0.0006 0.0012 CSSF micro 1 3954 0 0 0 0 0 0 0 0 0.0000 0.0000 0.0000 CSSF micro ALL 8794 4 3 0 1 0 0 3 1 0.7500 0.0003 0.0007 LDC-MEAN macro 0 0.0011 LDC-MEAN macro 1 0.0000 LDC-MEAN macro ALL 0.0007 LDC-MAX micro 0 1268 3 3 0 0 0 0 3 0 1.0000 0.0024 0.0047 LDC-MAX micro 1 900 0 0 0 0 0 0 0 0 0.0000 0.0000 0.0000 LDC-MAX micro ALL 2168 3 3 0 0 0 0 3 0 1.0000 0.0014 0.0028 LDC-MAX macro 0 0.0054 LDC-MAX macro 1 0.0000 LDC-MAX macro ALL 0.0035 ************************************************************* Run ID: ICTCAS_OKN3 Did the run access the live Web during the evaluation window: No Run Number of most similar Cold Start Slot Filling task submission: NA Did the run extract relations from the Cold Start source corpus: Yes Did the run generate meaningful confidence values: No Entity Discovery Evaluation: Prec Recall F1 Metric 0.229 0.175 0.198 strong_mention_match 0.210 0.161 0.182 strong_typed_mention_match 0.000 0.000 0.000 entity_match 0.176 0.105 0.132 b_cubed 0.199 0.152 0.172 mention_ceaf 0.187 0.143 0.162 typed_mention_ceaf 0.187 0.143 0.162 typed_mention_ceaf_plus Slot Filling Evaluation: Metric Hop GoldT Submit Correct Incorrect Inexact PIncorrect Dup Right Wrong Prec Recall F1 CSSF micro 0 4840 94 42 42 10 0 2 40 54 0.4255 0.0083 0.0162 CSSF micro 1 3954 8 0 0 0 8 0 0 8 0.0000 0.0000 0.0000 CSSF micro ALL 8794 102 42 42 10 8 2 40 62 0.3922 0.0045 0.0090 LDC-MEAN macro 0 0.0143 LDC-MEAN macro 1 0.0000 LDC-MEAN macro ALL 0.0092 LDC-MAX micro 0 1268 33 21 10 2 0 1 20 13 0.6061 0.0158 0.0307 LDC-MAX micro 1 900 1 0 0 0 1 0 0 1 0.0000 0.0000 0.0000 LDC-MAX micro ALL 2168 34 21 10 2 1 1 20 14 0.5882 0.0092 0.0182 LDC-MAX macro 0 0.0248 LDC-MAX macro 1 0.0000 LDC-MAX macro ALL 0.0159 ************************************************************* Run ID: ICTCAS_OKN4 Did the run access the live Web during the evaluation window: No Run Number of most similar Cold Start Slot Filling task submission: NA Did the run extract relations from the Cold Start source corpus: Yes Did the run generate meaningful confidence values: No Entity Discovery Evaluation: Prec Recall F1 Metric 0.229 0.175 0.198 strong_mention_match 0.210 0.161 0.182 strong_typed_mention_match 0.000 0.000 0.000 entity_match 0.222 0.092 0.130 b_cubed 0.173 0.132 0.150 mention_ceaf 0.163 0.125 0.142 typed_mention_ceaf 0.163 0.125 0.142 typed_mention_ceaf_plus Slot Filling Evaluation: Metric Hop GoldT Submit Correct Incorrect Inexact PIncorrect Dup Right Wrong Prec Recall F1 CSSF micro 0 4840 7 5 2 0 0 0 5 2 0.7143 0.0010 0.0021 CSSF micro 1 3954 0 0 0 0 0 0 0 0 0.0000 0.0000 0.0000 CSSF micro ALL 8794 7 5 2 0 0 0 5 2 0.7143 0.0006 0.0011 LDC-MEAN macro 0 0.0015 LDC-MEAN macro 1 0.0000 LDC-MEAN macro ALL 0.0010 LDC-MAX micro 0 1268 6 5 1 0 0 0 5 1 0.8333 0.0039 0.0078 LDC-MAX micro 1 900 0 0 0 0 0 0 0 0 0.0000 0.0000 0.0000 LDC-MAX micro ALL 2168 6 5 1 0 0 0 5 1 0.8333 0.0023 0.0046 LDC-MAX macro 0 0.0064 LDC-MAX macro 1 0.0000 LDC-MAX macro ALL 0.0041