======================================================= TAC KBP 2015 COLD START SLOT FILLING EVALUATION RESULTS ======================================================= Team ID: CMUML Organization: Carnegie Mellon University ************************************************************* Run ID: CMUML1 Did the run access the live Web during the evaluation window: No Run Number of most similar Cold Start KB Construction task submission: NA Did the run extract relations from the Cold Start source corpus: No Did the run generate meaningful confidence values: Yes Slot Filling Evaluation: Metric Hop GoldT Submit Correct Incorrect Inexact PIncorrect Dup Right Wrong Prec Recall F1 CSSF micro 0 4840 48 30 14 4 0 0 30 18 0.6250 0.0062 0.0123 CSSF micro 1 3954 0 0 0 0 0 0 0 0 0.0000 0.0000 0.0000 CSSF micro ALL 8794 48 30 14 4 0 0 30 18 0.6250 0.0034 0.0068 LDC-MEAN macro 0 0.0089 LDC-MEAN macro 1 0.0000 LDC-MEAN macro ALL 0.0057 LDC-MAX micro 0 1268 16 10 5 1 0 0 10 6 0.6250 0.0079 0.0156 LDC-MAX micro 1 900 0 0 0 0 0 0 0 0 0.0000 0.0000 0.0000 LDC-MAX micro ALL 2168 16 10 5 1 0 0 10 6 0.6250 0.0046 0.0092 LDC-MAX macro 0 0.0108 LDC-MAX macro 1 0.0000 LDC-MAX macro ALL 0.0069 ************************************************************* Run ID: CMUML2 Did the run access the live Web during the evaluation window: Yes Run Number of most similar Cold Start KB Construction task submission: NA Did the run extract relations from the Cold Start source corpus: No Did the run generate meaningful confidence values: Yes Slot Filling Evaluation: Metric Hop GoldT Submit Correct Incorrect Inexact PIncorrect Dup Right Wrong Prec Recall F1 CSSF micro 0 4840 49 30 15 4 0 0 30 19 0.6122 0.0062 0.0123 CSSF micro 1 3954 0 0 0 0 0 0 0 0 0.0000 0.0000 0.0000 CSSF micro ALL 8794 49 30 15 4 0 0 30 19 0.6122 0.0034 0.0068 LDC-MEAN macro 0 0.0089 LDC-MEAN macro 1 0.0000 LDC-MEAN macro ALL 0.0057 LDC-MAX micro 0 1268 13 10 2 1 0 0 10 3 0.7692 0.0079 0.0156 LDC-MAX micro 1 900 0 0 0 0 0 0 0 0 0.0000 0.0000 0.0000 LDC-MAX micro ALL 2168 13 10 2 1 0 0 10 3 0.7692 0.0046 0.0092 LDC-MAX macro 0 0.0108 LDC-MAX macro 1 0.0000 LDC-MAX macro ALL 0.0069 ************************************************************* Run ID: CMUML3 Did the run access the live Web during the evaluation window: No Run Number of most similar Cold Start KB Construction task submission: NA Did the run extract relations from the Cold Start source corpus: No Did the run generate meaningful confidence values: Yes Slot Filling Evaluation: Metric Hop GoldT Submit Correct Incorrect Inexact PIncorrect Dup Right Wrong Prec Recall F1 CSSF micro 0 4840 32 18 10 4 0 0 18 14 0.5625 0.0037 0.0074 CSSF micro 1 3954 0 0 0 0 0 0 0 0 0.0000 0.0000 0.0000 CSSF micro ALL 8794 32 18 10 4 0 0 18 14 0.5625 0.0020 0.0041 LDC-MEAN macro 0 0.0070 LDC-MEAN macro 1 0.0000 LDC-MEAN macro ALL 0.0045 LDC-MAX micro 0 1268 10 7 1 2 0 0 7 3 0.7000 0.0055 0.0110 LDC-MAX micro 1 900 0 0 0 0 0 0 0 0 0.0000 0.0000 0.0000 LDC-MAX micro ALL 2168 10 7 1 2 0 0 7 3 0.7000 0.0032 0.0064 LDC-MAX macro 0 0.0088 LDC-MAX macro 1 0.0000 LDC-MAX macro ALL 0.0056