==================================================== TAC KBP 2017 ENGLISH SLOT FILLING EVALUATION RESULTS ==================================================== Team ID: UNIST_SAIL Organization: Ulsan National Institute of Science and Technology ************************************************************* Run ID: UNIST_SAIL_SF_ENG_1 Did the run access the live Web during the evaluation window: No Did the run extract relations from the Cold Start source corpus: Yes Did the run generate meaningful confidence values: Yes Slot Filling Evaluation (Queries involve ONLY SF slots): Scores based on P/R/F1; only k=1 justification allowed: Metric RunID Hop Prec Recall F1 SF-ALL-Micro UNIST_SAIL_SF_ENG_1 0 0.0551 0.3420 0.0950 SF-ALL-Micro UNIST_SAIL_SF_ENG_1 1 0.0061 0.0874 0.0115 SF-ALL-Micro UNIST_SAIL_SF_ENG_1 ALL 0.0345 0.2808 0.0615 SF-ALL-Macro UNIST_SAIL_SF_ENG_1 0 0.1266 0.3705 0.1612 SF-ALL-Macro UNIST_SAIL_SF_ENG_1 1 0.0255 0.0687 0.0355 SF-ALL-Macro UNIST_SAIL_SF_ENG_1 ALL 0.0865 0.2509 0.1114 LDC-MAX-ALL-Micro UNIST_SAIL_SF_ENG_1 0 0.0583 0.3931 0.1015 LDC-MAX-ALL-Micro UNIST_SAIL_SF_ENG_1 1 0.0075 0.1266 0.0142 LDC-MAX-ALL-Micro UNIST_SAIL_SF_ENG_1 ALL 0.0365 0.3314 0.0658 LDC-MAX-ALL-Macro UNIST_SAIL_SF_ENG_1 0 0.1368 0.4143 0.1737 LDC-MAX-ALL-Macro UNIST_SAIL_SF_ENG_1 1 0.0378 0.0881 0.0503 LDC-MAX-ALL-Macro UNIST_SAIL_SF_ENG_1 ALL 0.0984 0.2876 0.1258 LDC-MEAN-ALL-Macro UNIST_SAIL_SF_ENG_1 0 0.1247 0.3813 0.1591 LDC-MEAN-ALL-Macro UNIST_SAIL_SF_ENG_1 1 0.0310 0.0818 0.0433 LDC-MEAN-ALL-Macro UNIST_SAIL_SF_ENG_1 ALL 0.0883 0.2650 0.1141 *ALL-Macro Prec, Recall and F1 refer to mean-precision, mean-recall and mean-F1. Scores based on confidence values and Average Precision (AP); only k=1 justification allowed: Metric RunID Hop AP SF-ALL-Macro UNIST_SAIL_SF_ENG_1 0 0.2043 SF-ALL-Macro UNIST_SAIL_SF_ENG_1 1 0.0081 SF-ALL-Macro UNIST_SAIL_SF_ENG_1 ALL 0.1360 LDC-MEAN-ALL-Macro UNIST_SAIL_SF_ENG_1 0 0.2121 LDC-MEAN-ALL-Macro UNIST_SAIL_SF_ENG_1 1 0.0097 LDC-MEAN-ALL-Macro UNIST_SAIL_SF_ENG_1 ALL 0.1471 (PRIMARY METRIC) *ALL-Macro AP refer to mean of corresponding AP values. ************************************************************* Run ID: UNIST_SAIL_SF_ENG_2 Did the run access the live Web during the evaluation window: Yes Did the run extract relations from the Cold Start source corpus: Yes Did the run generate meaningful confidence values: Yes Slot Filling Evaluation (Queries involve ONLY SF slots): Scores based on P/R/F1; only k=1 justification allowed: Metric RunID Hop Prec Recall F1 SF-ALL-Micro UNIST_SAIL_SF_ENG_2 0 0.2712 0.1520 0.1948 SF-ALL-Micro UNIST_SAIL_SF_ENG_2 1 0.0000 0.0000 0.0000 SF-ALL-Micro UNIST_SAIL_SF_ENG_2 ALL 0.2667 0.1155 0.1612 SF-ALL-Macro UNIST_SAIL_SF_ENG_2 0 0.2029 0.1937 0.1825 SF-ALL-Macro UNIST_SAIL_SF_ENG_2 1 0.0000 0.0000 0.0000 SF-ALL-Macro UNIST_SAIL_SF_ENG_2 ALL 0.1225 0.1169 0.1102 LDC-MAX-ALL-Micro UNIST_SAIL_SF_ENG_2 0 0.2742 0.1889 0.2237 LDC-MAX-ALL-Micro UNIST_SAIL_SF_ENG_2 1 0.0000 0.0000 0.0000 LDC-MAX-ALL-Micro UNIST_SAIL_SF_ENG_2 ALL 0.2690 0.1452 0.1886 LDC-MAX-ALL-Macro UNIST_SAIL_SF_ENG_2 0 0.2348 0.2306 0.2154 LDC-MAX-ALL-Macro UNIST_SAIL_SF_ENG_2 1 0.0000 0.0000 0.0000 LDC-MAX-ALL-Macro UNIST_SAIL_SF_ENG_2 ALL 0.1436 0.1411 0.1318 LDC-MEAN-ALL-Macro UNIST_SAIL_SF_ENG_2 0 0.2052 0.1993 0.1865 LDC-MEAN-ALL-Macro UNIST_SAIL_SF_ENG_2 1 0.0000 0.0000 0.0000 LDC-MEAN-ALL-Macro UNIST_SAIL_SF_ENG_2 ALL 0.1255 0.1219 0.1141 *ALL-Macro Prec, Recall and F1 refer to mean-precision, mean-recall and mean-F1. Scores based on confidence values and Average Precision (AP); only k=1 justification allowed: Metric RunID Hop AP SF-ALL-Macro UNIST_SAIL_SF_ENG_2 0 0.1777 SF-ALL-Macro UNIST_SAIL_SF_ENG_2 1 0.0000 SF-ALL-Macro UNIST_SAIL_SF_ENG_2 ALL 0.1310 LDC-MEAN-ALL-Macro UNIST_SAIL_SF_ENG_2 0 0.1820 LDC-MEAN-ALL-Macro UNIST_SAIL_SF_ENG_2 1 0.0000 LDC-MEAN-ALL-Macro UNIST_SAIL_SF_ENG_2 ALL 0.1352 (PRIMARY METRIC) *ALL-Macro AP refer to mean of corresponding AP values.