==================================================== TAC KBP 2017 CHINESE SLOT FILLING EVALUATION RESULTS ==================================================== Team ID: STANFORD Organization: Stanford University ************************************************************* Run ID: STANFORD_SF_CMN_1 Did the run access the live Web during the evaluation window: No Did the run extract relations from the Cold Start source corpus: Yes Did the run generate meaningful confidence values: Yes Slot Filling Evaluation (Queries involve ONLY SF slots): Scores based on P/R/F1; only k=1 justification allowed: Metric RunID Hop Prec Recall F1 SF-ALL-Micro STANFORD_SF_CMN_1 0 0.7475 0.3671 0.4924 SF-ALL-Micro STANFORD_SF_CMN_1 1 0.2800 0.1160 0.1641 SF-ALL-Micro STANFORD_SF_CMN_1 ALL 0.6553 0.3105 0.4213 SF-ALL-Macro STANFORD_SF_CMN_1 0 0.2544 0.2320 0.2355 SF-ALL-Macro STANFORD_SF_CMN_1 1 0.0881 0.0811 0.0804 SF-ALL-Macro STANFORD_SF_CMN_1 ALL 0.1978 0.1806 0.1827 LDC-MAX-ALL-Micro STANFORD_SF_CMN_1 0 0.7455 0.3695 0.4941 LDC-MAX-ALL-Micro STANFORD_SF_CMN_1 1 0.2667 0.1404 0.1839 LDC-MAX-ALL-Micro STANFORD_SF_CMN_1 ALL 0.6444 0.3233 0.4306 LDC-MAX-ALL-Macro STANFORD_SF_CMN_1 0 0.2856 0.2576 0.2623 LDC-MAX-ALL-Macro STANFORD_SF_CMN_1 1 0.0949 0.0974 0.0930 LDC-MAX-ALL-Macro STANFORD_SF_CMN_1 ALL 0.2173 0.2002 0.2017 LDC-MEAN-ALL-Macro STANFORD_SF_CMN_1 0 0.2664 0.2375 0.2419 LDC-MEAN-ALL-Macro STANFORD_SF_CMN_1 1 0.0938 0.0970 0.0924 LDC-MEAN-ALL-Macro STANFORD_SF_CMN_1 ALL 0.2046 0.1872 0.1884 *ALL-Macro Prec, Recall and F1 refer to mean-precision, mean-recall and mean-F1. Scores based on confidence values and Average Precision (AP); only k=1 justification allowed: Metric RunID Hop AP SF-ALL-Macro STANFORD_SF_CMN_1 0 0.2220 SF-ALL-Macro STANFORD_SF_CMN_1 1 0.0259 SF-ALL-Macro STANFORD_SF_CMN_1 ALL 0.1627 LDC-MEAN-ALL-Macro STANFORD_SF_CMN_1 0 0.2259 LDC-MEAN-ALL-Macro STANFORD_SF_CMN_1 1 0.0193 LDC-MEAN-ALL-Macro STANFORD_SF_CMN_1 ALL 0.1735 (PRIMARY METRIC) *ALL-Macro AP refer to mean of corresponding AP values. ************************************************************* Run ID: STANFORD_SF_CMN_2 Did the run access the live Web during the evaluation window: No Did the run extract relations from the Cold Start source corpus: Yes Did the run generate meaningful confidence values: Yes Slot Filling Evaluation (Queries involve ONLY SF slots): Scores based on P/R/F1; only k=1 justification allowed: Metric RunID Hop Prec Recall F1 SF-ALL-Micro STANFORD_SF_CMN_2 0 0.6676 0.3655 0.4724 SF-ALL-Micro STANFORD_SF_CMN_2 1 0.1923 0.1105 0.1404 SF-ALL-Micro STANFORD_SF_CMN_2 ALL 0.5563 0.3080 0.3965 SF-ALL-Macro STANFORD_SF_CMN_2 0 0.2514 0.2316 0.2332 SF-ALL-Macro STANFORD_SF_CMN_2 1 0.0873 0.0788 0.0788 SF-ALL-Macro STANFORD_SF_CMN_2 ALL 0.1956 0.1796 0.1806 LDC-MAX-ALL-Micro STANFORD_SF_CMN_2 0 0.6385 0.3673 0.4663 LDC-MAX-ALL-Micro STANFORD_SF_CMN_2 1 0.1685 0.1316 0.1478 LDC-MAX-ALL-Micro STANFORD_SF_CMN_2 ALL 0.5186 0.3198 0.3956 LDC-MAX-ALL-Macro STANFORD_SF_CMN_2 0 0.2809 0.2570 0.2588 LDC-MAX-ALL-Macro STANFORD_SF_CMN_2 1 0.0938 0.0942 0.0907 LDC-MAX-ALL-Macro STANFORD_SF_CMN_2 ALL 0.2139 0.1987 0.1986 LDC-MEAN-ALL-Macro STANFORD_SF_CMN_2 0 0.2618 0.2369 0.2385 LDC-MEAN-ALL-Macro STANFORD_SF_CMN_2 1 0.0927 0.0937 0.0901 LDC-MEAN-ALL-Macro STANFORD_SF_CMN_2 ALL 0.2013 0.1856 0.1853 *ALL-Macro Prec, Recall and F1 refer to mean-precision, mean-recall and mean-F1. Scores based on confidence values and Average Precision (AP); only k=1 justification allowed: Metric RunID Hop AP SF-ALL-Macro STANFORD_SF_CMN_2 0 0.2210 SF-ALL-Macro STANFORD_SF_CMN_2 1 0.0256 SF-ALL-Macro STANFORD_SF_CMN_2 ALL 0.1622 LDC-MEAN-ALL-Macro STANFORD_SF_CMN_2 0 0.2243 LDC-MEAN-ALL-Macro STANFORD_SF_CMN_2 1 0.0189 LDC-MEAN-ALL-Macro STANFORD_SF_CMN_2 ALL 0.1727 (PRIMARY METRIC) *ALL-Macro AP refer to mean of corresponding AP values. ************************************************************* Run ID: STANFORD_SF_CMN_3 Did the run access the live Web during the evaluation window: No Did the run extract relations from the Cold Start source corpus: Yes Did the run generate meaningful confidence values: Yes Slot Filling Evaluation (Queries involve ONLY SF slots): Scores based on P/R/F1; only k=1 justification allowed: Metric RunID Hop Prec Recall F1 SF-ALL-Micro STANFORD_SF_CMN_3 0 0.7475 0.3671 0.4924 SF-ALL-Micro STANFORD_SF_CMN_3 1 0.2800 0.1160 0.1641 SF-ALL-Micro STANFORD_SF_CMN_3 ALL 0.6553 0.3105 0.4213 SF-ALL-Macro STANFORD_SF_CMN_3 0 0.2544 0.2320 0.2355 SF-ALL-Macro STANFORD_SF_CMN_3 1 0.0881 0.0811 0.0804 SF-ALL-Macro STANFORD_SF_CMN_3 ALL 0.1978 0.1806 0.1827 LDC-MAX-ALL-Micro STANFORD_SF_CMN_3 0 0.7523 0.3695 0.4955 LDC-MAX-ALL-Micro STANFORD_SF_CMN_3 1 0.2667 0.1404 0.1839 LDC-MAX-ALL-Micro STANFORD_SF_CMN_3 ALL 0.6489 0.3233 0.4316 LDC-MAX-ALL-Macro STANFORD_SF_CMN_3 0 0.2856 0.2576 0.2623 LDC-MAX-ALL-Macro STANFORD_SF_CMN_3 1 0.0949 0.0974 0.0930 LDC-MAX-ALL-Macro STANFORD_SF_CMN_3 ALL 0.2173 0.2002 0.2017 LDC-MEAN-ALL-Macro STANFORD_SF_CMN_3 0 0.2664 0.2375 0.2419 LDC-MEAN-ALL-Macro STANFORD_SF_CMN_3 1 0.0938 0.0970 0.0924 LDC-MEAN-ALL-Macro STANFORD_SF_CMN_3 ALL 0.2046 0.1872 0.1884 *ALL-Macro Prec, Recall and F1 refer to mean-precision, mean-recall and mean-F1. Scores based on confidence values and Average Precision (AP); only k=1 justification allowed: Metric RunID Hop AP SF-ALL-Macro STANFORD_SF_CMN_3 0 0.2220 SF-ALL-Macro STANFORD_SF_CMN_3 1 0.0259 SF-ALL-Macro STANFORD_SF_CMN_3 ALL 0.1627 LDC-MEAN-ALL-Macro STANFORD_SF_CMN_3 0 0.2259 LDC-MEAN-ALL-Macro STANFORD_SF_CMN_3 1 0.0193 LDC-MEAN-ALL-Macro STANFORD_SF_CMN_3 ALL 0.1735 (PRIMARY METRIC) *ALL-Macro AP refer to mean of corresponding AP values. ************************************************************* Run ID: STANFORD_SF_CMN_4 Did the run access the live Web during the evaluation window: No Did the run extract relations from the Cold Start source corpus: Yes Did the run generate meaningful confidence values: Yes Slot Filling Evaluation (Queries involve ONLY SF slots): Scores based on P/R/F1; only k=1 justification allowed: Metric RunID Hop Prec Recall F1 SF-ALL-Micro STANFORD_SF_CMN_4 0 0.7475 0.3671 0.4924 SF-ALL-Micro STANFORD_SF_CMN_4 1 0.2763 0.1160 0.1634 SF-ALL-Micro STANFORD_SF_CMN_4 ALL 0.6535 0.3105 0.4210 SF-ALL-Macro STANFORD_SF_CMN_4 0 0.2544 0.2320 0.2355 SF-ALL-Macro STANFORD_SF_CMN_4 1 0.0881 0.0811 0.0804 SF-ALL-Macro STANFORD_SF_CMN_4 ALL 0.1978 0.1806 0.1827 LDC-MAX-ALL-Micro STANFORD_SF_CMN_4 0 0.7489 0.3695 0.4948 LDC-MAX-ALL-Micro STANFORD_SF_CMN_4 1 0.2623 0.1404 0.1829 LDC-MAX-ALL-Micro STANFORD_SF_CMN_4 ALL 0.6444 0.3233 0.4306 LDC-MAX-ALL-Macro STANFORD_SF_CMN_4 0 0.2856 0.2576 0.2623 LDC-MAX-ALL-Macro STANFORD_SF_CMN_4 1 0.0949 0.0974 0.0930 LDC-MAX-ALL-Macro STANFORD_SF_CMN_4 ALL 0.2173 0.2002 0.2017 LDC-MEAN-ALL-Macro STANFORD_SF_CMN_4 0 0.2664 0.2375 0.2419 LDC-MEAN-ALL-Macro STANFORD_SF_CMN_4 1 0.0938 0.0970 0.0924 LDC-MEAN-ALL-Macro STANFORD_SF_CMN_4 ALL 0.2046 0.1872 0.1884 *ALL-Macro Prec, Recall and F1 refer to mean-precision, mean-recall and mean-F1. Scores based on confidence values and Average Precision (AP); only k=1 justification allowed: Metric RunID Hop AP SF-ALL-Macro STANFORD_SF_CMN_4 0 0.2220 SF-ALL-Macro STANFORD_SF_CMN_4 1 0.0259 SF-ALL-Macro STANFORD_SF_CMN_4 ALL 0.1627 LDC-MEAN-ALL-Macro STANFORD_SF_CMN_4 0 0.2259 LDC-MEAN-ALL-Macro STANFORD_SF_CMN_4 1 0.0193 LDC-MEAN-ALL-Macro STANFORD_SF_CMN_4 ALL 0.1735 (PRIMARY METRIC) *ALL-Macro AP refer to mean of corresponding AP values.