======================================================================== TAC KBP 2016 SLOT FILLER VALIDATION ENGLISH FILTERING EVALUATION RESULTS ======================================================================== Team ID: UI_CCG Organization: University of Illinois at Urbana-Champaign ************************************************************* Run ID: UI_CCG1 Did the run access the live Web during the evaluation window: No Did this run judge each candidate slot filler independently of all other candidate slot fillers in the evaluation dataset: Yes Did this run judge candidate slot fillers for each slot-filling run independently of all other slot-filling runs in the evaluation dataset: Yes Did this run judge candidate slot fillers for each slot-filling team independently of all other slot-filling teams in the evaluation dataset: Yes Did this run make use of the slot filler or justification offsets provided for each candidate slot filler: Yes Did this run make use of the confidence values provided for each candidate slot filler: No Did this run make use of the system profiles for the slot filling runs: No Did this run make use of the preliminary assessments provided for some of the slot filler candidates: No CSSF micro-average F1 and change in F1 after applying SFV filter, at each hop level: Cold Start run hop0_F Post-filter hop1_F Post-filter ALL_F Post-filter BBN_KB_ENG_1 0.3174 -0.0044 0.1737 -0.0035 0.2693 -0.0045 BBN_KB_ENG_2 0.3040 -0.0082 0.1650 -0.0036 0.2571 -0.0069 BBN_KB_ENG_3 0.3150 -0.0048 0.1724 -0.0033 0.2651 -0.0045 BBN_KB_ENG_4 0.3188 -0.0047 0.1734 -0.0035 0.2703 -0.0046 BBN_KB_ENG_5 0.3041 -0.0044 0.1529 -0.0029 0.2475 -0.0041 CMUML_SF_ENG_1 0.0440 -0.0051 0.0000 0 0.0291 -0.0035 CMUML_SF_ENG_2 0.0371 -0.0047 0.0000 0 0.0249 -0.0032 CMUML_SF_ENG_3 0.0057 -0.0028 0.0000 0 0.0038 -0.0019 CMUML_SF_ENG_4 0.0369 -0.0026 0.0000 0 0.0246 -0.0018 CMUML_SF_ENG_5 0.0440 -0.0051 0.0000 0 0.0291 -0.0035 DCD_SF_SF_ENG_1 0.0804 -0.0097 0.0198 -0.003 0.0588 -0.0073 DCD_SF_SF_ENG_2 0.0904 -0.0054 0.0186 0.0005 0.0647 -0.0033 doughnutPRIS_SF_ENG_4 0.0865 -0.0082 0.0000 0 0.0648 -0.0065 doughnutPRIS_SF_ENG_5 0.1143 -0.0051 0.0000 0 0.0894 -0.0045 hltcoe_KB_ENG_1 0.2949 -0.0115 0.1202 -0.0017 0.2300 -0.0085 hltcoe_KB_ENG_2 0.2953 -0.0125 0.1180 -0.0022 0.2311 -0.0095 hltcoe_KB_ENG_3 0.2976 -0.0124 0.1173 -0.0023 0.2320 -0.0095 hltcoe_KB_ENG_4 0.2776 -0.005 0.0830 0.0014 0.1959 -0.0012 hltcoe_KB_ENG_5 0.2966 -0.0114 0.1202 -0.0017 0.2311 -0.0085 ICTCAS_OKN_KB_ENG_1 0.1487 -0.0002 0.0441 0.0014 0.1101 0.0005 ICTCAS_OKN_KB_ENG_2 0.1490 -0.0001 0.0374 0.0011 0.1081 0.0005 ICTCAS_OKN_KB_ENG_3 0.1186 -0.0003 0.0526 0.0045 0.0960 0.0023 ICTCAS_OKN_KB_ENG_4 0.1421 -0.0002 0.0315 0.0015 0.1041 0.0009 IRTSX_SF_ENG_1 0.0062 0 0.0000 0 0.0038 0.0001 IRTSX_SF_ENG_2 0.0064 0 0.0000 0 0.0038 0 IRTSX_SF_ENG_3 0.0072 0 0.0000 0 0.0051 0 lilian_KB_ENG_1 0.0452 -0.0021 0.0000 0 0.0311 -0.0016 lilian_KB_ENG_2 0.0465 -0.0024 0.0000 0 0.0317 -0.0017 lilian_KB_ENG_3 0.0468 -0.0025 0.0000 0 0.0318 -0.0017 MSR_SF_ENG_1 0.0275 0.0004 0.0000 0 0.0150 0.0002 MSR_SF_ENG_2 0.0297 -0.0054 0.0000 0 0.0198 -0.0036 MSR_SF_ENG_3 0.0201 0.0001 0.0000 0 0.0122 0.0001 MSR_SF_ENG_4 0.0635 -0.0004 0.0102 0.0001 0.0377 -0.0001 MSR_SF_ENG_5 0.0176 -0.0008 0.0000 0 0.0083 -0.0003 NAIST_CL_SF_ENG_1 0.0589 -0.0003 0.0100 0.0005 0.0146 0.0005 NAIST_CL_SF_ENG_2 0.0589 -0.0003 0.0000 0 0.0438 -0.0006 NAIST_CL_SF_ENG_3 0.0589 -0.0003 0.0000 0 0.0433 -0.0005 RPI_BLENDER_SF_ENG_1 0.1904 0.0002 0.1050 0 0.1644 0.0001 RPI_BLENDER_SF_ENG_2 0.1721 0.0002 0.0960 0 0.1487 0.0001 RPI_BLENDER_SF_ENG_3 0.1700 0.0002 0.0950 0 0.1470 0.0001 SoochowNLP_SF_ENG_1 0.0451 0.0003 0.0000 0 0.0363 0 SoochowNLP_SF_ENG_2 0.0372 0.001 0.0000 0 0.0275 0.0006 SoochowNLP_SF_ENG_3 0.0372 0.001 0.0000 0 0.0192 0.0003 SoochowNLP_SF_ENG_4 0.0372 0.001 0.0000 0 0.0262 0.0006 SoochowNLP_SF_ENG_5 0.0372 0.001 0.0000 0 0.0254 0.0005 Stanford_KB_ENG_1 0.2406 -0.005 0.1466 -0.0075 0.2100 -0.0062 Stanford_KB_ENG_2 0.1926 -0.0147 0.1039 -0.0099 0.1648 -0.0134 Stanford_KB_ENG_3 0.2633 -0.0037 0.1190 -0.0038 0.2018 -0.0046 Stanford_SF_ENG_1 0.2310 -0.0034 0.1197 -0.0041 0.1945 -0.0038 Stanford_SF_ENG_2 0.2444 -0.0065 0.1241 -0.002 0.1971 -0.0046 Stanford_SF_ENG_3 0.2846 0.0004 0.0819 0.0043 0.1751 0.0046 Stanford_SF_ENG_4 0.2044 -0.0105 0.1061 -0.0098 0.1735 -0.0103 Stanford_SF_ENG_5 0.2310 -0.0032 0.0798 0 0.1827 -0.0026 summa_KB_ENG_1 0.0255 -0.0056 0.0000 0 0.0171 -0.0037 summa_KB_ENG_2 0.0584 -0.008 0.0114 -0.0057 0.0430 -0.0073 summa_KB_ENG_5 0.0261 -0.0057 0.0116 -0.0058 0.0213 -0.0058 UMass_IESL_KB_ENG_1 0.1844 -0.0073 0.0628 -0.0009 0.1252 -0.0047 UMass_IESL_KB_ENG_2 0.1704 -0.0082 0.0687 -0.0025 0.1267 -0.0063 UMass_IESL_KB_ENG_3 0.2076 -0.0071 0.1480 -0.0012 0.1878 -0.0052 UMass_IESL_KB_ENG_4 0.1624 -0.0089 0.1061 0.0018 0.1423 -0.0052 UMass_IESL_KB_ENG_5 0.1732 -0.0064 0.0399 -0.0004 0.0885 -0.0031 UMass_IESL_SF_ENG_1 0.2207 -0.0152 0.0282 0 0.1794 -0.0134 UMass_IESL_SF_ENG_2 0.2224 -0.0184 0.0384 0 0.1751 -0.015 UMass_IESL_SF_ENG_3 0.2092 -0.0036 0.0954 0.0102 0.1668 0.0032 UMass_IESL_SF_ENG_4 0.2061 -0.0198 0.0282 0 0.1643 -0.0164 UMass_IESL_SF_ENG_5 0.2070 -0.0179 0.0285 0 0.1627 -0.0146 UNIST_SAIL_SF_ENG_1 0.0438 -0.0012 0.0000 0 0.0109 -0.0005 UNIST_SAIL_SF_ENG_2 0.0476 -0.0007 0.0000 0 0.0352 -0.0008 UNIST_SAIL_SF_ENG_3 0.0583 -0.001 0.0000 0 0.0426 -0.0009 UNIST_SAIL_SF_ENG_4 0.0788 0.0007 0.0000 0 0.0610 0.0001 Average change in F -0.00455429 -0.000771429 -0.00343571 MAX micro-average F1 and change in F1 after applying SFV filter, at each hop level: Cold Start run hop0_F Post-filter hop1_F Post-filter ALL_F Post-filter BBN_KB_ENG_1 0.3358 -0.0054 0.1659 -0.0045 0.2781 -0.0055 BBN_KB_ENG_2 0.3234 -0.0101 0.1542 -0.0046 0.2655 -0.0084 BBN_KB_ENG_3 0.3313 -0.0055 0.1648 -0.0041 0.2716 -0.0055 BBN_KB_ENG_4 0.3375 -0.0079 0.1655 -0.0045 0.2791 -0.0069 BBN_KB_ENG_5 0.3253 -0.0046 0.1442 -0.0035 0.2550 -0.0048 CMUML_SF_ENG_1 0.0462 -0.0066 0.0000 0 0.0309 -0.0046 CMUML_SF_ENG_2 0.0409 -0.0058 0.0000 0 0.0276 -0.004 CMUML_SF_ENG_3 0.0074 -0.0037 0.0000 0 0.0049 -0.0024 CMUML_SF_ENG_4 0.0370 -0.0033 0.0000 0 0.0248 -0.0022 CMUML_SF_ENG_5 0.0463 -0.0067 0.0000 0 0.0310 -0.0047 DCD_SF_SF_ENG_1 0.0889 -0.0106 0.0219 -0.0038 0.0652 -0.008 DCD_SF_SF_ENG_2 0.1015 -0.005 0.0208 0.0008 0.0728 -0.0028 doughnutPRIS_SF_ENG_4 0.0847 -0.0067 0.0000 0 0.0631 -0.0053 doughnutPRIS_SF_ENG_5 0.1138 -0.0037 0.0000 0 0.0887 -0.0034 hltcoe_KB_ENG_1 0.2912 -0.0118 0.1082 -0.0023 0.2222 -0.0086 hltcoe_KB_ENG_2 0.2887 -0.0102 0.1014 -0.003 0.2204 -0.0084 hltcoe_KB_ENG_3 0.2891 -0.012 0.1000 -0.003 0.2191 -0.0093 hltcoe_KB_ENG_4 0.2735 -0.0057 0.0734 0.0012 0.1872 -0.001 hltcoe_KB_ENG_5 0.2925 -0.0107 0.1082 -0.0023 0.2232 -0.0081 ICTCAS_OKN_KB_ENG_1 0.1507 -0.0013 0.0445 0.0011 0.1120 -0.0005 ICTCAS_OKN_KB_ENG_2 0.1521 -0.0013 0.0401 0.0008 0.1115 -0.0007 ICTCAS_OKN_KB_ENG_3 0.1441 -0.0002 0.0656 0.0063 0.1166 0.0036 ICTCAS_OKN_KB_ENG_4 0.1471 -0.0015 0.0216 0.0009 0.1055 -0.0003 IRTSX_SF_ENG_1 0.0081 0.0001 0.0000 0 0.0050 0.0001 IRTSX_SF_ENG_2 0.0088 -0.0006 0.0000 0 0.0059 -0.0011 IRTSX_SF_ENG_3 0.0097 0 0.0000 0 0.0068 0 lilian_KB_ENG_1 0.0446 -0.0028 0.0000 0 0.0308 -0.002 lilian_KB_ENG_2 0.0462 -0.0033 0.0000 0 0.0316 -0.0023 lilian_KB_ENG_3 0.0467 -0.0033 0.0000 0 0.0318 -0.0023 MSR_SF_ENG_1 0.0284 0.0004 0.0000 0 0.0153 0.0002 MSR_SF_ENG_2 0.0343 -0.0071 0.0000 0 0.0228 -0.0047 MSR_SF_ENG_3 0.0247 0.0002 0.0000 0 0.0150 0.0001 MSR_SF_ENG_4 0.0661 -0.0008 0.0117 0.0001 0.0393 -0.0003 MSR_SF_ENG_5 0.0207 -0.0011 0.0000 0 0.0096 -0.0004 NAIST_CL_SF_ENG_1 0.0535 -0.0012 0.0092 0.0004 0.0138 0.0004 NAIST_CL_SF_ENG_2 0.0534 -0.0011 0.0000 0 0.0397 -0.0012 NAIST_CL_SF_ENG_3 0.0533 -0.001 0.0000 0 0.0391 -0.0009 RPI_BLENDER_SF_ENG_1 0.1888 0.0017 0.0855 0.0009 0.1575 0.0015 RPI_BLENDER_SF_ENG_2 0.1647 0 0.0795 0 0.1385 0 RPI_BLENDER_SF_ENG_3 0.1628 0.0009 0.0789 0.0006 0.1371 0.0008 SoochowNLP_SF_ENG_1 0.0489 0.0001 0.0000 0 0.0390 -0.0002 SoochowNLP_SF_ENG_2 0.0450 0.0015 0.0000 0 0.0335 0.0008 SoochowNLP_SF_ENG_3 0.0451 0.0017 0.0000 0 0.0272 -0.0039 SoochowNLP_SF_ENG_4 0.0452 0.0022 0.0000 0 0.0322 0.0013 SoochowNLP_SF_ENG_5 0.0448 0.0021 0.0000 0 0.0311 0.0009 Stanford_KB_ENG_1 0.2401 0.0031 0.1554 -0.0096 0.2121 -0.0024 Stanford_KB_ENG_2 0.1957 -0.0181 0.1220 -0.0128 0.1725 -0.0164 Stanford_KB_ENG_3 0.2699 -0.0077 0.1154 -0.0046 0.2009 -0.0071 Stanford_SF_ENG_1 0.2324 -0.0055 0.1186 -0.0051 0.1938 -0.0056 Stanford_SF_ENG_2 0.2481 -0.0099 0.1191 -0.0026 0.1949 -0.0066 Stanford_SF_ENG_3 0.3003 0.0001 0.0732 0.0038 0.1691 0.0047 Stanford_SF_ENG_4 0.2063 -0.0134 0.1246 -0.0127 0.1806 -0.0132 Stanford_SF_ENG_5 0.2317 -0.0039 0.0860 0 0.1844 -0.0033 summa_KB_ENG_1 0.0293 -0.0073 0.0000 0 0.0198 -0.0049 summa_KB_ENG_2 0.0645 -0.0099 0.0149 -0.0074 0.0484 -0.0092 summa_KB_ENG_5 0.0338 -0.0074 0.0152 -0.0076 0.0276 -0.0074 UMass_IESL_KB_ENG_1 0.1954 -0.0104 0.0468 -0.0015 0.1214 -0.0065 UMass_IESL_KB_ENG_2 0.1801 -0.0108 0.0544 -0.0032 0.1262 -0.0084 UMass_IESL_KB_ENG_3 0.2075 -0.0099 0.1361 -0.002 0.1841 -0.0072 UMass_IESL_KB_ENG_4 0.1685 -0.0112 0.0802 0.0019 0.1362 -0.0065 UMass_IESL_KB_ENG_5 0.1842 -0.0091 0.0296 -0.0008 0.0829 -0.0045 UMass_IESL_SF_ENG_1 0.2222 -0.016 0.0226 0 0.1796 -0.0142 UMass_IESL_SF_ENG_2 0.2215 -0.0211 0.0336 0 0.1759 -0.017 UMass_IESL_SF_ENG_3 0.2166 -0.0048 0.1040 0.0139 0.1731 0.0043 UMass_IESL_SF_ENG_4 0.2043 -0.0206 0.0226 0 0.1625 -0.0172 UMass_IESL_SF_ENG_5 0.2015 -0.0192 0.0226 0 0.1563 -0.0154 UNIST_SAIL_SF_ENG_1 0.0414 -0.0016 0.0000 0 0.0136 -0.0009 UNIST_SAIL_SF_ENG_2 0.0459 -0.0036 0.0000 0 0.0340 -0.003 UNIST_SAIL_SF_ENG_3 0.0534 -0.0012 0.0000 0 0.0391 -0.0012 UNIST_SAIL_SF_ENG_4 0.0840 0.0019 0.0000 0 0.0653 0.0008 Average change in F -0.00527429 -0.00104 -0.00404714 MEAN macro-average F1 and change in F1 after applying SFV filter, at each hop level: Cold Start run hop0_F Post-filter hop1_F Post-filter ALL_F Post-filter BBN_KB_ENG_1 0.2276 -0.0038 0.1128 -0.0079 0.1822 -0.0053 BBN_KB_ENG_2 0.2231 -0.0089 0.0966 -0.0078 0.1731 -0.0085 BBN_KB_ENG_3 0.2322 -0.0039 0.1124 -0.0078 0.1849 -0.0055 BBN_KB_ENG_4 0.2292 -0.0039 0.1127 -0.0078 0.1832 -0.0054 BBN_KB_ENG_5 0.2263 -0.0038 0.1076 -0.0078 0.1794 -0.0054 CMUML_SF_ENG_1 0.0333 -0.0059 0.0000 0 0.0202 -0.0036 CMUML_SF_ENG_2 0.0256 -0.0059 0.0000 0 0.0155 -0.0036 CMUML_SF_ENG_3 0.0030 -0.0026 0.0000 0 0.0018 -0.0015 CMUML_SF_ENG_4 0.0299 -0.0034 0.0000 0 0.0181 -0.0021 CMUML_SF_ENG_5 0.0333 -0.0059 0.0000 0 0.0202 -0.0036 DCD_SF_SF_ENG_1 0.1007 -0.005 0.0296 -0.0078 0.0726 -0.0061 DCD_SF_SF_ENG_2 0.1160 -0.0029 0.0270 0 0.0809 -0.0018 doughnutPRIS_SF_ENG_4 0.0859 -0.0058 0.0000 0 0.0520 -0.0035 doughnutPRIS_SF_ENG_5 0.1284 -0.0046 0.0000 0 0.0777 -0.0028 hltcoe_KB_ENG_1 0.2046 -0.0058 0.1108 -0.0053 0.1675 -0.0056 hltcoe_KB_ENG_2 0.1992 -0.0066 0.0886 -0.0052 0.1555 -0.006 hltcoe_KB_ENG_3 0.1972 -0.0066 0.0886 -0.0052 0.1543 -0.006 hltcoe_KB_ENG_4 0.2022 -0.001 0.1045 -0.0052 0.1636 -0.0027 hltcoe_KB_ENG_5 0.2072 -0.0059 0.1108 -0.0053 0.1691 -0.0056 ICTCAS_OKN_KB_ENG_1 0.1240 -0.0015 0.0252 0 0.0850 -0.0009 ICTCAS_OKN_KB_ENG_2 0.1256 -0.0016 0.0198 0 0.0838 -0.001 ICTCAS_OKN_KB_ENG_3 0.1002 -0.002 0.0253 0 0.0706 -0.0012 ICTCAS_OKN_KB_ENG_4 0.1134 -0.0032 0.0140 0.0026 0.0741 -0.0009 IRTSX_SF_ENG_1 0.0033 0 0.0000 0 0.0020 0 IRTSX_SF_ENG_2 0.0037 0 0.0000 0 0.0022 0 IRTSX_SF_ENG_3 0.0046 0 0.0000 0 0.0028 0 lilian_KB_ENG_1 0.0270 -0.0007 0.0000 0 0.0163 -0.0004 lilian_KB_ENG_2 0.0270 -0.0007 0.0000 0 0.0163 -0.0004 lilian_KB_ENG_3 0.0275 -0.0007 0.0000 0 0.0166 -0.0004 MSR_SF_ENG_1 0.0313 0 0.0000 0 0.0189 0 MSR_SF_ENG_2 0.0427 -0.0069 0.0000 0 0.0258 -0.0041 MSR_SF_ENG_3 0.0223 0 0.0000 0 0.0135 0 MSR_SF_ENG_4 0.0735 -0.0021 0.0215 0 0.0529 -0.0012 MSR_SF_ENG_5 0.0210 -0.0009 0.0000 0 0.0127 -0.0005 NAIST_CL_SF_ENG_1 0.0935 -0.0051 0.0199 -0.0052 0.0644 -0.0051 NAIST_CL_SF_ENG_2 0.0935 -0.0051 0.0000 0 0.0566 -0.0031 NAIST_CL_SF_ENG_3 0.0935 -0.0051 0.0000 0 0.0566 -0.0031 RPI_BLENDER_SF_ENG_1 0.1858 0 0.0418 0 0.1289 0 RPI_BLENDER_SF_ENG_2 0.1551 0 0.0411 0 0.1101 0 RPI_BLENDER_SF_ENG_3 0.1541 0 0.0405 0 0.1092 0 SoochowNLP_SF_ENG_1 0.0251 0.0008 0.0000 0 0.0152 0.0004 SoochowNLP_SF_ENG_2 0.0206 0.0006 0.0000 0 0.0125 0.0003 SoochowNLP_SF_ENG_3 0.0206 0.0006 0.0000 0 0.0125 0.0003 SoochowNLP_SF_ENG_4 0.0206 0.0006 0.0000 0 0.0125 0.0003 SoochowNLP_SF_ENG_5 0.0206 0.0006 0.0000 0 0.0125 0.0003 Stanford_KB_ENG_1 0.2149 -0.0094 0.1087 -0.0156 0.1730 -0.0119 Stanford_KB_ENG_2 0.1519 -0.0187 0.0700 -0.0156 0.1195 -0.0174 Stanford_KB_ENG_3 0.2553 -0.0086 0.1526 -0.0156 0.2147 -0.0113 Stanford_SF_ENG_1 0.1876 -0.0039 0.0788 -0.0078 0.1446 -0.0054 Stanford_SF_ENG_2 0.2334 -0.0128 0.1125 -0.0156 0.1856 -0.0139 Stanford_SF_ENG_3 0.2640 -0.0082 0.1638 -0.0156 0.2244 -0.0111 Stanford_SF_ENG_4 0.1733 -0.014 0.0615 -0.0156 0.1291 -0.0146 Stanford_SF_ENG_5 0.1876 -0.0039 0.0379 0 0.1285 -0.0024 summa_KB_ENG_1 0.0068 -0.0011 0.0000 0 0.0041 -0.0006 summa_KB_ENG_2 0.0202 -0.002 0.0156 -0.0078 0.0184 -0.0043 summa_KB_ENG_5 0.0116 -0.0013 0.0156 -0.0078 0.0132 -0.0039 UMass_IESL_KB_ENG_1 0.1375 -0.0032 0.0672 -0.0078 0.1097 -0.005 UMass_IESL_KB_ENG_2 0.1240 -0.0047 0.0581 -0.0078 0.0980 -0.006 UMass_IESL_KB_ENG_3 0.1204 -0.0025 0.1093 -0.0078 0.1160 -0.0046 UMass_IESL_KB_ENG_4 0.1266 -0.0034 0.0587 0 0.0998 -0.0021 UMass_IESL_KB_ENG_5 0.1375 -0.0041 0.0834 -0.0078 0.1161 -0.0056 UMass_IESL_SF_ENG_1 0.1842 -0.0172 0.0102 0 0.1154 -0.0104 UMass_IESL_SF_ENG_2 0.1648 -0.0166 0.0258 0 0.1099 -0.0101 UMass_IESL_SF_ENG_3 0.1598 -0.0123 0.0864 0.0026 0.1308 -0.0064 UMass_IESL_SF_ENG_4 0.1624 -0.0216 0.0102 0 0.1023 -0.0131 UMass_IESL_SF_ENG_5 0.1637 -0.0148 0.0102 0 0.1031 -0.009 UNIST_SAIL_SF_ENG_1 0.0263 -0.0034 0.0000 0 0.0159 -0.002 UNIST_SAIL_SF_ENG_2 0.0270 -0.0032 0.0000 0 0.0164 -0.002 UNIST_SAIL_SF_ENG_3 0.0307 -0.0034 0.0000 0 0.0186 -0.0021 UNIST_SAIL_SF_ENG_4 0.0491 -0.0033 0.0000 0 0.0297 -0.002 Average change in F -0.00460286 -0.00316143 -0.00403571 ************************************************************* Run ID: UI_CCG2 Did the run access the live Web during the evaluation window: No Did this run judge each candidate slot filler independently of all other candidate slot fillers in the evaluation dataset: Yes Did this run judge candidate slot fillers for each slot-filling run independently of all other slot-filling runs in the evaluation dataset: Yes Did this run judge candidate slot fillers for each slot-filling team independently of all other slot-filling teams in the evaluation dataset: Yes Did this run make use of the slot filler or justification offsets provided for each candidate slot filler: Yes Did this run make use of the confidence values provided for each candidate slot filler: No Did this run make use of the system profiles for the slot filling runs: No Did this run make use of the preliminary assessments provided for some of the slot filler candidates: No CSSF micro-average F1 and change in F1 after applying SFV filter, at each hop level: Cold Start run hop0_F Post-filter hop1_F Post-filter ALL_F Post-filter BBN_KB_ENG_1 0.3174 -0.0302 0.1737 -0.0005 0.2693 -0.0208 BBN_KB_ENG_2 0.3040 -0.0263 0.1650 -0.0007 0.2571 -0.018 BBN_KB_ENG_3 0.3150 -0.0304 0.1724 -0.0006 0.2651 -0.0204 BBN_KB_ENG_4 0.3188 -0.0302 0.1734 -0.0005 0.2703 -0.0208 BBN_KB_ENG_5 0.3041 -0.0244 0.1529 -0.0008 0.2475 -0.0161 CMUML_SF_ENG_1 0.0440 -0.0107 0.0000 0 0.0291 -0.007 CMUML_SF_ENG_2 0.0371 -0.0098 0.0000 0 0.0249 -0.0067 CMUML_SF_ENG_3 0.0057 -0.0028 0.0000 0 0.0038 -0.0019 CMUML_SF_ENG_4 0.0369 -0.0083 0.0000 0 0.0246 -0.0055 CMUML_SF_ENG_5 0.0440 -0.0107 0.0000 0 0.0291 -0.007 DCD_SF_SF_ENG_1 0.0804 -0.0036 0.0198 0.0019 0.0588 -0.0014 DCD_SF_SF_ENG_2 0.0904 -0.0076 0.0186 0.002 0.0647 -0.0039 doughnutPRIS_SF_ENG_4 0.0865 -0.0254 0.0000 0 0.0648 -0.0198 doughnutPRIS_SF_ENG_5 0.1143 -0.0399 0.0000 0 0.0894 -0.0326 hltcoe_KB_ENG_1 0.2949 -0.0304 0.1202 0.0018 0.2300 -0.0162 hltcoe_KB_ENG_2 0.2953 -0.0282 0.1180 0.0107 0.2311 -0.012 hltcoe_KB_ENG_3 0.2976 -0.03 0.1173 0.0107 0.2320 -0.013 hltcoe_KB_ENG_4 0.2776 -0.0288 0.0830 0.0019 0.1959 -0.0113 hltcoe_KB_ENG_5 0.2966 -0.0318 0.1202 0.0018 0.2311 -0.0172 ICTCAS_OKN_KB_ENG_1 0.1487 -0.022 0.0441 -0.0063 0.1101 -0.0166 ICTCAS_OKN_KB_ENG_2 0.1490 -0.018 0.0374 0.0003 0.1081 -0.0116 ICTCAS_OKN_KB_ENG_3 0.1186 -0.0099 0.0526 -0.0041 0.0960 -0.0081 ICTCAS_OKN_KB_ENG_4 0.1421 -0.015 0.0315 -0.0073 0.1041 -0.0124 IRTSX_SF_ENG_1 0.0062 0.0001 0.0000 0 0.0038 0.0001 IRTSX_SF_ENG_2 0.0064 0 0.0000 0 0.0038 0 IRTSX_SF_ENG_3 0.0072 0 0.0000 0 0.0051 0 lilian_KB_ENG_1 0.0452 -0.0233 0.0000 0 0.0311 -0.0162 lilian_KB_ENG_2 0.0465 -0.024 0.0000 0 0.0317 -0.0165 lilian_KB_ENG_3 0.0468 -0.0242 0.0000 0 0.0318 -0.0166 MSR_SF_ENG_1 0.0275 -0.0008 0.0000 0 0.0150 -0.0003 MSR_SF_ENG_2 0.0297 0 0.0000 0 0.0198 0 MSR_SF_ENG_3 0.0201 -0.0006 0.0000 0 0.0122 -0.0003 MSR_SF_ENG_4 0.0635 -0.0197 0.0102 -0.0033 0.0377 -0.0117 MSR_SF_ENG_5 0.0176 -0.0022 0.0000 0 0.0083 -0.0009 NAIST_CL_SF_ENG_1 0.0589 -0.0114 0.0100 0.0009 0.0146 -0.0002 NAIST_CL_SF_ENG_2 0.0589 -0.0114 0.0000 0 0.0438 -0.0091 NAIST_CL_SF_ENG_3 0.0589 -0.0114 0.0000 0 0.0433 -0.0088 RPI_BLENDER_SF_ENG_1 0.1904 -0.016 0.1050 -0.0141 0.1644 -0.0148 RPI_BLENDER_SF_ENG_2 0.1721 -0.017 0.0960 -0.0149 0.1487 -0.0158 RPI_BLENDER_SF_ENG_3 0.1700 -0.0169 0.0950 -0.0146 0.1470 -0.0156 SoochowNLP_SF_ENG_1 0.0451 -0.0075 0.0000 0 0.0363 -0.0064 SoochowNLP_SF_ENG_2 0.0372 -0.0086 0.0000 0 0.0275 -0.0067 SoochowNLP_SF_ENG_3 0.0372 -0.0086 0.0000 0 0.0192 -0.0044 SoochowNLP_SF_ENG_4 0.0372 -0.0086 0.0000 0 0.0262 -0.0064 SoochowNLP_SF_ENG_5 0.0372 -0.0086 0.0000 0 0.0254 -0.0063 Stanford_KB_ENG_1 0.2406 -0.0408 0.1466 -0.0328 0.2100 -0.0381 Stanford_KB_ENG_2 0.1926 -0.049 0.1039 -0.029 0.1648 -0.043 Stanford_KB_ENG_3 0.2633 -0.0317 0.1190 -0.0214 0.2018 -0.027 Stanford_SF_ENG_1 0.2310 -0.042 0.1197 -0.0298 0.1945 -0.038 Stanford_SF_ENG_2 0.2444 -0.05 0.1241 -0.0343 0.1971 -0.042 Stanford_SF_ENG_3 0.2846 -0.0367 0.0819 -0.0056 0.1751 -0.0124 Stanford_SF_ENG_4 0.2044 -0.0588 0.1061 -0.0458 0.1735 -0.055 Stanford_SF_ENG_5 0.2310 -0.042 0.0798 -0.0025 0.1827 -0.0301 summa_KB_ENG_1 0.0255 -0.0056 0.0000 0 0.0171 -0.0037 summa_KB_ENG_2 0.0584 -0.0298 0.0114 -0.0114 0.0430 -0.0238 summa_KB_ENG_5 0.0261 -0.0173 0.0116 -0.0058 0.0213 -0.0135 UMass_IESL_KB_ENG_1 0.1844 -0.0425 0.0628 -0.0111 0.1252 -0.025 UMass_IESL_KB_ENG_2 0.1704 -0.0397 0.0687 -0.0117 0.1267 -0.0273 UMass_IESL_KB_ENG_3 0.2076 -0.0431 0.1480 -0.0178 0.1878 -0.0349 UMass_IESL_KB_ENG_4 0.1624 -0.0466 0.1061 -0.0183 0.1423 -0.0354 UMass_IESL_KB_ENG_5 0.1732 -0.0419 0.0399 -0.0096 0.0885 -0.0208 UMass_IESL_SF_ENG_1 0.2207 -0.0414 0.0282 0 0.1794 -0.036 UMass_IESL_SF_ENG_2 0.2224 -0.0531 0.0384 -0.0142 0.1751 -0.0462 UMass_IESL_SF_ENG_3 0.2092 -0.0445 0.0954 -0.0145 0.1668 -0.0317 UMass_IESL_SF_ENG_4 0.2061 -0.046 0.0282 0 0.1643 -0.0378 UMass_IESL_SF_ENG_5 0.2070 -0.0561 0.0285 0.0001 0.1627 -0.0446 UNIST_SAIL_SF_ENG_1 0.0438 -0.0066 0.0000 0 0.0109 -0.0017 UNIST_SAIL_SF_ENG_2 0.0476 -0.0059 0.0000 0 0.0352 -0.0044 UNIST_SAIL_SF_ENG_3 0.0583 -0.0061 0.0000 0 0.0426 -0.0045 UNIST_SAIL_SF_ENG_4 0.0788 -0.0012 0.0000 0 0.0610 -0.0016 Average change in F -0.0224786 -0.00501714 -0.0162243 MAX micro-average F1 and change in F1 after applying SFV filter, at each hop level: Cold Start run hop0_F Post-filter hop1_F Post-filter ALL_F Post-filter BBN_KB_ENG_1 0.3358 -0.0315 0.1659 -0.0009 0.2781 -0.0215 BBN_KB_ENG_2 0.3234 -0.0301 0.1542 -0.0011 0.2655 -0.0203 BBN_KB_ENG_3 0.3313 -0.0312 0.1648 -0.0008 0.2716 -0.0208 BBN_KB_ENG_4 0.3375 -0.0326 0.1655 -0.0009 0.2791 -0.0221 BBN_KB_ENG_5 0.3253 -0.0271 0.1442 -0.0011 0.2550 -0.0177 CMUML_SF_ENG_1 0.0462 -0.0102 0.0000 0 0.0309 -0.0069 CMUML_SF_ENG_2 0.0409 -0.0092 0.0000 0 0.0276 -0.0063 CMUML_SF_ENG_3 0.0074 -0.0037 0.0000 0 0.0049 -0.0024 CMUML_SF_ENG_4 0.0370 -0.0072 0.0000 0 0.0248 -0.0048 CMUML_SF_ENG_5 0.0463 -0.0105 0.0000 0 0.0310 -0.0072 DCD_SF_SF_ENG_1 0.0889 -0.0058 0.0219 0.0023 0.0652 -0.0027 DCD_SF_SF_ENG_2 0.1015 -0.0117 0.0208 0.0023 0.0728 -0.0063 doughnutPRIS_SF_ENG_4 0.0847 -0.0227 0.0000 0 0.0631 -0.0177 doughnutPRIS_SF_ENG_5 0.1138 -0.0335 0.0000 0 0.0887 -0.0276 hltcoe_KB_ENG_1 0.2912 -0.0304 0.1082 -0.002 0.2222 -0.0167 hltcoe_KB_ENG_2 0.2887 -0.0257 0.1014 0.0086 0.2204 -0.0103 hltcoe_KB_ENG_3 0.2891 -0.0296 0.1000 0.0085 0.2191 -0.0125 hltcoe_KB_ENG_4 0.2735 -0.0309 0.0734 -0.002 0.1872 -0.0126 hltcoe_KB_ENG_5 0.2925 -0.0321 0.1082 -0.002 0.2232 -0.0178 ICTCAS_OKN_KB_ENG_1 0.1507 -0.0167 0.0445 -0.0041 0.1120 -0.0125 ICTCAS_OKN_KB_ENG_2 0.1521 -0.0144 0.0401 0.0003 0.1115 -0.0095 ICTCAS_OKN_KB_ENG_3 0.1441 -0.0096 0.0656 -0.0052 0.1166 -0.0082 ICTCAS_OKN_KB_ENG_4 0.1471 -0.0127 0.0216 -0.005 0.1055 -0.0101 IRTSX_SF_ENG_1 0.0081 0.0002 0.0000 0 0.0050 0.0001 IRTSX_SF_ENG_2 0.0088 -0.0004 0.0000 0 0.0059 -0.0011 IRTSX_SF_ENG_3 0.0097 0.0001 0.0000 0 0.0068 0 lilian_KB_ENG_1 0.0446 -0.0233 0.0000 0 0.0308 -0.0162 lilian_KB_ENG_2 0.0462 -0.0243 0.0000 0 0.0316 -0.0167 lilian_KB_ENG_3 0.0467 -0.0247 0.0000 0 0.0318 -0.0169 MSR_SF_ENG_1 0.0284 -0.0013 0.0000 0 0.0153 -0.0006 MSR_SF_ENG_2 0.0343 0 0.0000 0 0.0228 0 MSR_SF_ENG_3 0.0247 -0.0007 0.0000 0 0.0150 -0.0005 MSR_SF_ENG_4 0.0661 -0.021 0.0117 -0.0026 0.0393 -0.0118 MSR_SF_ENG_5 0.0207 -0.003 0.0000 0 0.0096 -0.0012 NAIST_CL_SF_ENG_1 0.0535 -0.0081 0.0092 0.0013 0.0138 0.0005 NAIST_CL_SF_ENG_2 0.0534 -0.0081 0.0000 0 0.0397 -0.0065 NAIST_CL_SF_ENG_3 0.0533 -0.0079 0.0000 0 0.0391 -0.0061 RPI_BLENDER_SF_ENG_1 0.1888 -0.0187 0.0855 -0.02 0.1575 -0.0178 RPI_BLENDER_SF_ENG_2 0.1647 -0.0199 0.0795 -0.0209 0.1385 -0.0192 RPI_BLENDER_SF_ENG_3 0.1628 -0.0202 0.0789 -0.0211 0.1371 -0.0196 SoochowNLP_SF_ENG_1 0.0489 -0.0094 0.0000 0 0.0390 -0.0079 SoochowNLP_SF_ENG_2 0.0450 -0.0111 0.0000 0 0.0335 -0.0087 SoochowNLP_SF_ENG_3 0.0451 -0.0104 0.0000 0 0.0272 -0.0053 SoochowNLP_SF_ENG_4 0.0452 -0.0103 0.0000 0 0.0322 -0.0079 SoochowNLP_SF_ENG_5 0.0448 -0.011 0.0000 0 0.0311 -0.0081 Stanford_KB_ENG_1 0.2401 -0.0379 0.1554 -0.0322 0.2121 -0.0358 Stanford_KB_ENG_2 0.1957 -0.0459 0.1220 -0.0311 0.1725 -0.0414 Stanford_KB_ENG_3 0.2699 -0.0079 0.1154 -0.0194 0.2009 -0.0165 Stanford_SF_ENG_1 0.2324 -0.0408 0.1186 -0.0279 0.1938 -0.0364 Stanford_SF_ENG_2 0.2481 -0.0515 0.1191 -0.0356 0.1949 -0.0424 Stanford_SF_ENG_3 0.3003 -0.0374 0.0732 -0.0051 0.1691 -0.0094 Stanford_SF_ENG_4 0.2063 -0.0627 0.1246 -0.0529 0.1806 -0.0599 Stanford_SF_ENG_5 0.2317 -0.0398 0.0860 -0.0036 0.1844 -0.0288 summa_KB_ENG_1 0.0293 -0.0072 0.0000 0 0.0198 -0.0048 summa_KB_ENG_2 0.0645 -0.0385 0.0149 -0.0149 0.0484 -0.0309 summa_KB_ENG_5 0.0338 -0.0224 0.0152 -0.0075 0.0276 -0.0174 UMass_IESL_KB_ENG_1 0.1954 -0.0479 0.0468 -0.0094 0.1214 -0.0253 UMass_IESL_KB_ENG_2 0.1801 -0.0462 0.0544 -0.0132 0.1262 -0.0312 UMass_IESL_KB_ENG_3 0.2075 -0.0454 0.1361 -0.0243 0.1841 -0.0386 UMass_IESL_KB_ENG_4 0.1685 -0.0521 0.0802 -0.0157 0.1362 -0.0362 UMass_IESL_KB_ENG_5 0.1842 -0.0472 0.0296 -0.0095 0.0829 -0.0218 UMass_IESL_SF_ENG_1 0.2222 -0.0425 0.0226 0 0.1796 -0.0371 UMass_IESL_SF_ENG_2 0.2215 -0.0532 0.0336 -0.0133 0.1759 -0.0465 UMass_IESL_SF_ENG_3 0.2166 -0.0486 0.1040 -0.0164 0.1731 -0.0341 UMass_IESL_SF_ENG_4 0.2043 -0.0444 0.0226 0 0.1625 -0.0373 UMass_IESL_SF_ENG_5 0.2015 -0.0588 0.0226 0.0001 0.1563 -0.0465 UNIST_SAIL_SF_ENG_1 0.0414 -0.0055 0.0000 0 0.0136 -0.0019 UNIST_SAIL_SF_ENG_2 0.0459 -0.0073 0.0000 0 0.0340 -0.0056 UNIST_SAIL_SF_ENG_3 0.0534 -0.0055 0.0000 0 0.0391 -0.004 UNIST_SAIL_SF_ENG_4 0.0840 0.0049 0.0000 0 0.0653 0.0021 Average change in F -0.0227757 -0.00569 -0.0164814 MEAN macro-average F1 and change in F1 after applying SFV filter, at each hop level: Cold Start run hop0_F Post-filter hop1_F Post-filter ALL_F Post-filter BBN_KB_ENG_1 0.2276 -0.032 0.1128 -0.0079 0.1822 -0.0224 BBN_KB_ENG_2 0.2231 -0.0311 0.0966 -0.0078 0.1731 -0.0219 BBN_KB_ENG_3 0.2322 -0.0323 0.1124 -0.0078 0.1849 -0.0227 BBN_KB_ENG_4 0.2292 -0.032 0.1127 -0.0078 0.1832 -0.0225 BBN_KB_ENG_5 0.2263 -0.0274 0.1076 -0.0078 0.1794 -0.0197 CMUML_SF_ENG_1 0.0333 -0.0093 0.0000 0 0.0202 -0.0057 CMUML_SF_ENG_2 0.0256 -0.0093 0.0000 0 0.0155 -0.0057 CMUML_SF_ENG_3 0.0030 -0.0026 0.0000 0 0.0018 -0.0015 CMUML_SF_ENG_4 0.0299 -0.0068 0.0000 0 0.0181 -0.0041 CMUML_SF_ENG_5 0.0333 -0.0093 0.0000 0 0.0202 -0.0057 DCD_SF_SF_ENG_1 0.1007 -0.0046 0.0296 0 0.0726 -0.0028 DCD_SF_SF_ENG_2 0.1160 -0.0183 0.0270 0 0.0809 -0.0111 doughnutPRIS_SF_ENG_4 0.0859 -0.0279 0.0000 0 0.0520 -0.0169 doughnutPRIS_SF_ENG_5 0.1284 -0.0462 0.0000 0 0.0777 -0.028 hltcoe_KB_ENG_1 0.2046 -0.0213 0.1108 -0.0365 0.1675 -0.0272 hltcoe_KB_ENG_2 0.1992 -0.0203 0.0886 -0.013 0.1555 -0.0174 hltcoe_KB_ENG_3 0.1972 -0.0218 0.0886 -0.013 0.1543 -0.0183 hltcoe_KB_ENG_4 0.2022 -0.0285 0.1045 -0.0341 0.1636 -0.0307 hltcoe_KB_ENG_5 0.2072 -0.0264 0.1108 -0.0365 0.1691 -0.0304 ICTCAS_OKN_KB_ENG_1 0.1240 -0.0201 0.0252 -0.0052 0.0850 -0.0143 ICTCAS_OKN_KB_ENG_2 0.1256 -0.0191 0.0198 0 0.0838 -0.0115 ICTCAS_OKN_KB_ENG_3 0.1002 -0.0127 0.0253 -0.0026 0.0706 -0.0087 ICTCAS_OKN_KB_ENG_4 0.1134 -0.0158 0.0140 -0.0053 0.0741 -0.0116 IRTSX_SF_ENG_1 0.0033 0 0.0000 0 0.0020 0 IRTSX_SF_ENG_2 0.0037 0 0.0000 0 0.0022 0 IRTSX_SF_ENG_3 0.0046 0 0.0000 0 0.0028 0 lilian_KB_ENG_1 0.0270 -0.0155 0.0000 0 0.0163 -0.0094 lilian_KB_ENG_2 0.0270 -0.0155 0.0000 0 0.0163 -0.0094 lilian_KB_ENG_3 0.0275 -0.0155 0.0000 0 0.0166 -0.0093 MSR_SF_ENG_1 0.0313 -0.0035 0.0000 0 0.0189 -0.0021 MSR_SF_ENG_2 0.0427 0 0.0000 0 0.0258 0 MSR_SF_ENG_3 0.0223 -0.0002 0.0000 0 0.0135 -0.0001 MSR_SF_ENG_4 0.0735 -0.0256 0.0215 -0.0065 0.0529 -0.018 MSR_SF_ENG_5 0.0210 -0.0054 0.0000 0 0.0127 -0.0032 NAIST_CL_SF_ENG_1 0.0935 -0.0204 0.0199 0 0.0644 -0.0123 NAIST_CL_SF_ENG_2 0.0935 -0.0204 0.0000 0 0.0566 -0.0124 NAIST_CL_SF_ENG_3 0.0935 -0.0204 0.0000 0 0.0566 -0.0124 RPI_BLENDER_SF_ENG_1 0.1858 -0.0124 0.0418 -0.0208 0.1289 -0.0157 RPI_BLENDER_SF_ENG_2 0.1551 -0.0133 0.0411 -0.0208 0.1101 -0.0163 RPI_BLENDER_SF_ENG_3 0.1541 -0.0134 0.0405 -0.0209 0.1092 -0.0163 SoochowNLP_SF_ENG_1 0.0251 -0.0092 0.0000 0 0.0152 -0.0056 SoochowNLP_SF_ENG_2 0.0206 -0.0073 0.0000 0 0.0125 -0.0045 SoochowNLP_SF_ENG_3 0.0206 -0.0073 0.0000 0 0.0125 -0.0045 SoochowNLP_SF_ENG_4 0.0206 -0.0073 0.0000 0 0.0125 -0.0045 SoochowNLP_SF_ENG_5 0.0206 -0.0073 0.0000 0 0.0125 -0.0045 Stanford_KB_ENG_1 0.2149 -0.0556 0.1087 -0.0323 0.1730 -0.0464 Stanford_KB_ENG_2 0.1519 -0.0517 0.0700 -0.0222 0.1195 -0.04 Stanford_KB_ENG_3 0.2553 -0.0592 0.1526 -0.0472 0.2147 -0.0544 Stanford_SF_ENG_1 0.1876 -0.0522 0.0788 -0.0312 0.1446 -0.0439 Stanford_SF_ENG_2 0.2334 -0.0663 0.1125 -0.0415 0.1856 -0.0565 Stanford_SF_ENG_3 0.2640 -0.0698 0.1638 -0.0546 0.2244 -0.0638 Stanford_SF_ENG_4 0.1733 -0.0636 0.0615 -0.03 0.1291 -0.0503 Stanford_SF_ENG_5 0.1876 -0.0522 0.0379 -0.0031 0.1285 -0.0328 summa_KB_ENG_1 0.0068 -0.0011 0.0000 0 0.0041 -0.0006 summa_KB_ENG_2 0.0202 -0.011 0.0156 -0.0156 0.0184 -0.0128 summa_KB_ENG_5 0.0116 -0.0052 0.0156 -0.0078 0.0132 -0.0063 UMass_IESL_KB_ENG_1 0.1375 -0.0412 0.0672 -0.0219 0.1097 -0.0335 UMass_IESL_KB_ENG_2 0.1240 -0.0386 0.0581 -0.0175 0.0980 -0.0303 UMass_IESL_KB_ENG_3 0.1204 -0.0321 0.1093 -0.0201 0.1160 -0.0273 UMass_IESL_KB_ENG_4 0.1266 -0.0356 0.0587 -0.0219 0.0998 -0.0302 UMass_IESL_KB_ENG_5 0.1375 -0.0434 0.0834 -0.0282 0.1161 -0.0374 UMass_IESL_SF_ENG_1 0.1842 -0.0535 0.0102 0 0.1154 -0.0323 UMass_IESL_SF_ENG_2 0.1648 -0.0556 0.0258 -0.0156 0.1099 -0.0398 UMass_IESL_SF_ENG_3 0.1598 -0.0473 0.0864 -0.0292 0.1308 -0.0401 UMass_IESL_SF_ENG_4 0.1624 -0.0498 0.0102 0 0.1023 -0.0302 UMass_IESL_SF_ENG_5 0.1637 -0.0538 0.0102 0 0.1031 -0.0326 UNIST_SAIL_SF_ENG_1 0.0263 -0.0077 0.0000 0 0.0159 -0.0046 UNIST_SAIL_SF_ENG_2 0.0270 -0.0076 0.0000 0 0.0164 -0.0047 UNIST_SAIL_SF_ENG_3 0.0307 -0.0077 0.0000 0 0.0186 -0.0047 UNIST_SAIL_SF_ENG_4 0.0491 -0.0075 0.0000 0 0.0297 -0.0045 Average change in F -0.0237757 -0.00991714 -0.0183043 ************************************************************* Run ID: UI_CCG3 Did the run access the live Web during the evaluation window: No Did this run judge each candidate slot filler independently of all other candidate slot fillers in the evaluation dataset: Yes Did this run judge candidate slot fillers for each slot-filling run independently of all other slot-filling runs in the evaluation dataset: Yes Did this run judge candidate slot fillers for each slot-filling team independently of all other slot-filling teams in the evaluation dataset: Yes Did this run make use of the slot filler or justification offsets provided for each candidate slot filler: Yes Did this run make use of the confidence values provided for each candidate slot filler: No Did this run make use of the system profiles for the slot filling runs: No Did this run make use of the preliminary assessments provided for some of the slot filler candidates: No CSSF micro-average F1 and change in F1 after applying SFV filter, at each hop level: Cold Start run hop0_F Post-filter hop1_F Post-filter ALL_F Post-filter BBN_KB_ENG_1 0.3174 -0.0044 0.1737 -0.0035 0.2693 -0.0045 BBN_KB_ENG_2 0.3040 -0.0082 0.1650 -0.0036 0.2571 -0.0069 BBN_KB_ENG_3 0.3150 -0.0048 0.1724 -0.0033 0.2651 -0.0045 BBN_KB_ENG_4 0.3188 -0.0047 0.1734 -0.0035 0.2703 -0.0046 BBN_KB_ENG_5 0.3041 -0.0044 0.1529 -0.0029 0.2475 -0.0041 CMUML_SF_ENG_1 0.0440 -0.0051 0.0000 0 0.0291 -0.0035 CMUML_SF_ENG_2 0.0371 -0.0047 0.0000 0 0.0249 -0.0032 CMUML_SF_ENG_3 0.0057 -0.0028 0.0000 0 0.0038 -0.0019 CMUML_SF_ENG_4 0.0369 -0.0026 0.0000 0 0.0246 -0.0018 CMUML_SF_ENG_5 0.0440 -0.0051 0.0000 0 0.0291 -0.0035 DCD_SF_SF_ENG_1 0.0804 -0.0097 0.0198 -0.003 0.0588 -0.0073 DCD_SF_SF_ENG_2 0.0904 -0.0054 0.0186 0.0005 0.0647 -0.0033 doughnutPRIS_SF_ENG_4 0.0865 -0.0082 0.0000 0 0.0648 -0.0065 doughnutPRIS_SF_ENG_5 0.1143 -0.0051 0.0000 0 0.0894 -0.0045 hltcoe_KB_ENG_1 0.2949 -0.0115 0.1202 -0.0017 0.2300 -0.0085 hltcoe_KB_ENG_2 0.2953 -0.0125 0.1180 -0.0022 0.2311 -0.0095 hltcoe_KB_ENG_3 0.2976 -0.0124 0.1173 -0.0023 0.2320 -0.0095 hltcoe_KB_ENG_4 0.2776 -0.005 0.0830 0.0014 0.1959 -0.0012 hltcoe_KB_ENG_5 0.2966 -0.0114 0.1202 -0.0017 0.2311 -0.0085 ICTCAS_OKN_KB_ENG_1 0.1487 -0.0002 0.0441 0.0014 0.1101 0.0005 ICTCAS_OKN_KB_ENG_2 0.1490 -0.0001 0.0374 0.0011 0.1081 0.0005 ICTCAS_OKN_KB_ENG_3 0.1186 -0.0003 0.0526 0.0045 0.0960 0.0023 ICTCAS_OKN_KB_ENG_4 0.1421 -0.0002 0.0315 0.0015 0.1041 0.0009 IRTSX_SF_ENG_1 0.0062 0 0.0000 0 0.0038 0.0001 IRTSX_SF_ENG_2 0.0064 0 0.0000 0 0.0038 0 IRTSX_SF_ENG_3 0.0072 0 0.0000 0 0.0051 0 lilian_KB_ENG_1 0.0452 -0.0021 0.0000 0 0.0311 -0.0016 lilian_KB_ENG_2 0.0465 -0.0024 0.0000 0 0.0317 -0.0017 lilian_KB_ENG_3 0.0468 -0.0025 0.0000 0 0.0318 -0.0017 MSR_SF_ENG_1 0.0275 0.0004 0.0000 0 0.0150 0.0002 MSR_SF_ENG_2 0.0297 -0.0054 0.0000 0 0.0198 -0.0036 MSR_SF_ENG_3 0.0201 0.0001 0.0000 0 0.0122 0.0001 MSR_SF_ENG_4 0.0635 -0.0004 0.0102 0.0001 0.0377 -0.0001 MSR_SF_ENG_5 0.0176 -0.0008 0.0000 0 0.0083 -0.0003 NAIST_CL_SF_ENG_1 0.0589 -0.0003 0.0100 0.0005 0.0146 0.0005 NAIST_CL_SF_ENG_2 0.0589 -0.0003 0.0000 0 0.0438 -0.0006 NAIST_CL_SF_ENG_3 0.0589 -0.0003 0.0000 0 0.0433 -0.0005 RPI_BLENDER_SF_ENG_1 0.1904 0.0002 0.1050 0 0.1644 0.0001 RPI_BLENDER_SF_ENG_2 0.1721 0.0002 0.0960 0 0.1487 0.0001 RPI_BLENDER_SF_ENG_3 0.1700 0.0002 0.0950 0 0.1470 0.0001 SoochowNLP_SF_ENG_1 0.0451 0.0003 0.0000 0 0.0363 0 SoochowNLP_SF_ENG_2 0.0372 0.001 0.0000 0 0.0275 0.0006 SoochowNLP_SF_ENG_3 0.0372 0.001 0.0000 0 0.0192 0.0003 SoochowNLP_SF_ENG_4 0.0372 0.001 0.0000 0 0.0262 0.0006 SoochowNLP_SF_ENG_5 0.0372 0.001 0.0000 0 0.0254 0.0005 Stanford_KB_ENG_1 0.2406 -0.005 0.1466 -0.0075 0.2100 -0.0062 Stanford_KB_ENG_2 0.1926 -0.0147 0.1039 -0.0099 0.1648 -0.0134 Stanford_KB_ENG_3 0.2633 -0.0037 0.1190 -0.0038 0.2018 -0.0046 Stanford_SF_ENG_1 0.2310 -0.0034 0.1197 -0.0041 0.1945 -0.0038 Stanford_SF_ENG_2 0.2444 -0.0065 0.1241 -0.002 0.1971 -0.0046 Stanford_SF_ENG_3 0.2846 0.0004 0.0819 0.0043 0.1751 0.0046 Stanford_SF_ENG_4 0.2044 -0.0105 0.1061 -0.0098 0.1735 -0.0103 Stanford_SF_ENG_5 0.2310 -0.0032 0.0798 0 0.1827 -0.0026 summa_KB_ENG_1 0.0255 -0.0056 0.0000 0 0.0171 -0.0037 summa_KB_ENG_2 0.0584 -0.008 0.0114 -0.0057 0.0430 -0.0073 summa_KB_ENG_5 0.0261 -0.0057 0.0116 -0.0058 0.0213 -0.0058 UMass_IESL_KB_ENG_1 0.1844 -0.0073 0.0628 -0.0009 0.1252 -0.0047 UMass_IESL_KB_ENG_2 0.1704 -0.0082 0.0687 -0.0025 0.1267 -0.0063 UMass_IESL_KB_ENG_3 0.2076 -0.0071 0.1480 -0.0012 0.1878 -0.0052 UMass_IESL_KB_ENG_4 0.1624 -0.0089 0.1061 0.0018 0.1423 -0.0052 UMass_IESL_KB_ENG_5 0.1732 -0.0064 0.0399 -0.0004 0.0885 -0.0031 UMass_IESL_SF_ENG_1 0.2207 -0.0152 0.0282 0 0.1794 -0.0134 UMass_IESL_SF_ENG_2 0.2224 -0.0184 0.0384 0 0.1751 -0.015 UMass_IESL_SF_ENG_3 0.2092 -0.0036 0.0954 0.0102 0.1668 0.0032 UMass_IESL_SF_ENG_4 0.2061 -0.0198 0.0282 0 0.1643 -0.0164 UMass_IESL_SF_ENG_5 0.2070 -0.0179 0.0285 0 0.1627 -0.0146 UNIST_SAIL_SF_ENG_1 0.0438 -0.0012 0.0000 0 0.0109 -0.0005 UNIST_SAIL_SF_ENG_2 0.0476 -0.0007 0.0000 0 0.0352 -0.0008 UNIST_SAIL_SF_ENG_3 0.0583 -0.001 0.0000 0 0.0426 -0.0009 UNIST_SAIL_SF_ENG_4 0.0788 0.0007 0.0000 0 0.0610 0.0001 Average change in F -0.00455429 -0.000771429 -0.00343571 MAX micro-average F1 and change in F1 after applying SFV filter, at each hop level: Cold Start run hop0_F Post-filter hop1_F Post-filter ALL_F Post-filter BBN_KB_ENG_1 0.3358 -0.0071 0.1659 -0.0045 0.2781 -0.0064 BBN_KB_ENG_2 0.3234 -0.0093 0.1542 -0.0046 0.2655 -0.008 BBN_KB_ENG_3 0.3313 -0.006 0.1648 -0.0041 0.2716 -0.0057 BBN_KB_ENG_4 0.3375 -0.0084 0.1655 -0.0045 0.2791 -0.0071 BBN_KB_ENG_5 0.3253 -0.0054 0.1442 -0.0035 0.2550 -0.0052 CMUML_SF_ENG_1 0.0462 -0.0066 0.0000 0 0.0309 -0.0047 CMUML_SF_ENG_2 0.0409 -0.0059 0.0000 0 0.0276 -0.0042 CMUML_SF_ENG_3 0.0074 -0.0037 0.0000 0 0.0049 -0.0024 CMUML_SF_ENG_4 0.0370 -0.0034 0.0000 0 0.0248 -0.0022 CMUML_SF_ENG_5 0.0463 -0.0067 0.0000 0 0.0310 -0.0049 DCD_SF_SF_ENG_1 0.0889 -0.0107 0.0219 -0.0039 0.0652 -0.0082 DCD_SF_SF_ENG_2 0.1015 -0.0046 0.0208 0.0009 0.0728 -0.0024 doughnutPRIS_SF_ENG_4 0.0847 -0.0066 0.0000 0 0.0631 -0.0052 doughnutPRIS_SF_ENG_5 0.1138 -0.0036 0.0000 0 0.0887 -0.0033 hltcoe_KB_ENG_1 0.2912 -0.0098 0.1082 -0.0023 0.2222 -0.0077 hltcoe_KB_ENG_2 0.2887 -0.0105 0.1014 -0.003 0.2204 -0.0086 hltcoe_KB_ENG_3 0.2891 -0.0127 0.1000 -0.003 0.2191 -0.01 hltcoe_KB_ENG_4 0.2735 -0.0067 0.0734 0.001 0.1872 -0.0015 hltcoe_KB_ENG_5 0.2925 -0.0098 0.1082 -0.0023 0.2232 -0.0077 ICTCAS_OKN_KB_ENG_1 0.1507 -0.0015 0.0445 0.0011 0.1120 -0.0006 ICTCAS_OKN_KB_ENG_2 0.1521 -0.0018 0.0401 0.0008 0.1115 -0.0009 ICTCAS_OKN_KB_ENG_3 0.1441 0.0011 0.0656 0.0063 0.1166 0.0043 ICTCAS_OKN_KB_ENG_4 0.1471 -0.0023 0.0216 0.0009 0.1055 -0.0007 IRTSX_SF_ENG_1 0.0081 0.0007 0.0000 0 0.0050 0.001 IRTSX_SF_ENG_2 0.0088 0.0001 0.0000 0 0.0059 0 IRTSX_SF_ENG_3 0.0097 -0.0001 0.0000 0 0.0068 -0.0001 lilian_KB_ENG_1 0.0446 -0.0028 0.0000 0 0.0308 -0.002 lilian_KB_ENG_2 0.0462 -0.0032 0.0000 0 0.0316 -0.0022 lilian_KB_ENG_3 0.0467 -0.0035 0.0000 0 0.0318 -0.0024 MSR_SF_ENG_1 0.0284 0.0008 0.0000 0 0.0153 0.0004 MSR_SF_ENG_2 0.0343 -0.0071 0.0000 0 0.0228 -0.0047 MSR_SF_ENG_3 0.0247 0.0003 0.0000 0 0.0150 0 MSR_SF_ENG_4 0.0661 -0.0008 0.0117 0 0.0393 -0.0005 MSR_SF_ENG_5 0.0207 -0.001 0.0000 0 0.0096 -0.0004 NAIST_CL_SF_ENG_1 0.0535 -0.0013 0.0092 -0.0011 0.0138 -0.0017 NAIST_CL_SF_ENG_2 0.0534 -0.0011 0.0000 0 0.0397 -0.0012 NAIST_CL_SF_ENG_3 0.0533 -0.0011 0.0000 0 0.0391 -0.001 RPI_BLENDER_SF_ENG_1 0.1888 0.0017 0.0855 0.0009 0.1575 0.0015 RPI_BLENDER_SF_ENG_2 0.1647 0.0007 0.0795 0.0008 0.1385 0.0008 RPI_BLENDER_SF_ENG_3 0.1628 0.0002 0.0789 0.0006 0.1371 0.0004 SoochowNLP_SF_ENG_1 0.0489 -0.0002 0.0000 0 0.0390 -0.0004 SoochowNLP_SF_ENG_2 0.0450 0.0011 0.0000 0 0.0335 0.0006 SoochowNLP_SF_ENG_3 0.0451 0.002 0.0000 0 0.0272 -0.0033 SoochowNLP_SF_ENG_4 0.0452 0.001 0.0000 0 0.0322 0.0005 SoochowNLP_SF_ENG_5 0.0448 0.0019 0.0000 0 0.0311 0.0009 Stanford_KB_ENG_1 0.2401 0.0034 0.1554 -0.0096 0.2121 -0.0022 Stanford_KB_ENG_2 0.1957 -0.0159 0.1220 -0.0128 0.1725 -0.015 Stanford_KB_ENG_3 0.2699 0.0161 0.1154 -0.0046 0.2009 0.0022 Stanford_SF_ENG_1 0.2324 -0.0055 0.1186 -0.0051 0.1938 -0.0056 Stanford_SF_ENG_2 0.2481 -0.0102 0.1191 -0.0026 0.1949 -0.0068 Stanford_SF_ENG_3 0.3003 -0.0023 0.0732 0.0039 0.1691 0.0042 Stanford_SF_ENG_4 0.2063 -0.0134 0.1246 -0.0127 0.1806 -0.0132 Stanford_SF_ENG_5 0.2317 -0.0045 0.0860 0 0.1844 -0.0036 summa_KB_ENG_1 0.0293 -0.0073 0.0000 0 0.0198 -0.0049 summa_KB_ENG_2 0.0645 -0.0103 0.0149 -0.0074 0.0484 -0.0094 summa_KB_ENG_5 0.0338 -0.0074 0.0152 -0.0076 0.0276 -0.0074 UMass_IESL_KB_ENG_1 0.1954 -0.0101 0.0468 -0.0015 0.1214 -0.0064 UMass_IESL_KB_ENG_2 0.1801 -0.0111 0.0544 -0.0032 0.1262 -0.0085 UMass_IESL_KB_ENG_3 0.2075 -0.0087 0.1361 -0.002 0.1841 -0.0065 UMass_IESL_KB_ENG_4 0.1685 -0.0112 0.0802 0.0019 0.1362 -0.0065 UMass_IESL_KB_ENG_5 0.1842 -0.0087 0.0296 -0.0009 0.0829 -0.0046 UMass_IESL_SF_ENG_1 0.2222 -0.0157 0.0226 0 0.1796 -0.014 UMass_IESL_SF_ENG_2 0.2215 -0.0217 0.0336 0 0.1759 -0.0174 UMass_IESL_SF_ENG_3 0.2166 -0.0051 0.1040 0.0139 0.1731 0.0042 UMass_IESL_SF_ENG_4 0.2043 -0.0197 0.0226 0 0.1625 -0.0167 UMass_IESL_SF_ENG_5 0.2015 -0.0187 0.0226 0 0.1563 -0.0151 UNIST_SAIL_SF_ENG_1 0.0414 -0.0018 0.0000 0 0.0136 -0.0009 UNIST_SAIL_SF_ENG_2 0.0459 -0.0041 0.0000 0 0.0340 -0.0033 UNIST_SAIL_SF_ENG_3 0.0534 -0.0017 0.0000 0 0.0391 -0.0014 UNIST_SAIL_SF_ENG_4 0.0840 0.0008 0.0000 0 0.0653 0.0001 Average change in F -0.00497857 -0.00105429 -0.00394 MEAN macro-average F1 and change in F1 after applying SFV filter, at each hop level: Cold Start run hop0_F Post-filter hop1_F Post-filter ALL_F Post-filter BBN_KB_ENG_1 0.2276 -0.0038 0.1128 -0.0079 0.1822 -0.0053 BBN_KB_ENG_2 0.2231 -0.0089 0.0966 -0.0078 0.1731 -0.0085 BBN_KB_ENG_3 0.2322 -0.0039 0.1124 -0.0078 0.1849 -0.0055 BBN_KB_ENG_4 0.2292 -0.0039 0.1127 -0.0078 0.1832 -0.0054 BBN_KB_ENG_5 0.2263 -0.0038 0.1076 -0.0078 0.1794 -0.0054 CMUML_SF_ENG_1 0.0333 -0.0059 0.0000 0 0.0202 -0.0036 CMUML_SF_ENG_2 0.0256 -0.0059 0.0000 0 0.0155 -0.0036 CMUML_SF_ENG_3 0.0030 -0.0026 0.0000 0 0.0018 -0.0015 CMUML_SF_ENG_4 0.0299 -0.0034 0.0000 0 0.0181 -0.0021 CMUML_SF_ENG_5 0.0333 -0.0059 0.0000 0 0.0202 -0.0036 DCD_SF_SF_ENG_1 0.1007 -0.005 0.0296 -0.0078 0.0726 -0.0061 DCD_SF_SF_ENG_2 0.1160 -0.0029 0.0270 0 0.0809 -0.0018 doughnutPRIS_SF_ENG_4 0.0859 -0.0058 0.0000 0 0.0520 -0.0035 doughnutPRIS_SF_ENG_5 0.1284 -0.0046 0.0000 0 0.0777 -0.0028 hltcoe_KB_ENG_1 0.2046 -0.0058 0.1108 -0.0053 0.1675 -0.0056 hltcoe_KB_ENG_2 0.1992 -0.0066 0.0886 -0.0052 0.1555 -0.006 hltcoe_KB_ENG_3 0.1972 -0.0066 0.0886 -0.0052 0.1543 -0.006 hltcoe_KB_ENG_4 0.2022 -0.001 0.1045 -0.0052 0.1636 -0.0027 hltcoe_KB_ENG_5 0.2072 -0.0059 0.1108 -0.0053 0.1691 -0.0056 ICTCAS_OKN_KB_ENG_1 0.1240 -0.0015 0.0252 0 0.0850 -0.0009 ICTCAS_OKN_KB_ENG_2 0.1256 -0.0016 0.0198 0 0.0838 -0.001 ICTCAS_OKN_KB_ENG_3 0.1002 -0.002 0.0253 0 0.0706 -0.0012 ICTCAS_OKN_KB_ENG_4 0.1134 -0.0032 0.0140 0.0026 0.0741 -0.0009 IRTSX_SF_ENG_1 0.0033 0 0.0000 0 0.0020 0 IRTSX_SF_ENG_2 0.0037 0 0.0000 0 0.0022 0 IRTSX_SF_ENG_3 0.0046 0 0.0000 0 0.0028 0 lilian_KB_ENG_1 0.0270 -0.0007 0.0000 0 0.0163 -0.0004 lilian_KB_ENG_2 0.0270 -0.0007 0.0000 0 0.0163 -0.0004 lilian_KB_ENG_3 0.0275 -0.0007 0.0000 0 0.0166 -0.0004 MSR_SF_ENG_1 0.0313 0 0.0000 0 0.0189 0 MSR_SF_ENG_2 0.0427 -0.0069 0.0000 0 0.0258 -0.0041 MSR_SF_ENG_3 0.0223 0 0.0000 0 0.0135 0 MSR_SF_ENG_4 0.0735 -0.0021 0.0215 0 0.0529 -0.0012 MSR_SF_ENG_5 0.0210 -0.0009 0.0000 0 0.0127 -0.0005 NAIST_CL_SF_ENG_1 0.0935 -0.0051 0.0199 -0.0052 0.0644 -0.0051 NAIST_CL_SF_ENG_2 0.0935 -0.0051 0.0000 0 0.0566 -0.0031 NAIST_CL_SF_ENG_3 0.0935 -0.0051 0.0000 0 0.0566 -0.0031 RPI_BLENDER_SF_ENG_1 0.1858 0 0.0418 0 0.1289 0 RPI_BLENDER_SF_ENG_2 0.1551 0 0.0411 0 0.1101 0 RPI_BLENDER_SF_ENG_3 0.1541 0 0.0405 0 0.1092 0 SoochowNLP_SF_ENG_1 0.0251 0.0008 0.0000 0 0.0152 0.0004 SoochowNLP_SF_ENG_2 0.0206 0.0006 0.0000 0 0.0125 0.0003 SoochowNLP_SF_ENG_3 0.0206 0.0006 0.0000 0 0.0125 0.0003 SoochowNLP_SF_ENG_4 0.0206 0.0006 0.0000 0 0.0125 0.0003 SoochowNLP_SF_ENG_5 0.0206 0.0006 0.0000 0 0.0125 0.0003 Stanford_KB_ENG_1 0.2149 -0.0094 0.1087 -0.0156 0.1730 -0.0119 Stanford_KB_ENG_2 0.1519 -0.0187 0.0700 -0.0156 0.1195 -0.0174 Stanford_KB_ENG_3 0.2553 -0.0086 0.1526 -0.0156 0.2147 -0.0113 Stanford_SF_ENG_1 0.1876 -0.0039 0.0788 -0.0078 0.1446 -0.0054 Stanford_SF_ENG_2 0.2334 -0.0128 0.1125 -0.0156 0.1856 -0.0139 Stanford_SF_ENG_3 0.2640 -0.0082 0.1638 -0.0156 0.2244 -0.0111 Stanford_SF_ENG_4 0.1733 -0.014 0.0615 -0.0156 0.1291 -0.0146 Stanford_SF_ENG_5 0.1876 -0.0039 0.0379 0 0.1285 -0.0024 summa_KB_ENG_1 0.0068 -0.0011 0.0000 0 0.0041 -0.0006 summa_KB_ENG_2 0.0202 -0.002 0.0156 -0.0078 0.0184 -0.0043 summa_KB_ENG_5 0.0116 -0.0013 0.0156 -0.0078 0.0132 -0.0039 UMass_IESL_KB_ENG_1 0.1375 -0.0032 0.0672 -0.0078 0.1097 -0.005 UMass_IESL_KB_ENG_2 0.1240 -0.0047 0.0581 -0.0078 0.0980 -0.006 UMass_IESL_KB_ENG_3 0.1204 -0.0025 0.1093 -0.0078 0.1160 -0.0046 UMass_IESL_KB_ENG_4 0.1266 -0.0034 0.0587 0 0.0998 -0.0021 UMass_IESL_KB_ENG_5 0.1375 -0.0041 0.0834 -0.0078 0.1161 -0.0056 UMass_IESL_SF_ENG_1 0.1842 -0.0172 0.0102 0 0.1154 -0.0104 UMass_IESL_SF_ENG_2 0.1648 -0.0166 0.0258 0 0.1099 -0.0101 UMass_IESL_SF_ENG_3 0.1598 -0.0123 0.0864 0.0026 0.1308 -0.0064 UMass_IESL_SF_ENG_4 0.1624 -0.0216 0.0102 0 0.1023 -0.0131 UMass_IESL_SF_ENG_5 0.1637 -0.0148 0.0102 0 0.1031 -0.009 UNIST_SAIL_SF_ENG_1 0.0263 -0.0034 0.0000 0 0.0159 -0.002 UNIST_SAIL_SF_ENG_2 0.0270 -0.0032 0.0000 0 0.0164 -0.002 UNIST_SAIL_SF_ENG_3 0.0307 -0.0034 0.0000 0 0.0186 -0.0021 UNIST_SAIL_SF_ENG_4 0.0491 -0.0033 0.0000 0 0.0297 -0.002 Average change in F -0.00460286 -0.00316143 -0.00403571