======================================================================== TAC KBP 2016 SLOT FILLER VALIDATION ENGLISH FILTERING EVALUATION RESULTS ======================================================================== Team ID: IRTSX Organization: IRT-SystemX & LIMSI ************************************************************* Run ID: IRTSX1 Did the run access the live Web during the evaluation window: No Did this run judge each candidate slot filler independently of all other candidate slot fillers in the evaluation dataset: Yes Did this run judge candidate slot fillers for each slot-filling run independently of all other slot-filling runs in the evaluation dataset: Yes Did this run judge candidate slot fillers for each slot-filling team independently of all other slot-filling teams in the evaluation dataset: Yes Did this run make use of the slot filler or justification offsets provided for each candidate slot filler: Yes Did this run make use of the confidence values provided for each candidate slot filler: Yes Did this run make use of the system profiles for the slot filling runs: Yes Did this run make use of the preliminary assessments provided for some of the slot filler candidates: No CSSF micro-average F1 and change in F1 after applying SFV filter, at each hop level: Cold Start run hop0_F Post-filter hop1_F Post-filter ALL_F Post-filter BBN_KB_ENG_1 0.3174 -0.1189 0.1737 -0.0945 0.2693 -0.109 BBN_KB_ENG_2 0.3040 -0.122 0.1650 -0.0909 0.2571 -0.1093 BBN_KB_ENG_3 0.3150 -0.1282 0.1724 -0.0791 0.2651 -0.1087 BBN_KB_ENG_4 0.3188 -0.1208 0.1734 -0.0942 0.2703 -0.1102 BBN_KB_ENG_5 0.3041 -0.1263 0.1529 -0.046 0.2475 -0.0929 CMUML_SF_ENG_1 0.0440 -0.0266 0.0000 0 0.0291 -0.0174 CMUML_SF_ENG_2 0.0371 -0.0139 0.0000 0 0.0249 -0.0094 CMUML_SF_ENG_3 0.0057 -0.0057 0.0000 0 0.0038 -0.0038 CMUML_SF_ENG_4 0.0369 -0.0224 0.0000 0 0.0246 -0.0149 CMUML_SF_ENG_5 0.0440 -0.0266 0.0000 0 0.0291 -0.0174 DCD_SF_SF_ENG_1 0.0804 -0.0413 0.0198 -0.0092 0.0588 -0.0291 DCD_SF_SF_ENG_2 0.0904 -0.0509 0.0186 -0.0186 0.0647 -0.0382 doughnutPRIS_SF_ENG_4 0.0865 -0.0354 0.0000 0 0.0648 -0.0298 doughnutPRIS_SF_ENG_5 0.1143 -0.0595 0.0000 0 0.0894 -0.0515 hltcoe_KB_ENG_1 0.2949 -0.1222 0.1202 -0.081 0.2300 -0.0994 hltcoe_KB_ENG_2 0.2953 -0.1402 0.1180 -0.1008 0.2311 -0.1194 hltcoe_KB_ENG_3 0.2976 -0.1444 0.1173 -0.1001 0.2320 -0.1217 hltcoe_KB_ENG_4 0.2776 -0.127 0.0830 -0.0772 0.1959 -0.0907 hltcoe_KB_ENG_5 0.2966 -0.1239 0.1202 -0.081 0.2311 -0.1005 ICTCAS_OKN_KB_ENG_1 0.1487 -0.0852 0.0441 -0.0383 0.1101 -0.0652 ICTCAS_OKN_KB_ENG_2 0.1490 -0.0803 0.0374 -0.0374 0.1081 -0.0615 ICTCAS_OKN_KB_ENG_3 0.1186 -0.0651 0.0526 -0.0247 0.0960 -0.0511 ICTCAS_OKN_KB_ENG_4 0.1421 -0.0707 0.0315 -0.0199 0.1041 -0.0519 IRTSX_SF_ENG_1 0.0062 -0.0033 0.0000 0 0.0038 -0.0018 IRTSX_SF_ENG_2 0.0064 -0.0005 0.0000 0 0.0038 0.0001 IRTSX_SF_ENG_3 0.0072 -0.0014 0.0000 0 0.0051 -0.0012 lilian_KB_ENG_1 0.0452 -0.0165 0.0000 0 0.0311 -0.0118 lilian_KB_ENG_2 0.0465 -0.0206 0.0000 0 0.0317 -0.0143 lilian_KB_ENG_3 0.0468 -0.018 0.0000 0 0.0318 -0.0125 MSR_SF_ENG_1 0.0275 -0.0164 0.0000 0 0.0150 -0.0077 MSR_SF_ENG_2 0.0297 -0.0211 0.0000 0 0.0198 -0.014 MSR_SF_ENG_3 0.0201 -0.0172 0.0000 0 0.0122 -0.0102 MSR_SF_ENG_4 0.0635 -0.0267 0.0102 -0.0044 0.0377 -0.011 MSR_SF_ENG_5 0.0176 -0.0066 0.0000 0 0.0083 -0.001 NAIST_CL_SF_ENG_1 0.0589 -0.0419 0.0100 -0.01 0.0146 -0.0037 NAIST_CL_SF_ENG_2 0.0589 -0.0419 0.0000 0 0.0438 -0.0323 NAIST_CL_SF_ENG_3 0.0589 -0.0419 0.0000 0 0.0433 -0.0318 RPI_BLENDER_SF_ENG_1 0.1904 -0.0748 0.1050 -0.0383 0.1644 -0.0645 RPI_BLENDER_SF_ENG_2 0.1721 -0.0713 0.0960 -0.0193 0.1487 -0.0558 RPI_BLENDER_SF_ENG_3 0.1700 -0.0693 0.0950 -0.0183 0.1470 -0.0541 SoochowNLP_SF_ENG_1 0.0451 -0.0102 0.0000 0 0.0363 -0.0123 SoochowNLP_SF_ENG_2 0.0372 -0.0095 0.0000 0 0.0275 -0.0087 SoochowNLP_SF_ENG_3 0.0372 -0.0095 0.0000 0 0.0192 -0.0005 SoochowNLP_SF_ENG_4 0.0372 -0.0095 0.0000 0 0.0262 -0.0075 SoochowNLP_SF_ENG_5 0.0372 -0.0095 0.0000 0 0.0254 -0.0069 Stanford_KB_ENG_1 0.2406 -0.1244 0.1466 -0.1239 0.2100 -0.1234 Stanford_KB_ENG_2 0.1926 -0.1173 0.1039 -0.0922 0.1648 -0.1101 Stanford_KB_ENG_3 0.2633 -0.0985 0.1190 -0.0624 0.2018 -0.072 Stanford_SF_ENG_1 0.2310 -0.1113 0.1197 -0.0798 0.1945 -0.1002 Stanford_SF_ENG_2 0.2444 -0.1045 0.1241 -0.0853 0.1971 -0.089 Stanford_SF_ENG_3 0.2846 -0.1121 0.0819 -0.0061 0.1751 -0.0338 Stanford_SF_ENG_4 0.2044 -0.1039 0.1061 -0.0887 0.1735 -0.0995 Stanford_SF_ENG_5 0.2310 -0.1115 0.0798 -0.074 0.1827 -0.0988 summa_KB_ENG_1 0.0255 -0.0167 0.0000 0 0.0171 -0.0113 summa_KB_ENG_2 0.0584 -0.0323 0.0114 0.0002 0.0430 -0.0217 summa_KB_ENG_5 0.0261 -0.0144 0.0116 -0.0058 0.0213 -0.0115 UMass_IESL_KB_ENG_1 0.1844 -0.0762 0.0628 -0.0221 0.1252 -0.0401 UMass_IESL_KB_ENG_2 0.1704 -0.0556 0.0687 -0.0128 0.1267 -0.0309 UMass_IESL_KB_ENG_3 0.2076 -0.1106 0.1480 -0.1078 0.1878 -0.1093 UMass_IESL_KB_ENG_4 0.1624 -0.0659 0.1061 -0.0589 0.1423 -0.0624 UMass_IESL_KB_ENG_5 0.1732 -0.053 0.0399 0.0031 0.0885 0.0048 UMass_IESL_SF_ENG_1 0.2207 -0.097 0.0282 0.0005 0.1794 -0.0847 UMass_IESL_SF_ENG_2 0.2224 -0.0969 0.0384 0.0007 0.1751 -0.0768 UMass_IESL_SF_ENG_3 0.2092 -0.0945 0.0954 -0.0669 0.1668 -0.0796 UMass_IESL_SF_ENG_4 0.2061 -0.0914 0.0282 0.0005 0.1643 -0.0764 UMass_IESL_SF_ENG_5 0.2070 -0.0855 0.0285 0.0002 0.1627 -0.0696 UNIST_SAIL_SF_ENG_1 0.0438 -0.0235 0.0000 0 0.0109 0.0027 UNIST_SAIL_SF_ENG_2 0.0476 -0.0189 0.0000 0 0.0352 -0.0159 UNIST_SAIL_SF_ENG_3 0.0583 -0.0297 0.0000 0 0.0426 -0.0234 UNIST_SAIL_SF_ENG_4 0.0788 -0.0424 0.0000 0 0.0610 -0.0364 Average change in F -0.0611871 -0.0280671 -0.0476543 MAX micro-average F1 and change in F1 after applying SFV filter, at each hop level: Cold Start run hop0_F Post-filter hop1_F Post-filter ALL_F Post-filter BBN_KB_ENG_1 0.3358 -0.1236 0.1659 -0.0906 0.2781 -0.1096 BBN_KB_ENG_2 0.3234 -0.1231 0.1542 -0.0726 0.2655 -0.103 BBN_KB_ENG_3 0.3313 -0.1336 0.1648 -0.0776 0.2716 -0.1099 BBN_KB_ENG_4 0.3375 -0.1266 0.1655 -0.0902 0.2791 -0.1113 BBN_KB_ENG_5 0.3253 -0.1327 0.1442 -0.0386 0.2550 -0.0909 CMUML_SF_ENG_1 0.0462 -0.0274 0.0000 0 0.0309 -0.0182 CMUML_SF_ENG_2 0.0409 -0.0183 0.0000 0 0.0276 -0.0124 CMUML_SF_ENG_3 0.0074 -0.0074 0.0000 0 0.0049 -0.0049 CMUML_SF_ENG_4 0.0370 -0.0219 0.0000 0 0.0248 -0.0147 CMUML_SF_ENG_5 0.0463 -0.0275 0.0000 0 0.0310 -0.0183 DCD_SF_SF_ENG_1 0.0889 -0.0483 0.0219 -0.0149 0.0652 -0.0356 DCD_SF_SF_ENG_2 0.1015 -0.0572 0.0208 -0.0208 0.0728 -0.0428 doughnutPRIS_SF_ENG_4 0.0847 -0.0352 0.0000 0 0.0631 -0.0292 doughnutPRIS_SF_ENG_5 0.1138 -0.0591 0.0000 0 0.0887 -0.0508 hltcoe_KB_ENG_1 0.2912 -0.1048 0.1082 -0.0782 0.2222 -0.084 hltcoe_KB_ENG_2 0.2887 -0.1189 0.1014 -0.0862 0.2204 -0.0983 hltcoe_KB_ENG_3 0.2891 -0.1221 0.1000 -0.0848 0.2191 -0.0991 hltcoe_KB_ENG_4 0.2735 -0.1105 0.0734 -0.0658 0.1872 -0.0722 hltcoe_KB_ENG_5 0.2925 -0.1061 0.1082 -0.0782 0.2232 -0.085 ICTCAS_OKN_KB_ENG_1 0.1507 -0.0828 0.0445 -0.0369 0.1120 -0.0633 ICTCAS_OKN_KB_ENG_2 0.1521 -0.0808 0.0401 -0.0401 0.1115 -0.0628 ICTCAS_OKN_KB_ENG_3 0.1441 -0.0751 0.0656 -0.0294 0.1166 -0.0586 ICTCAS_OKN_KB_ENG_4 0.1471 -0.0759 0.0216 -0.014 0.1055 -0.0545 IRTSX_SF_ENG_1 0.0081 -0.0043 0.0000 0 0.0050 -0.0024 IRTSX_SF_ENG_2 0.0088 -0.0012 0.0000 0 0.0059 -0.0008 IRTSX_SF_ENG_3 0.0097 -0.0021 0.0000 0 0.0068 -0.0017 lilian_KB_ENG_1 0.0446 -0.0148 0.0000 0 0.0308 -0.0107 lilian_KB_ENG_2 0.0462 -0.0201 0.0000 0 0.0316 -0.014 lilian_KB_ENG_3 0.0467 -0.0168 0.0000 0 0.0318 -0.0116 MSR_SF_ENG_1 0.0284 -0.014 0.0000 0 0.0153 -0.0057 MSR_SF_ENG_2 0.0343 -0.0269 0.0000 0 0.0228 -0.0178 MSR_SF_ENG_3 0.0247 -0.0209 0.0000 0 0.0150 -0.0124 MSR_SF_ENG_4 0.0661 -0.0365 0.0117 -0.004 0.0393 -0.0168 MSR_SF_ENG_5 0.0207 -0.0064 0.0000 0 0.0096 -0.0001 NAIST_CL_SF_ENG_1 0.0535 -0.0423 0.0092 -0.0092 0.0138 -0.0067 NAIST_CL_SF_ENG_2 0.0534 -0.0423 0.0000 0 0.0397 -0.0322 NAIST_CL_SF_ENG_3 0.0533 -0.0422 0.0000 0 0.0391 -0.0316 RPI_BLENDER_SF_ENG_1 0.1888 -0.0762 0.0855 -0.0269 0.1575 -0.062 RPI_BLENDER_SF_ENG_2 0.1647 -0.0714 0.0795 -0.0143 0.1385 -0.0543 RPI_BLENDER_SF_ENG_3 0.1628 -0.07 0.0789 -0.0139 0.1371 -0.0533 SoochowNLP_SF_ENG_1 0.0489 -0.0101 0.0000 0 0.0390 -0.0124 SoochowNLP_SF_ENG_2 0.0450 -0.0125 0.0000 0 0.0335 -0.0113 SoochowNLP_SF_ENG_3 0.0451 -0.0124 0.0000 0 0.0272 -0.0051 SoochowNLP_SF_ENG_4 0.0452 -0.0125 0.0000 0 0.0322 -0.0101 SoochowNLP_SF_ENG_5 0.0448 -0.0124 0.0000 0 0.0311 -0.0092 Stanford_KB_ENG_1 0.2401 -0.1157 0.1554 -0.1329 0.2121 -0.1199 Stanford_KB_ENG_2 0.1957 -0.1195 0.1220 -0.1067 0.1725 -0.1158 Stanford_KB_ENG_3 0.2699 -0.1081 0.1154 -0.0556 0.2009 -0.0725 Stanford_SF_ENG_1 0.2324 -0.1113 0.1186 -0.081 0.1938 -0.099 Stanford_SF_ENG_2 0.2481 -0.1055 0.1191 -0.0823 0.1949 -0.0852 Stanford_SF_ENG_3 0.3003 -0.1256 0.0732 0.0001 0.1691 -0.0267 Stanford_SF_ENG_4 0.2063 -0.1036 0.1246 -0.1018 0.1806 -0.1033 Stanford_SF_ENG_5 0.2317 -0.1104 0.0860 -0.0783 0.1844 -0.0985 summa_KB_ENG_1 0.0293 -0.018 0.0000 0 0.0198 -0.0122 summa_KB_ENG_2 0.0645 -0.0308 0.0149 0.0004 0.0484 -0.0208 summa_KB_ENG_5 0.0338 -0.0186 0.0152 -0.0075 0.0276 -0.0149 UMass_IESL_KB_ENG_1 0.1954 -0.0786 0.0468 -0.0143 0.1214 -0.0338 UMass_IESL_KB_ENG_2 0.1801 -0.059 0.0544 -0.0103 0.1262 -0.0297 UMass_IESL_KB_ENG_3 0.2075 -0.107 0.1361 -0.0982 0.1841 -0.1037 UMass_IESL_KB_ENG_4 0.1685 -0.0704 0.0802 -0.0463 0.1362 -0.06 UMass_IESL_KB_ENG_5 0.1842 -0.0577 0.0296 0.0063 0.0829 0.0112 UMass_IESL_SF_ENG_1 0.2222 -0.0999 0.0226 0.0002 0.1796 -0.0874 UMass_IESL_SF_ENG_2 0.2215 -0.1011 0.0336 -0.0032 0.1759 -0.083 UMass_IESL_SF_ENG_3 0.2166 -0.1061 0.1040 -0.0814 0.1731 -0.0902 UMass_IESL_SF_ENG_4 0.2043 -0.0951 0.0226 0.0002 0.1625 -0.0801 UMass_IESL_SF_ENG_5 0.2015 -0.0807 0.0226 0.0002 0.1563 -0.0655 UNIST_SAIL_SF_ENG_1 0.0414 -0.0226 0.0000 0 0.0136 -0.001 UNIST_SAIL_SF_ENG_2 0.0459 -0.0197 0.0000 0 0.0340 -0.0164 UNIST_SAIL_SF_ENG_3 0.0534 -0.0237 0.0000 0 0.0391 -0.019 UNIST_SAIL_SF_ENG_4 0.0840 -0.0439 0.0000 0 0.0653 -0.0381 Average change in F -0.0617114 -0.0268514 -0.0468157 MEAN macro-average F1 and change in F1 after applying SFV filter, at each hop level: Cold Start run hop0_F Post-filter hop1_F Post-filter ALL_F Post-filter BBN_KB_ENG_1 0.2276 -0.0945 0.1128 -0.0606 0.1822 -0.0811 BBN_KB_ENG_2 0.2231 -0.0902 0.0966 -0.0526 0.1731 -0.0753 BBN_KB_ENG_3 0.2322 -0.1018 0.1124 -0.058 0.1849 -0.0845 BBN_KB_ENG_4 0.2292 -0.0961 0.1127 -0.0605 0.1832 -0.0821 BBN_KB_ENG_5 0.2263 -0.1032 0.1076 -0.0522 0.1794 -0.083 CMUML_SF_ENG_1 0.0333 -0.0182 0.0000 0 0.0202 -0.011 CMUML_SF_ENG_2 0.0256 -0.0117 0.0000 0 0.0155 -0.0071 CMUML_SF_ENG_3 0.0030 -0.003 0.0000 0 0.0018 -0.0018 CMUML_SF_ENG_4 0.0299 -0.0168 0.0000 0 0.0181 -0.0102 CMUML_SF_ENG_5 0.0333 -0.0189 0.0000 0 0.0202 -0.0115 DCD_SF_SF_ENG_1 0.1007 -0.0628 0.0296 -0.0244 0.0726 -0.0476 DCD_SF_SF_ENG_2 0.1160 -0.0761 0.0270 -0.027 0.0809 -0.0568 doughnutPRIS_SF_ENG_4 0.0859 -0.0424 0.0000 0 0.0520 -0.0257 doughnutPRIS_SF_ENG_5 0.1284 -0.0807 0.0000 0 0.0777 -0.0489 hltcoe_KB_ENG_1 0.2046 -0.0949 0.1108 -0.0851 0.1675 -0.091 hltcoe_KB_ENG_2 0.1992 -0.1048 0.0886 -0.0786 0.1555 -0.0944 hltcoe_KB_ENG_3 0.1972 -0.0987 0.0886 -0.0786 0.1543 -0.0908 hltcoe_KB_ENG_4 0.2022 -0.109 0.1045 -0.0967 0.1636 -0.1041 hltcoe_KB_ENG_5 0.2072 -0.0975 0.1108 -0.0851 0.1691 -0.0926 ICTCAS_OKN_KB_ENG_1 0.1240 -0.0766 0.0252 -0.0213 0.0850 -0.0548 ICTCAS_OKN_KB_ENG_2 0.1256 -0.075 0.0198 -0.0198 0.0838 -0.0532 ICTCAS_OKN_KB_ENG_3 0.1002 -0.056 0.0253 -0.0214 0.0706 -0.0423 ICTCAS_OKN_KB_ENG_4 0.1134 -0.06 0.0140 -0.0109 0.0741 -0.0406 IRTSX_SF_ENG_1 0.0033 -0.0013 0.0000 0 0.0020 -0.0008 IRTSX_SF_ENG_2 0.0037 0.0009 0.0000 0 0.0022 0.0006 IRTSX_SF_ENG_3 0.0046 -0.0015 0.0000 0 0.0028 -0.0009 lilian_KB_ENG_1 0.0270 -0.0074 0.0000 0 0.0163 -0.0044 lilian_KB_ENG_2 0.0270 -0.0108 0.0000 0 0.0163 -0.0065 lilian_KB_ENG_3 0.0275 -0.0074 0.0000 0 0.0166 -0.0044 MSR_SF_ENG_1 0.0313 -0.0201 0.0000 0 0.0189 -0.0121 MSR_SF_ENG_2 0.0427 -0.0376 0.0000 0 0.0258 -0.0227 MSR_SF_ENG_3 0.0223 -0.0208 0.0000 0 0.0135 -0.0126 MSR_SF_ENG_4 0.0735 -0.0484 0.0215 -0.0176 0.0529 -0.0362 MSR_SF_ENG_5 0.0210 -0.0078 0.0000 0 0.0127 -0.0047 NAIST_CL_SF_ENG_1 0.0935 -0.0799 0.0199 -0.0199 0.0644 -0.0562 NAIST_CL_SF_ENG_2 0.0935 -0.0799 0.0000 0 0.0566 -0.0484 NAIST_CL_SF_ENG_3 0.0935 -0.0799 0.0000 0 0.0566 -0.0484 RPI_BLENDER_SF_ENG_1 0.1858 -0.0669 0.0418 -0.0075 0.1289 -0.0434 RPI_BLENDER_SF_ENG_2 0.1551 -0.0561 0.0411 -0.0071 0.1101 -0.0368 RPI_BLENDER_SF_ENG_3 0.1541 -0.0551 0.0405 -0.0065 0.1092 -0.0359 SoochowNLP_SF_ENG_1 0.0251 -0.0034 0.0000 0 0.0152 -0.0021 SoochowNLP_SF_ENG_2 0.0206 -0.0014 0.0000 0 0.0125 -0.0009 SoochowNLP_SF_ENG_3 0.0206 -0.0014 0.0000 0 0.0125 -0.0009 SoochowNLP_SF_ENG_4 0.0206 -0.0014 0.0000 0 0.0125 -0.0009 SoochowNLP_SF_ENG_5 0.0206 -0.0014 0.0000 0 0.0125 -0.0009 Stanford_KB_ENG_1 0.2149 -0.1227 0.1087 -0.0879 0.1730 -0.109 Stanford_KB_ENG_2 0.1519 -0.095 0.0700 -0.0544 0.1195 -0.0789 Stanford_KB_ENG_3 0.2553 -0.1244 0.1526 -0.1099 0.2147 -0.1187 Stanford_SF_ENG_1 0.1876 -0.0964 0.0788 -0.0483 0.1446 -0.0774 Stanford_SF_ENG_2 0.2334 -0.1274 0.1125 -0.082 0.1856 -0.1094 Stanford_SF_ENG_3 0.2640 -0.1444 0.1638 -0.1047 0.2244 -0.1287 Stanford_SF_ENG_4 0.1733 -0.1015 0.0615 -0.042 0.1291 -0.078 Stanford_SF_ENG_5 0.1876 -0.0964 0.0379 -0.0301 0.1285 -0.0702 summa_KB_ENG_1 0.0068 -0.0041 0.0000 0 0.0041 -0.0024 summa_KB_ENG_2 0.0202 -0.0069 0.0156 0 0.0184 -0.0042 summa_KB_ENG_5 0.0116 -0.0056 0.0156 -0.0078 0.0132 -0.0065 UMass_IESL_KB_ENG_1 0.1375 -0.0526 0.0672 -0.0485 0.1097 -0.0509 UMass_IESL_KB_ENG_2 0.1240 -0.0334 0.0581 -0.0362 0.0980 -0.0346 UMass_IESL_KB_ENG_3 0.1204 -0.0526 0.1093 -0.0781 0.1160 -0.0626 UMass_IESL_KB_ENG_4 0.1266 -0.0532 0.0587 -0.036 0.0998 -0.0465 UMass_IESL_KB_ENG_5 0.1375 -0.0478 0.0834 -0.0568 0.1161 -0.0513 UMass_IESL_SF_ENG_1 0.1842 -0.0805 0.0102 0.0015 0.1154 -0.048 UMass_IESL_SF_ENG_2 0.1648 -0.0675 0.0258 -0.0063 0.1099 -0.0433 UMass_IESL_SF_ENG_3 0.1598 -0.0825 0.0864 -0.0734 0.1308 -0.0789 UMass_IESL_SF_ENG_4 0.1624 -0.0639 0.0102 0.0015 0.1023 -0.0381 UMass_IESL_SF_ENG_5 0.1637 -0.071 0.0102 0.0015 0.1031 -0.0424 UNIST_SAIL_SF_ENG_1 0.0263 -0.0098 0.0000 0 0.0159 -0.0059 UNIST_SAIL_SF_ENG_2 0.0270 -0.0063 0.0000 0 0.0164 -0.0038 UNIST_SAIL_SF_ENG_3 0.0307 -0.0105 0.0000 0 0.0186 -0.0064 UNIST_SAIL_SF_ENG_4 0.0491 -0.0277 0.0000 0 0.0297 -0.0167 Average change in F -0.0537429 -0.0255614 -0.04261 ************************************************************* Run ID: IRTSX2 Did the run access the live Web during the evaluation window: No Did this run judge each candidate slot filler independently of all other candidate slot fillers in the evaluation dataset: Yes Did this run judge candidate slot fillers for each slot-filling run independently of all other slot-filling runs in the evaluation dataset: Yes Did this run judge candidate slot fillers for each slot-filling team independently of all other slot-filling teams in the evaluation dataset: Yes Did this run make use of the slot filler or justification offsets provided for each candidate slot filler: Yes Did this run make use of the confidence values provided for each candidate slot filler: Yes Did this run make use of the system profiles for the slot filling runs: Yes Did this run make use of the preliminary assessments provided for some of the slot filler candidates: No CSSF micro-average F1 and change in F1 after applying SFV filter, at each hop level: Cold Start run hop0_F Post-filter hop1_F Post-filter ALL_F Post-filter BBN_KB_ENG_1 0.3174 -0.1097 0.1737 -0.0703 0.2693 -0.0954 BBN_KB_ENG_2 0.3040 -0.1211 0.1650 -0.0821 0.2571 -0.1065 BBN_KB_ENG_3 0.3150 -0.1215 0.1724 -0.0647 0.2651 -0.0995 BBN_KB_ENG_4 0.3188 -0.1119 0.1734 -0.07 0.2703 -0.0968 BBN_KB_ENG_5 0.3041 -0.1178 0.1529 -0.0412 0.2475 -0.0857 CMUML_SF_ENG_1 0.0440 -0.0237 0.0000 0 0.0291 -0.0156 CMUML_SF_ENG_2 0.0371 -0.0112 0.0000 0 0.0249 -0.0076 CMUML_SF_ENG_3 0.0057 -0.0057 0.0000 0 0.0038 -0.0038 CMUML_SF_ENG_4 0.0369 -0.0195 0.0000 0 0.0246 -0.013 CMUML_SF_ENG_5 0.0440 -0.0266 0.0000 0 0.0291 -0.0175 DCD_SF_SF_ENG_1 0.0804 -0.0369 0.0198 -0.0095 0.0588 -0.0263 DCD_SF_SF_ENG_2 0.0904 -0.046 0.0186 -0.0186 0.0647 -0.0348 doughnutPRIS_SF_ENG_4 0.0865 -0.0268 0.0000 0 0.0648 -0.024 doughnutPRIS_SF_ENG_5 0.1143 -0.0383 0.0000 0 0.0894 -0.0369 hltcoe_KB_ENG_1 0.2949 -0.0944 0.1202 -0.0671 0.2300 -0.0768 hltcoe_KB_ENG_2 0.2953 -0.1086 0.1180 -0.084 0.2311 -0.0919 hltcoe_KB_ENG_3 0.2976 -0.1132 0.1173 -0.0833 0.2320 -0.0944 hltcoe_KB_ENG_4 0.2776 -0.0879 0.0830 -0.0543 0.1959 -0.0558 hltcoe_KB_ENG_5 0.2966 -0.0961 0.1202 -0.0671 0.2311 -0.0779 ICTCAS_OKN_KB_ENG_1 0.1487 -0.0559 0.0441 -0.0053 0.1101 -0.0348 ICTCAS_OKN_KB_ENG_2 0.1490 -0.0536 0.0374 -0.004 0.1081 -0.0327 ICTCAS_OKN_KB_ENG_3 0.1186 -0.0443 0.0526 -0.0138 0.0960 -0.0335 ICTCAS_OKN_KB_ENG_4 0.1421 -0.0438 0.0315 0.0028 0.1041 -0.0261 IRTSX_SF_ENG_1 0.0062 -0.0062 0.0000 0 0.0038 -0.0038 IRTSX_SF_ENG_2 0.0064 -0.0035 0.0000 0 0.0038 -0.0018 IRTSX_SF_ENG_3 0.0072 -0.0043 0.0000 0 0.0051 -0.0031 lilian_KB_ENG_1 0.0452 -0.0164 0.0000 0 0.0311 -0.0118 lilian_KB_ENG_2 0.0465 -0.0205 0.0000 0 0.0317 -0.0143 lilian_KB_ENG_3 0.0468 -0.0179 0.0000 0 0.0318 -0.0124 MSR_SF_ENG_1 0.0275 -0.0165 0.0000 0 0.0150 -0.0077 MSR_SF_ENG_2 0.0297 -0.0183 0.0000 0 0.0198 -0.0121 MSR_SF_ENG_3 0.0201 -0.0084 0.0000 0 0.0122 -0.0044 MSR_SF_ENG_4 0.0635 -0.027 0.0102 -0.0044 0.0377 -0.0113 MSR_SF_ENG_5 0.0176 -0.0039 0.0000 0 0.0083 0.0008 NAIST_CL_SF_ENG_1 0.0589 -0.0201 0.0100 -0.0009 0.0146 0.013 NAIST_CL_SF_ENG_2 0.0589 -0.0201 0.0000 0 0.0438 -0.0175 NAIST_CL_SF_ENG_3 0.0589 -0.0201 0.0000 0 0.0433 -0.017 RPI_BLENDER_SF_ENG_1 0.1904 -0.0723 0.1050 -0.0383 0.1644 -0.0628 RPI_BLENDER_SF_ENG_2 0.1721 -0.0688 0.0960 -0.0193 0.1487 -0.0541 RPI_BLENDER_SF_ENG_3 0.1700 -0.0667 0.0950 -0.0183 0.1470 -0.0524 SoochowNLP_SF_ENG_1 0.0451 -0.0144 0.0000 0 0.0363 -0.0155 SoochowNLP_SF_ENG_2 0.0372 -0.0144 0.0000 0 0.0275 -0.0122 SoochowNLP_SF_ENG_3 0.0372 -0.0144 0.0000 0 0.0192 -0.0039 SoochowNLP_SF_ENG_4 0.0372 -0.0144 0.0000 0 0.0262 -0.0109 SoochowNLP_SF_ENG_5 0.0372 -0.0144 0.0000 0 0.0254 -0.0101 Stanford_KB_ENG_1 0.2406 -0.0929 0.1466 -0.1238 0.2100 -0.1014 Stanford_KB_ENG_2 0.1926 -0.0625 0.1039 -0.0752 0.1648 -0.0667 Stanford_KB_ENG_3 0.2633 -0.0526 0.1190 -0.05 0.2018 -0.0378 Stanford_SF_ENG_1 0.2310 -0.0708 0.1197 -0.0708 0.1945 -0.0702 Stanford_SF_ENG_2 0.2444 -0.0678 0.1241 -0.0798 0.1971 -0.0615 Stanford_SF_ENG_3 0.2846 -0.0829 0.0819 0.0033 0.1751 -0.0111 Stanford_SF_ENG_4 0.2044 -0.057 0.1061 -0.0831 0.1735 -0.0652 Stanford_SF_ENG_5 0.2310 -0.071 0.0798 -0.0624 0.1827 -0.0665 summa_KB_ENG_1 0.0255 -0.0168 0.0000 0 0.0171 -0.0113 summa_KB_ENG_2 0.0584 -0.0267 0.0114 0.0002 0.0430 -0.0179 summa_KB_ENG_5 0.0261 -0.0202 0.0116 -0.0058 0.0213 -0.0154 UMass_IESL_KB_ENG_1 0.1844 -0.0682 0.0628 -0.0311 0.1252 -0.0372 UMass_IESL_KB_ENG_2 0.1704 -0.0506 0.0687 -0.0187 0.1267 -0.0295 UMass_IESL_KB_ENG_3 0.2076 -0.0991 0.1480 -0.0972 0.1878 -0.098 UMass_IESL_KB_ENG_4 0.1624 -0.058 0.1061 -0.0739 0.1423 -0.0619 UMass_IESL_KB_ENG_5 0.1732 -0.0496 0.0399 -0.005 0.0885 0.0049 UMass_IESL_SF_ENG_1 0.2207 -0.0491 0.0282 -0.0224 0.1794 -0.0573 UMass_IESL_SF_ENG_2 0.2224 -0.0684 0.0384 -0.0211 0.1751 -0.0626 UMass_IESL_SF_ENG_3 0.2092 -0.061 0.0954 -0.0673 0.1668 -0.0566 UMass_IESL_SF_ENG_4 0.2061 -0.0578 0.0282 -0.0224 0.1643 -0.0594 UMass_IESL_SF_ENG_5 0.2070 -0.0512 0.0285 -0.0227 0.1627 -0.052 UNIST_SAIL_SF_ENG_1 0.0438 -0.0237 0.0000 0 0.0109 0.0026 UNIST_SAIL_SF_ENG_2 0.0476 -0.022 0.0000 0 0.0352 -0.018 UNIST_SAIL_SF_ENG_3 0.0583 -0.0188 0.0000 0 0.0426 -0.0159 UNIST_SAIL_SF_ENG_4 0.0788 -0.0344 0.0000 0 0.0610 -0.0308 Average change in F -0.04758 -0.0248986 -0.0372757 MAX micro-average F1 and change in F1 after applying SFV filter, at each hop level: Cold Start run hop0_F Post-filter hop1_F Post-filter ALL_F Post-filter BBN_KB_ENG_1 0.3358 -0.1132 0.1659 -0.0589 0.2781 -0.0931 BBN_KB_ENG_2 0.3234 -0.1208 0.1542 -0.0612 0.2655 -0.0987 BBN_KB_ENG_3 0.3313 -0.1248 0.1648 -0.0522 0.2716 -0.0959 BBN_KB_ENG_4 0.3375 -0.1171 0.1655 -0.0585 0.2791 -0.0953 BBN_KB_ENG_5 0.3253 -0.1214 0.1442 -0.0258 0.2550 -0.0793 CMUML_SF_ENG_1 0.0462 -0.0237 0.0000 0 0.0309 -0.0158 CMUML_SF_ENG_2 0.0409 -0.011 0.0000 0 0.0276 -0.0075 CMUML_SF_ENG_3 0.0074 -0.0074 0.0000 0 0.0049 -0.0049 CMUML_SF_ENG_4 0.0370 -0.0181 0.0000 0 0.0248 -0.0121 CMUML_SF_ENG_5 0.0463 -0.0275 0.0000 0 0.0310 -0.0184 DCD_SF_SF_ENG_1 0.0889 -0.0425 0.0219 -0.0152 0.0652 -0.0319 DCD_SF_SF_ENG_2 0.1015 -0.0506 0.0208 -0.0208 0.0728 -0.0384 doughnutPRIS_SF_ENG_4 0.0847 -0.0276 0.0000 0 0.0631 -0.024 doughnutPRIS_SF_ENG_5 0.1138 -0.0415 0.0000 0 0.0887 -0.0387 hltcoe_KB_ENG_1 0.2912 -0.0827 0.1082 -0.0577 0.2222 -0.0628 hltcoe_KB_ENG_2 0.2887 -0.096 0.1014 -0.0637 0.2204 -0.0751 hltcoe_KB_ENG_3 0.2891 -0.0964 0.1000 -0.063 0.2191 -0.0746 hltcoe_KB_ENG_4 0.2735 -0.0802 0.0734 -0.0431 0.1872 -0.0437 hltcoe_KB_ENG_5 0.2925 -0.084 0.1082 -0.0586 0.2232 -0.0647 ICTCAS_OKN_KB_ENG_1 0.1507 -0.054 0.0445 -0.0009 0.1120 -0.0324 ICTCAS_OKN_KB_ENG_2 0.1521 -0.053 0.0401 -0.0033 0.1115 -0.0322 ICTCAS_OKN_KB_ENG_3 0.1441 -0.0487 0.0656 -0.0154 0.1166 -0.0361 ICTCAS_OKN_KB_ENG_4 0.1471 -0.0502 0.0216 0.0011 0.1055 -0.0319 IRTSX_SF_ENG_1 0.0081 -0.0081 0.0000 0 0.0050 -0.005 IRTSX_SF_ENG_2 0.0088 -0.005 0.0000 0 0.0059 -0.0034 IRTSX_SF_ENG_3 0.0097 -0.0059 0.0000 0 0.0068 -0.0043 lilian_KB_ENG_1 0.0446 -0.0147 0.0000 0 0.0308 -0.0107 lilian_KB_ENG_2 0.0462 -0.0199 0.0000 0 0.0316 -0.0139 lilian_KB_ENG_3 0.0467 -0.0167 0.0000 0 0.0318 -0.0116 MSR_SF_ENG_1 0.0284 -0.0142 0.0000 0 0.0153 -0.0059 MSR_SF_ENG_2 0.0343 -0.0231 0.0000 0 0.0228 -0.0153 MSR_SF_ENG_3 0.0247 -0.0095 0.0000 0 0.0150 -0.0048 MSR_SF_ENG_4 0.0661 -0.0334 0.0117 -0.0041 0.0393 -0.0147 MSR_SF_ENG_5 0.0207 -0.0028 0.0000 0 0.0096 0.0022 NAIST_CL_SF_ENG_1 0.0535 -0.0211 0.0092 0.002 0.0138 0.0103 NAIST_CL_SF_ENG_2 0.0534 -0.021 0.0000 0 0.0397 -0.0176 NAIST_CL_SF_ENG_3 0.0533 -0.021 0.0000 0 0.0391 -0.0171 RPI_BLENDER_SF_ENG_1 0.1888 -0.073 0.0855 -0.0269 0.1575 -0.0598 RPI_BLENDER_SF_ENG_2 0.1647 -0.0685 0.0795 -0.0145 0.1385 -0.0524 RPI_BLENDER_SF_ENG_3 0.1628 -0.0666 0.0789 -0.0139 0.1371 -0.051 SoochowNLP_SF_ENG_1 0.0489 -0.0163 0.0000 0 0.0390 -0.0168 SoochowNLP_SF_ENG_2 0.0450 -0.0192 0.0000 0 0.0335 -0.016 SoochowNLP_SF_ENG_3 0.0451 -0.0192 0.0000 0 0.0272 -0.0097 SoochowNLP_SF_ENG_4 0.0452 -0.0193 0.0000 0 0.0322 -0.0147 SoochowNLP_SF_ENG_5 0.0448 -0.019 0.0000 0 0.0311 -0.0136 Stanford_KB_ENG_1 0.2401 -0.0782 0.1554 -0.1254 0.2121 -0.0912 Stanford_KB_ENG_2 0.1957 -0.0514 0.1220 -0.0843 0.1725 -0.0615 Stanford_KB_ENG_3 0.2699 -0.0519 0.1154 -0.0331 0.2009 -0.0281 Stanford_SF_ENG_1 0.2324 -0.0669 0.1186 -0.0625 0.1938 -0.0638 Stanford_SF_ENG_2 0.2481 -0.0589 0.1191 -0.068 0.1949 -0.0483 Stanford_SF_ENG_3 0.3003 -0.0822 0.0732 0.0255 0.1691 0.0106 Stanford_SF_ENG_4 0.2063 -0.0517 0.1246 -0.0945 0.1806 -0.065 Stanford_SF_ENG_5 0.2317 -0.0659 0.0860 -0.0631 0.1844 -0.0625 summa_KB_ENG_1 0.0293 -0.018 0.0000 0 0.0198 -0.0122 summa_KB_ENG_2 0.0645 -0.027 0.0149 0.0004 0.0484 -0.0182 summa_KB_ENG_5 0.0338 -0.0262 0.0152 -0.0075 0.0276 -0.0199 UMass_IESL_KB_ENG_1 0.1954 -0.0627 0.0468 -0.013 0.1214 -0.0219 UMass_IESL_KB_ENG_2 0.1801 -0.0458 0.0544 -0.0037 0.1262 -0.0188 UMass_IESL_KB_ENG_3 0.2075 -0.095 0.1361 -0.0771 0.1841 -0.0889 UMass_IESL_KB_ENG_4 0.1685 -0.0535 0.0802 -0.0525 0.1362 -0.0505 UMass_IESL_KB_ENG_5 0.1842 -0.0449 0.0296 0.0083 0.0829 0.0214 UMass_IESL_SF_ENG_1 0.2222 -0.0528 0.0226 -0.0149 0.1796 -0.0585 UMass_IESL_SF_ENG_2 0.2215 -0.073 0.0336 -0.0107 0.1759 -0.0653 UMass_IESL_SF_ENG_3 0.2166 -0.0651 0.1040 -0.0671 0.1731 -0.0575 UMass_IESL_SF_ENG_4 0.2043 -0.0605 0.0226 -0.0149 0.1625 -0.0601 UMass_IESL_SF_ENG_5 0.2015 -0.0487 0.0226 -0.0149 0.1563 -0.0473 UNIST_SAIL_SF_ENG_1 0.0414 -0.0227 0.0000 0 0.0136 -0.001 UNIST_SAIL_SF_ENG_2 0.0459 -0.0237 0.0000 0 0.0340 -0.019 UNIST_SAIL_SF_ENG_3 0.0534 -0.0201 0.0000 0 0.0391 -0.0166 UNIST_SAIL_SF_ENG_4 0.0840 -0.0439 0.0000 0 0.0653 -0.038 Average change in F -0.0471514 -0.0203943 -0.0347914 MEAN macro-average F1 and change in F1 after applying SFV filter, at each hop level: Cold Start run hop0_F Post-filter hop1_F Post-filter ALL_F Post-filter BBN_KB_ENG_1 0.2276 -0.079 0.1128 -0.0442 0.1822 -0.0652 BBN_KB_ENG_2 0.2231 -0.0762 0.0966 -0.0462 0.1731 -0.0643 BBN_KB_ENG_3 0.2322 -0.0858 0.1124 -0.0435 0.1849 -0.0692 BBN_KB_ENG_4 0.2292 -0.0806 0.1127 -0.0441 0.1832 -0.0662 BBN_KB_ENG_5 0.2263 -0.0891 0.1076 -0.0388 0.1794 -0.0692 CMUML_SF_ENG_1 0.0333 -0.0169 0.0000 0 0.0202 -0.0103 CMUML_SF_ENG_2 0.0256 -0.0085 0.0000 0 0.0155 -0.0052 CMUML_SF_ENG_3 0.0030 -0.003 0.0000 0 0.0018 -0.0018 CMUML_SF_ENG_4 0.0299 -0.0148 0.0000 0 0.0181 -0.0089 CMUML_SF_ENG_5 0.0333 -0.0182 0.0000 0 0.0202 -0.011 DCD_SF_SF_ENG_1 0.1007 -0.0564 0.0296 -0.0244 0.0726 -0.0437 DCD_SF_SF_ENG_2 0.1160 -0.0659 0.0270 -0.027 0.0809 -0.0506 doughnutPRIS_SF_ENG_4 0.0859 -0.03 0.0000 0 0.0520 -0.0182 doughnutPRIS_SF_ENG_5 0.1284 -0.0628 0.0000 0 0.0777 -0.038 hltcoe_KB_ENG_1 0.2046 -0.0632 0.1108 -0.0694 0.1675 -0.0656 hltcoe_KB_ENG_2 0.1992 -0.0719 0.0886 -0.0628 0.1555 -0.0683 hltcoe_KB_ENG_3 0.1972 -0.0689 0.0886 -0.0628 0.1543 -0.0665 hltcoe_KB_ENG_4 0.2022 -0.0691 0.1045 -0.0865 0.1636 -0.076 hltcoe_KB_ENG_5 0.2072 -0.0658 0.1108 -0.0694 0.1691 -0.0672 ICTCAS_OKN_KB_ENG_1 0.1240 -0.0472 0.0252 -0.0078 0.0850 -0.0316 ICTCAS_OKN_KB_ENG_2 0.1256 -0.0461 0.0198 -0.0063 0.0838 -0.0303 ICTCAS_OKN_KB_ENG_3 0.1002 -0.0385 0.0253 -0.0144 0.0706 -0.029 ICTCAS_OKN_KB_ENG_4 0.1134 -0.0373 0.0140 0.0034 0.0741 -0.0212 IRTSX_SF_ENG_1 0.0033 -0.0033 0.0000 0 0.0020 -0.002 IRTSX_SF_ENG_2 0.0037 -0.0011 0.0000 0 0.0022 -0.0007 IRTSX_SF_ENG_3 0.0046 -0.0036 0.0000 0 0.0028 -0.0022 lilian_KB_ENG_1 0.0270 -0.008 0.0000 0 0.0163 -0.0048 lilian_KB_ENG_2 0.0270 -0.0114 0.0000 0 0.0163 -0.0068 lilian_KB_ENG_3 0.0275 -0.0079 0.0000 0 0.0166 -0.0048 MSR_SF_ENG_1 0.0313 -0.0214 0.0000 0 0.0189 -0.0129 MSR_SF_ENG_2 0.0427 -0.0356 0.0000 0 0.0258 -0.0215 MSR_SF_ENG_3 0.0223 -0.0154 0.0000 0 0.0135 -0.0093 MSR_SF_ENG_4 0.0735 -0.0468 0.0215 -0.0176 0.0529 -0.0352 MSR_SF_ENG_5 0.0210 -0.0044 0.0000 0 0.0127 -0.0027 NAIST_CL_SF_ENG_1 0.0935 -0.0518 0.0199 -0.0189 0.0644 -0.0388 NAIST_CL_SF_ENG_2 0.0935 -0.0518 0.0000 0 0.0566 -0.0314 NAIST_CL_SF_ENG_3 0.0935 -0.0518 0.0000 0 0.0566 -0.0314 RPI_BLENDER_SF_ENG_1 0.1858 -0.0641 0.0418 -0.0075 0.1289 -0.0417 RPI_BLENDER_SF_ENG_2 0.1551 -0.0533 0.0411 -0.0071 0.1101 -0.0351 RPI_BLENDER_SF_ENG_3 0.1541 -0.0523 0.0405 -0.0065 0.1092 -0.0342 SoochowNLP_SF_ENG_1 0.0251 -0.0015 0.0000 0 0.0152 -0.0009 SoochowNLP_SF_ENG_2 0.0206 -0.0011 0.0000 0 0.0125 -0.0007 SoochowNLP_SF_ENG_3 0.0206 -0.0011 0.0000 0 0.0125 -0.0007 SoochowNLP_SF_ENG_4 0.0206 -0.0011 0.0000 0 0.0125 -0.0007 SoochowNLP_SF_ENG_5 0.0206 -0.0011 0.0000 0 0.0125 -0.0007 Stanford_KB_ENG_1 0.2149 -0.0805 0.1087 -0.0886 0.1730 -0.0838 Stanford_KB_ENG_2 0.1519 -0.0436 0.0700 -0.0401 0.1195 -0.0421 Stanford_KB_ENG_3 0.2553 -0.0635 0.1526 -0.1047 0.2147 -0.0797 Stanford_SF_ENG_1 0.1876 -0.0481 0.0788 -0.034 0.1446 -0.0425 Stanford_SF_ENG_2 0.2334 -0.0768 0.1125 -0.0716 0.1856 -0.0747 Stanford_SF_ENG_3 0.2640 -0.0908 0.1638 -0.0981 0.2244 -0.0937 Stanford_SF_ENG_4 0.1733 -0.0497 0.0615 -0.0355 0.1291 -0.044 Stanford_SF_ENG_5 0.1876 -0.0481 0.0379 -0.0145 0.1285 -0.0349 summa_KB_ENG_1 0.0068 -0.0034 0.0000 0 0.0041 -0.002 summa_KB_ENG_2 0.0202 -0.0091 0.0156 0 0.0184 -0.0055 summa_KB_ENG_5 0.0116 -0.0081 0.0156 -0.0078 0.0132 -0.008 UMass_IESL_KB_ENG_1 0.1375 -0.0392 0.0672 -0.0458 0.1097 -0.0418 UMass_IESL_KB_ENG_2 0.1240 -0.0234 0.0581 -0.0284 0.0980 -0.0254 UMass_IESL_KB_ENG_3 0.1204 -0.0509 0.1093 -0.0645 0.1160 -0.0562 UMass_IESL_KB_ENG_4 0.1266 -0.0391 0.0587 -0.0373 0.0998 -0.0384 UMass_IESL_KB_ENG_5 0.1375 -0.0349 0.0834 -0.0542 0.1161 -0.0425 UMass_IESL_SF_ENG_1 0.1842 -0.0387 0.0102 -0.0063 0.1154 -0.0259 UMass_IESL_SF_ENG_2 0.1648 -0.0422 0.0258 -0.0102 0.1099 -0.0295 UMass_IESL_SF_ENG_3 0.1598 -0.0578 0.0864 -0.0538 0.1308 -0.0563 UMass_IESL_SF_ENG_4 0.1624 -0.0372 0.0102 -0.0063 0.1023 -0.025 UMass_IESL_SF_ENG_5 0.1637 -0.038 0.0102 -0.0063 0.1031 -0.0255 UNIST_SAIL_SF_ENG_1 0.0263 -0.0055 0.0000 0 0.0159 -0.0033 UNIST_SAIL_SF_ENG_2 0.0270 -0.0028 0.0000 0 0.0164 -0.0017 UNIST_SAIL_SF_ENG_3 0.0307 -0.0018 0.0000 0 0.0186 -0.0011 UNIST_SAIL_SF_ENG_4 0.0491 -0.0194 0.0000 0 0.0297 -0.0117 Average change in F -0.0376814 -0.0215686 -0.0313129 ************************************************************* Run ID: IRTSX3 Did the run access the live Web during the evaluation window: No Did this run judge each candidate slot filler independently of all other candidate slot fillers in the evaluation dataset: Yes Did this run judge candidate slot fillers for each slot-filling run independently of all other slot-filling runs in the evaluation dataset: Yes Did this run judge candidate slot fillers for each slot-filling team independently of all other slot-filling teams in the evaluation dataset: Yes Did this run make use of the slot filler or justification offsets provided for each candidate slot filler: Yes Did this run make use of the confidence values provided for each candidate slot filler: Yes Did this run make use of the system profiles for the slot filling runs: Yes Did this run make use of the preliminary assessments provided for some of the slot filler candidates: No CSSF micro-average F1 and change in F1 after applying SFV filter, at each hop level: Cold Start run hop0_F Post-filter hop1_F Post-filter ALL_F Post-filter BBN_KB_ENG_1 0.3174 -0.1459 0.1737 -0.1133 0.2693 -0.1327 BBN_KB_ENG_2 0.3040 -0.1527 0.1650 -0.1052 0.2571 -0.1348 BBN_KB_ENG_3 0.3150 -0.1555 0.1724 -0.107 0.2651 -0.1355 BBN_KB_ENG_4 0.3188 -0.1477 0.1734 -0.113 0.2703 -0.134 BBN_KB_ENG_5 0.3041 -0.1555 0.1529 -0.0687 0.2475 -0.1197 CMUML_SF_ENG_1 0.0440 -0.0267 0.0000 0 0.0291 -0.0175 CMUML_SF_ENG_2 0.0371 -0.0112 0.0000 0 0.0249 -0.0075 CMUML_SF_ENG_3 0.0057 -0.0057 0.0000 0 0.0038 -0.0038 CMUML_SF_ENG_4 0.0369 -0.0224 0.0000 0 0.0246 -0.0149 CMUML_SF_ENG_5 0.0440 -0.0267 0.0000 0 0.0291 -0.0175 DCD_SF_SF_ENG_1 0.0804 -0.0514 0.0198 -0.0198 0.0588 -0.0393 DCD_SF_SF_ENG_2 0.0904 -0.0666 0.0186 -0.0186 0.0647 -0.0488 doughnutPRIS_SF_ENG_4 0.0865 -0.0397 0.0000 0 0.0648 -0.0329 doughnutPRIS_SF_ENG_5 0.1143 -0.0601 0.0000 0 0.0894 -0.0523 hltcoe_KB_ENG_1 0.2949 -0.143 0.1202 -0.0808 0.2300 -0.1134 hltcoe_KB_ENG_2 0.2953 -0.1611 0.1180 -0.1122 0.2311 -0.137 hltcoe_KB_ENG_3 0.2976 -0.1648 0.1173 -0.1115 0.2320 -0.139 hltcoe_KB_ENG_4 0.2776 -0.1397 0.0830 -0.0658 0.1959 -0.0961 hltcoe_KB_ENG_5 0.2966 -0.1424 0.1202 -0.0808 0.2311 -0.1128 ICTCAS_OKN_KB_ENG_1 0.1487 -0.0945 0.0441 -0.0383 0.1101 -0.0713 ICTCAS_OKN_KB_ENG_2 0.1490 -0.0868 0.0374 -0.0374 0.1081 -0.0656 ICTCAS_OKN_KB_ENG_3 0.1186 -0.0713 0.0526 -0.0241 0.0960 -0.0549 ICTCAS_OKN_KB_ENG_4 0.1421 -0.0827 0.0315 -0.0199 0.1041 -0.0599 IRTSX_SF_ENG_1 0.0062 -0.0033 0.0000 0 0.0038 -0.0019 IRTSX_SF_ENG_2 0.0064 -0.0006 0.0000 0 0.0038 0.0001 IRTSX_SF_ENG_3 0.0072 -0.0043 0.0000 0 0.0051 -0.0032 lilian_KB_ENG_1 0.0452 -0.0278 0.0000 0 0.0311 -0.0195 lilian_KB_ENG_2 0.0465 -0.032 0.0000 0 0.0317 -0.022 lilian_KB_ENG_3 0.0468 -0.0294 0.0000 0 0.0318 -0.0202 MSR_SF_ENG_1 0.0275 -0.0084 0.0000 0 0.0150 -0.0024 MSR_SF_ENG_2 0.0297 -0.0268 0.0000 0 0.0198 -0.0179 MSR_SF_ENG_3 0.0201 -0.0172 0.0000 0 0.0122 -0.0103 MSR_SF_ENG_4 0.0635 -0.0241 0.0102 -0.0044 0.0377 -0.0093 MSR_SF_ENG_5 0.0176 -0.0015 0.0000 0 0.0083 0.0023 NAIST_CL_SF_ENG_1 0.0589 -0.0476 0.0100 -0.01 0.0146 -0.0075 NAIST_CL_SF_ENG_2 0.0589 -0.0476 0.0000 0 0.0438 -0.0362 NAIST_CL_SF_ENG_3 0.0589 -0.0476 0.0000 0 0.0433 -0.0357 RPI_BLENDER_SF_ENG_1 0.1904 -0.0742 0.1050 -0.0383 0.1644 -0.0641 RPI_BLENDER_SF_ENG_2 0.1721 -0.0708 0.0960 -0.0193 0.1487 -0.0554 RPI_BLENDER_SF_ENG_3 0.1700 -0.0687 0.0950 -0.0183 0.1470 -0.0537 SoochowNLP_SF_ENG_1 0.0451 -0.0109 0.0000 0 0.0363 -0.0127 SoochowNLP_SF_ENG_2 0.0372 -0.0178 0.0000 0 0.0275 -0.0143 SoochowNLP_SF_ENG_3 0.0372 -0.0178 0.0000 0 0.0192 -0.0061 SoochowNLP_SF_ENG_4 0.0372 -0.0178 0.0000 0 0.0262 -0.013 SoochowNLP_SF_ENG_5 0.0372 -0.0178 0.0000 0 0.0254 -0.0122 Stanford_KB_ENG_1 0.2406 -0.1166 0.1466 -0.1126 0.2100 -0.1146 Stanford_KB_ENG_2 0.1926 -0.1068 0.1039 -0.0865 0.1648 -0.1011 Stanford_KB_ENG_3 0.2633 -0.121 0.1190 -0.086 0.2018 -0.0936 Stanford_SF_ENG_1 0.2310 -0.1235 0.1197 -0.1024 0.1945 -0.1156 Stanford_SF_ENG_2 0.2444 -0.1065 0.1241 -0.0906 0.1971 -0.0919 Stanford_SF_ENG_3 0.2846 -0.1486 0.0819 -0.0333 0.1751 -0.0665 Stanford_SF_ENG_4 0.2044 -0.1013 0.1061 -0.0888 0.1735 -0.0978 Stanford_SF_ENG_5 0.2310 -0.1236 0.0798 -0.074 0.1827 -0.1075 summa_KB_ENG_1 0.0255 -0.0197 0.0000 0 0.0171 -0.0132 summa_KB_ENG_2 0.0584 -0.0381 0.0114 0.0002 0.0430 -0.0256 summa_KB_ENG_5 0.0261 -0.0085 0.0116 -0.0058 0.0213 -0.0077 UMass_IESL_KB_ENG_1 0.1844 -0.1059 0.0628 -0.0408 0.1252 -0.0654 UMass_IESL_KB_ENG_2 0.1704 -0.0854 0.0687 -0.0347 0.1267 -0.0583 UMass_IESL_KB_ENG_3 0.2076 -0.1291 0.1480 -0.1193 0.1878 -0.1256 UMass_IESL_KB_ENG_4 0.1624 -0.0889 0.1061 -0.0838 0.1423 -0.0856 UMass_IESL_KB_ENG_5 0.1732 -0.0906 0.0399 -0.0129 0.0885 -0.0243 UMass_IESL_SF_ENG_1 0.2207 -0.1177 0.0282 -0.0282 0.1794 -0.1078 UMass_IESL_SF_ENG_2 0.2224 -0.1334 0.0384 -0.0384 0.1751 -0.1136 UMass_IESL_SF_ENG_3 0.2092 -0.1046 0.0954 -0.0786 0.1668 -0.0906 UMass_IESL_SF_ENG_4 0.2061 -0.1106 0.0282 -0.0282 0.1643 -0.0985 UMass_IESL_SF_ENG_5 0.2070 -0.0937 0.0285 -0.0285 0.1627 -0.084 UNIST_SAIL_SF_ENG_1 0.0438 -0.0264 0.0000 0 0.0109 0.0008 UNIST_SAIL_SF_ENG_2 0.0476 -0.0274 0.0000 0 0.0352 -0.0217 UNIST_SAIL_SF_ENG_3 0.0583 -0.0219 0.0000 0 0.0426 -0.018 UNIST_SAIL_SF_ENG_4 0.0788 -0.0403 0.0000 0 0.0610 -0.0348 Average change in F -0.0708314 -0.0339986 -0.0559843 MAX micro-average F1 and change in F1 after applying SFV filter, at each hop level: Cold Start run hop0_F Post-filter hop1_F Post-filter ALL_F Post-filter BBN_KB_ENG_1 0.3358 -0.165 0.1659 -0.1086 0.2781 -0.143 BBN_KB_ENG_2 0.3234 -0.1663 0.1542 -0.084 0.2655 -0.136 BBN_KB_ENG_3 0.3313 -0.1726 0.1648 -0.101 0.2716 -0.1431 BBN_KB_ENG_4 0.3375 -0.1667 0.1655 -0.1082 0.2791 -0.144 BBN_KB_ENG_5 0.3253 -0.174 0.1442 -0.0555 0.2550 -0.124 CMUML_SF_ENG_1 0.0462 -0.0275 0.0000 0 0.0309 -0.0183 CMUML_SF_ENG_2 0.0409 -0.0147 0.0000 0 0.0276 -0.01 CMUML_SF_ENG_3 0.0074 -0.0074 0.0000 0 0.0049 -0.0049 CMUML_SF_ENG_4 0.0370 -0.022 0.0000 0 0.0248 -0.0147 CMUML_SF_ENG_5 0.0463 -0.0275 0.0000 0 0.0310 -0.0184 DCD_SF_SF_ENG_1 0.0889 -0.0581 0.0219 -0.0219 0.0652 -0.0444 DCD_SF_SF_ENG_2 0.1015 -0.0742 0.0208 -0.0208 0.0728 -0.0545 doughnutPRIS_SF_ENG_4 0.0847 -0.0413 0.0000 0 0.0631 -0.0335 doughnutPRIS_SF_ENG_5 0.1138 -0.0609 0.0000 0 0.0887 -0.0524 hltcoe_KB_ENG_1 0.2912 -0.1371 0.1082 -0.0709 0.2222 -0.1043 hltcoe_KB_ENG_2 0.2887 -0.1524 0.1014 -0.0938 0.2204 -0.1238 hltcoe_KB_ENG_3 0.2891 -0.1551 0.1000 -0.0924 0.2191 -0.1243 hltcoe_KB_ENG_4 0.2735 -0.1289 0.0734 -0.0508 0.1872 -0.0809 hltcoe_KB_ENG_5 0.2925 -0.1353 0.1082 -0.0707 0.2232 -0.103 ICTCAS_OKN_KB_ENG_1 0.1507 -0.0944 0.0445 -0.0368 0.1120 -0.071 ICTCAS_OKN_KB_ENG_2 0.1521 -0.0891 0.0401 -0.0401 0.1115 -0.0682 ICTCAS_OKN_KB_ENG_3 0.1441 -0.0834 0.0656 -0.0284 0.1166 -0.0635 ICTCAS_OKN_KB_ENG_4 0.1471 -0.0911 0.0216 -0.0139 0.1055 -0.0646 IRTSX_SF_ENG_1 0.0081 -0.0044 0.0000 0 0.0050 -0.0025 IRTSX_SF_ENG_2 0.0088 -0.0013 0.0000 0 0.0059 -0.0008 IRTSX_SF_ENG_3 0.0097 -0.0059 0.0000 0 0.0068 -0.0043 lilian_KB_ENG_1 0.0446 -0.0258 0.0000 0 0.0308 -0.0181 lilian_KB_ENG_2 0.0462 -0.0311 0.0000 0 0.0316 -0.0214 lilian_KB_ENG_3 0.0467 -0.0279 0.0000 0 0.0318 -0.0192 MSR_SF_ENG_1 0.0284 -0.0072 0.0000 0 0.0153 -0.0013 MSR_SF_ENG_2 0.0343 -0.0306 0.0000 0 0.0228 -0.0203 MSR_SF_ENG_3 0.0247 -0.0209 0.0000 0 0.0150 -0.0125 MSR_SF_ENG_4 0.0661 -0.0331 0.0117 -0.0041 0.0393 -0.0145 MSR_SF_ENG_5 0.0207 -0.0031 0.0000 0 0.0096 0.002 NAIST_CL_SF_ENG_1 0.0535 -0.0461 0.0092 -0.0092 0.0138 -0.0092 NAIST_CL_SF_ENG_2 0.0534 -0.046 0.0000 0 0.0397 -0.0347 NAIST_CL_SF_ENG_3 0.0533 -0.0459 0.0000 0 0.0391 -0.0341 RPI_BLENDER_SF_ENG_1 0.1888 -0.0754 0.0855 -0.0269 0.1575 -0.0616 RPI_BLENDER_SF_ENG_2 0.1647 -0.0708 0.0795 -0.0143 0.1385 -0.0539 RPI_BLENDER_SF_ENG_3 0.1628 -0.0686 0.0789 -0.0137 0.1371 -0.0523 SoochowNLP_SF_ENG_1 0.0489 -0.0085 0.0000 0 0.0390 -0.0109 SoochowNLP_SF_ENG_2 0.0450 -0.0203 0.0000 0 0.0335 -0.0165 SoochowNLP_SF_ENG_3 0.0451 -0.0202 0.0000 0 0.0272 -0.0103 SoochowNLP_SF_ENG_4 0.0452 -0.0202 0.0000 0 0.0322 -0.0151 SoochowNLP_SF_ENG_5 0.0448 -0.0199 0.0000 0 0.0311 -0.0141 Stanford_KB_ENG_1 0.2401 -0.1095 0.1554 -0.1181 0.2121 -0.1109 Stanford_KB_ENG_2 0.1957 -0.1097 0.1220 -0.0991 0.1725 -0.1066 Stanford_KB_ENG_3 0.2699 -0.1296 0.1154 -0.0792 0.2009 -0.0929 Stanford_SF_ENG_1 0.2324 -0.1205 0.1186 -0.0959 0.1938 -0.1101 Stanford_SF_ENG_2 0.2481 -0.1045 0.1191 -0.0822 0.1949 -0.0846 Stanford_SF_ENG_3 0.3003 -0.162 0.0732 -0.0228 0.1691 -0.058 Stanford_SF_ENG_4 0.2063 -0.097 0.1246 -0.1019 0.1806 -0.0988 Stanford_SF_ENG_5 0.2317 -0.12 0.0860 -0.0784 0.1844 -0.1054 summa_KB_ENG_1 0.0293 -0.0217 0.0000 0 0.0198 -0.0147 summa_KB_ENG_2 0.0645 -0.0383 0.0149 0.0004 0.0484 -0.0258 summa_KB_ENG_5 0.0338 -0.011 0.0152 -0.0075 0.0276 -0.0098 UMass_IESL_KB_ENG_1 0.1954 -0.1116 0.0468 -0.0254 0.1214 -0.0581 UMass_IESL_KB_ENG_2 0.1801 -0.0908 0.0544 -0.0245 0.1262 -0.0562 UMass_IESL_KB_ENG_3 0.2075 -0.1237 0.1361 -0.0984 0.1841 -0.1153 UMass_IESL_KB_ENG_4 0.1685 -0.0905 0.0802 -0.0584 0.1362 -0.0766 UMass_IESL_KB_ENG_5 0.1842 -0.0949 0.0296 -0.0012 0.0829 -0.0135 UMass_IESL_SF_ENG_1 0.2222 -0.1212 0.0226 -0.0226 0.1796 -0.1093 UMass_IESL_SF_ENG_2 0.2215 -0.1306 0.0336 -0.0336 0.1759 -0.1126 UMass_IESL_SF_ENG_3 0.2166 -0.1123 0.1040 -0.0822 0.1731 -0.0955 UMass_IESL_SF_ENG_4 0.2043 -0.114 0.0226 -0.0226 0.1625 -0.1002 UMass_IESL_SF_ENG_5 0.2015 -0.0861 0.0226 -0.0226 0.1563 -0.0761 UNIST_SAIL_SF_ENG_1 0.0414 -0.0263 0.0000 0 0.0136 -0.0034 UNIST_SAIL_SF_ENG_2 0.0459 -0.0271 0.0000 0 0.0340 -0.0214 UNIST_SAIL_SF_ENG_3 0.0534 -0.0172 0.0000 0 0.0391 -0.0144 UNIST_SAIL_SF_ENG_4 0.0840 -0.0447 0.0000 0 0.0653 -0.0384 Average change in F -0.0727571 -0.0306029 -0.0554143 MEAN macro-average F1 and change in F1 after applying SFV filter, at each hop level: Cold Start run hop0_F Post-filter hop1_F Post-filter ALL_F Post-filter BBN_KB_ENG_1 0.2276 -0.125 0.1128 -0.0796 0.1822 -0.107 BBN_KB_ENG_2 0.2231 -0.1227 0.0966 -0.0686 0.1731 -0.1013 BBN_KB_ENG_3 0.2322 -0.1301 0.1124 -0.079 0.1849 -0.1099 BBN_KB_ENG_4 0.2292 -0.1266 0.1127 -0.0795 0.1832 -0.108 BBN_KB_ENG_5 0.2263 -0.1355 0.1076 -0.072 0.1794 -0.1104 CMUML_SF_ENG_1 0.0333 -0.0193 0.0000 0 0.0202 -0.0117 CMUML_SF_ENG_2 0.0256 -0.0104 0.0000 0 0.0155 -0.0063 CMUML_SF_ENG_3 0.0030 -0.003 0.0000 0 0.0018 -0.0018 CMUML_SF_ENG_4 0.0299 -0.0168 0.0000 0 0.0181 -0.0102 CMUML_SF_ENG_5 0.0333 -0.0193 0.0000 0 0.0202 -0.0117 DCD_SF_SF_ENG_1 0.1007 -0.0729 0.0296 -0.0296 0.0726 -0.0558 DCD_SF_SF_ENG_2 0.1160 -0.0865 0.0270 -0.027 0.0809 -0.0631 doughnutPRIS_SF_ENG_4 0.0859 -0.0475 0.0000 0 0.0520 -0.0287 doughnutPRIS_SF_ENG_5 0.1284 -0.0775 0.0000 0 0.0777 -0.0469 hltcoe_KB_ENG_1 0.2046 -0.1065 0.1108 -0.0743 0.1675 -0.0938 hltcoe_KB_ENG_2 0.1992 -0.1161 0.0886 -0.0834 0.1555 -0.1032 hltcoe_KB_ENG_3 0.1972 -0.1095 0.0886 -0.0834 0.1543 -0.0992 hltcoe_KB_ENG_4 0.2022 -0.1121 0.1045 -0.0837 0.1636 -0.1009 hltcoe_KB_ENG_5 0.2072 -0.104 0.1108 -0.0743 0.1691 -0.0923 ICTCAS_OKN_KB_ENG_1 0.1240 -0.0821 0.0252 -0.0213 0.0850 -0.0581 ICTCAS_OKN_KB_ENG_2 0.1256 -0.0787 0.0198 -0.0198 0.0838 -0.0554 ICTCAS_OKN_KB_ENG_3 0.1002 -0.0603 0.0253 -0.0214 0.0706 -0.0449 ICTCAS_OKN_KB_ENG_4 0.1134 -0.0703 0.0140 -0.0109 0.0741 -0.0468 IRTSX_SF_ENG_1 0.0033 -0.002 0.0000 0 0.0020 -0.0012 IRTSX_SF_ENG_2 0.0037 0.0006 0.0000 0 0.0022 0.0004 IRTSX_SF_ENG_3 0.0046 -0.0031 0.0000 0 0.0028 -0.0019 lilian_KB_ENG_1 0.0270 -0.0135 0.0000 0 0.0163 -0.0081 lilian_KB_ENG_2 0.0270 -0.0169 0.0000 0 0.0163 -0.0102 lilian_KB_ENG_3 0.0275 -0.0135 0.0000 0 0.0166 -0.0081 MSR_SF_ENG_1 0.0313 -0.0169 0.0000 0 0.0189 -0.0102 MSR_SF_ENG_2 0.0427 -0.0407 0.0000 0 0.0258 -0.0246 MSR_SF_ENG_3 0.0223 -0.021 0.0000 0 0.0135 -0.0127 MSR_SF_ENG_4 0.0735 -0.0425 0.0215 -0.0189 0.0529 -0.0331 MSR_SF_ENG_5 0.0210 -0.0034 0.0000 0 0.0127 -0.0021 NAIST_CL_SF_ENG_1 0.0935 -0.085 0.0199 -0.0199 0.0644 -0.0593 NAIST_CL_SF_ENG_2 0.0935 -0.085 0.0000 0 0.0566 -0.0515 NAIST_CL_SF_ENG_3 0.0935 -0.085 0.0000 0 0.0566 -0.0515 RPI_BLENDER_SF_ENG_1 0.1858 -0.0673 0.0418 -0.0075 0.1289 -0.0437 RPI_BLENDER_SF_ENG_2 0.1551 -0.0565 0.0411 -0.0071 0.1101 -0.037 RPI_BLENDER_SF_ENG_3 0.1541 -0.0555 0.0405 -0.0065 0.1092 -0.0361 SoochowNLP_SF_ENG_1 0.0251 -0.0057 0.0000 0 0.0152 -0.0035 SoochowNLP_SF_ENG_2 0.0206 -0.008 0.0000 0 0.0125 -0.0049 SoochowNLP_SF_ENG_3 0.0206 -0.008 0.0000 0 0.0125 -0.0049 SoochowNLP_SF_ENG_4 0.0206 -0.008 0.0000 0 0.0125 -0.0049 SoochowNLP_SF_ENG_5 0.0206 -0.008 0.0000 0 0.0125 -0.0049 Stanford_KB_ENG_1 0.2149 -0.105 0.1087 -0.0735 0.1730 -0.0926 Stanford_KB_ENG_2 0.1519 -0.0858 0.0700 -0.0505 0.1195 -0.0718 Stanford_KB_ENG_3 0.2553 -0.1375 0.1526 -0.1174 0.2147 -0.1296 Stanford_SF_ENG_1 0.1876 -0.0982 0.0788 -0.0593 0.1446 -0.0828 Stanford_SF_ENG_2 0.2334 -0.114 0.1125 -0.0773 0.1856 -0.0995 Stanford_SF_ENG_3 0.2640 -0.1594 0.1638 -0.113 0.2244 -0.1411 Stanford_SF_ENG_4 0.1733 -0.0936 0.0615 -0.042 0.1291 -0.0732 Stanford_SF_ENG_5 0.1876 -0.0982 0.0379 -0.0301 0.1285 -0.0713 summa_KB_ENG_1 0.0068 -0.0047 0.0000 0 0.0041 -0.0028 summa_KB_ENG_2 0.0202 -0.009 0.0156 0 0.0184 -0.0055 summa_KB_ENG_5 0.0116 -0.0038 0.0156 -0.0078 0.0132 -0.0054 UMass_IESL_KB_ENG_1 0.1375 -0.0674 0.0672 -0.0568 0.1097 -0.0632 UMass_IESL_KB_ENG_2 0.1240 -0.053 0.0581 -0.0446 0.0980 -0.0497 UMass_IESL_KB_ENG_3 0.1204 -0.0643 0.1093 -0.0702 0.1160 -0.0667 UMass_IESL_KB_ENG_4 0.1266 -0.0629 0.0587 -0.0478 0.0998 -0.0569 UMass_IESL_KB_ENG_5 0.1375 -0.066 0.0834 -0.0655 0.1161 -0.0658 UMass_IESL_SF_ENG_1 0.1842 -0.1046 0.0102 -0.0102 0.1154 -0.0672 UMass_IESL_SF_ENG_2 0.1648 -0.0966 0.0258 -0.0258 0.1099 -0.0686 UMass_IESL_SF_ENG_3 0.1598 -0.088 0.0864 -0.063 0.1308 -0.0781 UMass_IESL_SF_ENG_4 0.1624 -0.0923 0.0102 -0.0102 0.1023 -0.0599 UMass_IESL_SF_ENG_5 0.1637 -0.0728 0.0102 -0.0102 0.1031 -0.0481 UNIST_SAIL_SF_ENG_1 0.0263 -0.0111 0.0000 0 0.0159 -0.0067 UNIST_SAIL_SF_ENG_2 0.0270 -0.0113 0.0000 0 0.0164 -0.0069 UNIST_SAIL_SF_ENG_3 0.0307 -0.0104 0.0000 0 0.0186 -0.0063 UNIST_SAIL_SF_ENG_4 0.0491 -0.0284 0.0000 0 0.0297 -0.0172 Average change in F -0.0602271 -0.0277557 -0.0474043 ************************************************************* Run ID: IRTSX4 Did the run access the live Web during the evaluation window: No Did this run judge each candidate slot filler independently of all other candidate slot fillers in the evaluation dataset: Yes Did this run judge candidate slot fillers for each slot-filling run independently of all other slot-filling runs in the evaluation dataset: Yes Did this run judge candidate slot fillers for each slot-filling team independently of all other slot-filling teams in the evaluation dataset: Yes Did this run make use of the slot filler or justification offsets provided for each candidate slot filler: Yes Did this run make use of the confidence values provided for each candidate slot filler: Yes Did this run make use of the system profiles for the slot filling runs: Yes Did this run make use of the preliminary assessments provided for some of the slot filler candidates: No CSSF micro-average F1 and change in F1 after applying SFV filter, at each hop level: Cold Start run hop0_F Post-filter hop1_F Post-filter ALL_F Post-filter BBN_KB_ENG_1 0.3174 -0.144 0.1737 -0.1055 0.2693 -0.13 BBN_KB_ENG_2 0.3040 -0.1344 0.1650 -0.1171 0.2571 -0.1264 BBN_KB_ENG_3 0.3150 -0.1534 0.1724 -0.1047 0.2651 -0.1341 BBN_KB_ENG_4 0.3188 -0.1434 0.1734 -0.1052 0.2703 -0.1295 BBN_KB_ENG_5 0.3041 -0.1408 0.1529 -0.0971 0.2475 -0.1198 CMUML_SF_ENG_1 0.0440 -0.0295 0.0000 0 0.0291 -0.0194 CMUML_SF_ENG_2 0.0371 -0.0112 0.0000 0 0.0249 -0.0077 CMUML_SF_ENG_3 0.0057 -0.0057 0.0000 0 0.0038 -0.0038 CMUML_SF_ENG_4 0.0369 -0.0223 0.0000 0 0.0246 -0.0149 CMUML_SF_ENG_5 0.0440 -0.0266 0.0000 0 0.0291 -0.0175 DCD_SF_SF_ENG_1 0.0804 -0.0227 0.0198 -0.0151 0.0588 -0.0191 DCD_SF_SF_ENG_2 0.0904 -0.0204 0.0186 -0.0091 0.0647 -0.0151 doughnutPRIS_SF_ENG_4 0.0865 0.0037 0.0000 0 0.0648 -0.0021 doughnutPRIS_SF_ENG_5 0.1143 0.0026 0.0000 0 0.0894 -0.0068 hltcoe_KB_ENG_1 0.2949 -0.0915 0.1202 -0.0706 0.2300 -0.0774 hltcoe_KB_ENG_2 0.2953 -0.1224 0.1180 -0.1011 0.2311 -0.1071 hltcoe_KB_ENG_3 0.2976 -0.1251 0.1173 -0.1004 0.2320 -0.1082 hltcoe_KB_ENG_4 0.2776 -0.1062 0.0830 -0.0772 0.1959 -0.0759 hltcoe_KB_ENG_5 0.2966 -0.0939 0.1202 -0.0707 0.2311 -0.079 ICTCAS_OKN_KB_ENG_1 0.1487 -0.0347 0.0441 0.0033 0.1101 -0.0181 ICTCAS_OKN_KB_ENG_2 0.1490 -0.0303 0.0374 0.0101 0.1081 -0.0128 ICTCAS_OKN_KB_ENG_3 0.1186 -0.0245 0.0526 -0.0051 0.0960 -0.0176 ICTCAS_OKN_KB_ENG_4 0.1421 -0.0241 0.0315 -0.0205 0.1041 -0.0208 IRTSX_SF_ENG_1 0.0062 -0.0033 0.0000 0 0.0038 -0.0019 IRTSX_SF_ENG_2 0.0064 -0.0035 0.0000 0 0.0038 -0.0019 IRTSX_SF_ENG_3 0.0072 -0.0043 0.0000 0 0.0051 -0.0031 lilian_KB_ENG_1 0.0452 -0.0364 0.0000 0 0.0311 -0.0253 lilian_KB_ENG_2 0.0465 -0.0377 0.0000 0 0.0317 -0.0258 lilian_KB_ENG_3 0.0468 -0.0322 0.0000 0 0.0318 -0.0221 MSR_SF_ENG_1 0.0275 -0.0085 0.0000 0 0.0150 -0.0028 MSR_SF_ENG_2 0.0297 -0.0142 0.0000 0 0.0198 -0.0092 MSR_SF_ENG_3 0.0201 -0.0056 0.0000 0 0.0122 -0.0025 MSR_SF_ENG_4 0.0635 -0.0275 0.0102 -0.001 0.0377 -0.0113 MSR_SF_ENG_5 0.0176 0.0009 0.0000 0 0.0083 0.003 NAIST_CL_SF_ENG_1 0.0589 0.0023 0.0100 0.0022 0.0146 0.0164 NAIST_CL_SF_ENG_2 0.0589 0.0023 0.0000 0 0.0438 -0.0006 NAIST_CL_SF_ENG_3 0.0589 0.0023 0.0000 0 0.0433 -0.0001 RPI_BLENDER_SF_ENG_1 0.1904 -0.0583 0.1050 -0.0285 0.1644 -0.0502 RPI_BLENDER_SF_ENG_2 0.1721 -0.0589 0.0960 -0.0195 0.1487 -0.0475 RPI_BLENDER_SF_ENG_3 0.1700 -0.057 0.0950 -0.0185 0.1470 -0.0458 SoochowNLP_SF_ENG_1 0.0451 -0.0002 0.0000 0 0.0363 -0.0053 SoochowNLP_SF_ENG_2 0.0372 -0.01 0.0000 0 0.0275 -0.0089 SoochowNLP_SF_ENG_3 0.0372 -0.01 0.0000 0 0.0192 -0.001 SoochowNLP_SF_ENG_4 0.0372 -0.01 0.0000 0 0.0262 -0.0079 SoochowNLP_SF_ENG_5 0.0372 -0.01 0.0000 0 0.0254 -0.0076 Stanford_KB_ENG_1 0.2406 -0.1436 0.1466 -0.0899 0.2100 -0.1266 Stanford_KB_ENG_2 0.1926 -0.1201 0.1039 -0.0981 0.1648 -0.1139 Stanford_KB_ENG_3 0.2633 -0.1114 0.1190 -0.0836 0.2018 -0.0883 Stanford_SF_ENG_1 0.2310 -0.0328 0.1197 -0.1197 0.1945 -0.0561 Stanford_SF_ENG_2 0.2444 -0.1466 0.1241 -0.1015 0.1971 -0.1233 Stanford_SF_ENG_3 0.2846 -0.1378 0.0819 -0.0509 0.1751 -0.0663 Stanford_SF_ENG_4 0.2044 -0.1402 0.1061 -0.1061 0.1735 -0.1301 Stanford_SF_ENG_5 0.2310 -0.0277 0.0798 -0.0133 0.1827 -0.0221 summa_KB_ENG_1 0.0255 -0.0052 0.0000 0 0.0171 -0.0035 summa_KB_ENG_2 0.0584 -0.0353 0.0114 -0.0056 0.0430 -0.0256 summa_KB_ENG_5 0.0261 -0.0173 0.0116 -0.0058 0.0213 -0.0135 UMass_IESL_KB_ENG_1 0.1844 -0.1205 0.0628 -0.0512 0.1252 -0.0783 UMass_IESL_KB_ENG_2 0.1704 -0.1116 0.0687 -0.0571 0.1267 -0.0833 UMass_IESL_KB_ENG_3 0.2076 -0.1403 0.1480 -0.1422 0.1878 -0.1405 UMass_IESL_KB_ENG_4 0.1624 -0.0987 0.1061 -0.0717 0.1423 -0.0881 UMass_IESL_KB_ENG_5 0.1732 -0.0887 0.0399 -0.0091 0.0885 -0.0222 UMass_IESL_SF_ENG_1 0.2207 -0.088 0.0282 -0.0108 0.1794 -0.082 UMass_IESL_SF_ENG_2 0.2224 -0.0953 0.0384 -0.0268 0.1751 -0.0836 UMass_IESL_SF_ENG_3 0.2092 -0.0845 0.0954 -0.0326 0.1668 -0.0629 UMass_IESL_SF_ENG_4 0.2061 -0.0808 0.0282 -0.0108 0.1643 -0.0725 UMass_IESL_SF_ENG_5 0.2070 -0.0767 0.0285 -0.0285 0.1627 -0.072 UNIST_SAIL_SF_ENG_1 0.0438 -0.0107 0.0000 0 0.0109 0.0114 UNIST_SAIL_SF_ENG_2 0.0476 -0.0096 0.0000 0 0.0352 -0.0092 UNIST_SAIL_SF_ENG_3 0.0583 -0.0226 0.0000 0 0.0426 -0.0183 UNIST_SAIL_SF_ENG_4 0.0788 -0.0331 0.0000 0 0.0610 -0.0297 Average change in F -0.0551386 -0.0309514 -0.0446429 MAX micro-average F1 and change in F1 after applying SFV filter, at each hop level: Cold Start run hop0_F Post-filter hop1_F Post-filter ALL_F Post-filter BBN_KB_ENG_1 0.3358 -0.1522 0.1659 -0.0976 0.2781 -0.1319 BBN_KB_ENG_2 0.3234 -0.1419 0.1542 -0.1059 0.2655 -0.1266 BBN_KB_ENG_3 0.3313 -0.16 0.1648 -0.0972 0.2716 -0.1343 BBN_KB_ENG_4 0.3375 -0.1515 0.1655 -0.0972 0.2791 -0.1312 BBN_KB_ENG_5 0.3253 -0.1545 0.1442 -0.0919 0.2550 -0.1239 CMUML_SF_ENG_1 0.0462 -0.0273 0.0000 0 0.0309 -0.0182 CMUML_SF_ENG_2 0.0409 -0.0074 0.0000 0 0.0276 -0.0053 CMUML_SF_ENG_3 0.0074 -0.0074 0.0000 0 0.0049 -0.0049 CMUML_SF_ENG_4 0.0370 -0.018 0.0000 0 0.0248 -0.0121 CMUML_SF_ENG_5 0.0463 -0.0237 0.0000 0 0.0310 -0.0158 DCD_SF_SF_ENG_1 0.0889 -0.0207 0.0219 -0.0157 0.0652 -0.0176 DCD_SF_SF_ENG_2 0.1015 -0.0229 0.0208 -0.0081 0.0728 -0.016 doughnutPRIS_SF_ENG_4 0.0847 0.007 0.0000 0 0.0631 0.0006 doughnutPRIS_SF_ENG_5 0.1138 0.0048 0.0000 0 0.0887 -0.0049 hltcoe_KB_ENG_1 0.2912 -0.0823 0.1082 -0.0681 0.2222 -0.0675 hltcoe_KB_ENG_2 0.2887 -0.1078 0.1014 -0.0865 0.2204 -0.0909 hltcoe_KB_ENG_3 0.2891 -0.1091 0.1000 -0.0851 0.2191 -0.0901 hltcoe_KB_ENG_4 0.2735 -0.0981 0.0734 -0.0659 0.1872 -0.0637 hltcoe_KB_ENG_5 0.2925 -0.0846 0.1082 -0.0685 0.2232 -0.0695 ICTCAS_OKN_KB_ENG_1 0.1507 -0.0305 0.0445 0.0116 0.1120 -0.0125 ICTCAS_OKN_KB_ENG_2 0.1521 -0.0248 0.0401 0.0162 0.1115 -0.0071 ICTCAS_OKN_KB_ENG_3 0.1441 -0.0262 0.0656 -0.0048 0.1166 -0.0181 ICTCAS_OKN_KB_ENG_4 0.1471 -0.0244 0.0216 -0.0144 0.1055 -0.0198 IRTSX_SF_ENG_1 0.0081 -0.0043 0.0000 0 0.0050 -0.0025 IRTSX_SF_ENG_2 0.0088 -0.005 0.0000 0 0.0059 -0.0034 IRTSX_SF_ENG_3 0.0097 -0.0059 0.0000 0 0.0068 -0.0043 lilian_KB_ENG_1 0.0446 -0.037 0.0000 0 0.0308 -0.0257 lilian_KB_ENG_2 0.0462 -0.0386 0.0000 0 0.0316 -0.0265 lilian_KB_ENG_3 0.0467 -0.0316 0.0000 0 0.0318 -0.0217 MSR_SF_ENG_1 0.0284 -0.0068 0.0000 0 0.0153 -0.0015 MSR_SF_ENG_2 0.0343 -0.0141 0.0000 0 0.0228 -0.0089 MSR_SF_ENG_3 0.0247 -0.0096 0.0000 0 0.0150 -0.0049 MSR_SF_ENG_4 0.0661 -0.0263 0.0117 0.0002 0.0393 -0.0094 MSR_SF_ENG_5 0.0207 0.0005 0.0000 0 0.0096 0.0032 NAIST_CL_SF_ENG_1 0.0535 0.0036 0.0092 -0.0001 0.0138 0.0129 NAIST_CL_SF_ENG_2 0.0534 0.004 0.0000 0 0.0397 0.0009 NAIST_CL_SF_ENG_3 0.0533 0.0038 0.0000 0 0.0391 0.0014 RPI_BLENDER_SF_ENG_1 0.1888 -0.0577 0.0855 -0.0205 0.1575 -0.0474 RPI_BLENDER_SF_ENG_2 0.1647 -0.0591 0.0795 -0.0148 0.1385 -0.046 RPI_BLENDER_SF_ENG_3 0.1628 -0.0574 0.0789 -0.0142 0.1371 -0.0447 SoochowNLP_SF_ENG_1 0.0489 0.0029 0.0000 0 0.0390 -0.0032 SoochowNLP_SF_ENG_2 0.0450 -0.0131 0.0000 0 0.0335 -0.0117 SoochowNLP_SF_ENG_3 0.0451 -0.0134 0.0000 0 0.0272 -0.0058 SoochowNLP_SF_ENG_4 0.0452 -0.0134 0.0000 0 0.0322 -0.0107 SoochowNLP_SF_ENG_5 0.0448 -0.0128 0.0000 0 0.0311 -0.01 Stanford_KB_ENG_1 0.2401 -0.1277 0.1554 -0.083 0.2121 -0.1133 Stanford_KB_ENG_2 0.1957 -0.1202 0.1220 -0.1143 0.1725 -0.1186 Stanford_KB_ENG_3 0.2699 -0.0963 0.1154 -0.0701 0.2009 -0.0699 Stanford_SF_ENG_1 0.2324 -0.0271 0.1186 -0.1186 0.1938 -0.0506 Stanford_SF_ENG_2 0.2481 -0.1347 0.1191 -0.0895 0.1949 -0.108 Stanford_SF_ENG_3 0.3003 -0.1411 0.0732 -0.0336 0.1691 -0.0499 Stanford_SF_ENG_4 0.2063 -0.1338 0.1246 -0.1246 0.1806 -0.1313 Stanford_SF_ENG_5 0.2317 -0.0241 0.0860 -0.0066 0.1844 -0.017 summa_KB_ENG_1 0.0293 -0.0068 0.0000 0 0.0198 -0.0047 summa_KB_ENG_2 0.0645 -0.0384 0.0149 -0.0072 0.0484 -0.0283 summa_KB_ENG_5 0.0338 -0.0223 0.0152 -0.0075 0.0276 -0.0174 UMass_IESL_KB_ENG_1 0.1954 -0.1235 0.0468 -0.0392 0.1214 -0.0701 UMass_IESL_KB_ENG_2 0.1801 -0.1183 0.0544 -0.0468 0.1262 -0.0819 UMass_IESL_KB_ENG_3 0.2075 -0.1345 0.1361 -0.1284 0.1841 -0.1322 UMass_IESL_KB_ENG_4 0.1685 -0.1005 0.0802 -0.0574 0.1362 -0.0827 UMass_IESL_KB_ENG_5 0.1842 -0.0852 0.0296 -0.003 0.0829 -0.0085 UMass_IESL_SF_ENG_1 0.2222 -0.0855 0.0226 -0.0073 0.1796 -0.0798 UMass_IESL_SF_ENG_2 0.2215 -0.0897 0.0336 -0.0183 0.1759 -0.0798 UMass_IESL_SF_ENG_3 0.2166 -0.092 0.1040 -0.0426 0.1731 -0.0698 UMass_IESL_SF_ENG_4 0.2043 -0.0846 0.0226 -0.0073 0.1625 -0.0751 UMass_IESL_SF_ENG_5 0.2015 -0.0646 0.0226 -0.0226 0.1563 -0.0607 UNIST_SAIL_SF_ENG_1 0.0414 -0.0025 0.0000 0 0.0136 0.0128 UNIST_SAIL_SF_ENG_2 0.0459 -0.0039 0.0000 0 0.0340 -0.0051 UNIST_SAIL_SF_ENG_3 0.0534 -0.0111 0.0000 0 0.0391 -0.01 UNIST_SAIL_SF_ENG_4 0.0840 -0.0286 0.0000 0 0.0653 -0.0271 Average change in F -0.0535971 -0.0279486 -0.04186 MEAN macro-average F1 and change in F1 after applying SFV filter, at each hop level: Cold Start run hop0_F Post-filter hop1_F Post-filter ALL_F Post-filter BBN_KB_ENG_1 0.2276 -0.1111 0.1128 -0.0842 0.1822 -0.1004 BBN_KB_ENG_2 0.2231 -0.1054 0.0966 -0.0747 0.1731 -0.0932 BBN_KB_ENG_3 0.2322 -0.1221 0.1124 -0.0838 0.1849 -0.107 BBN_KB_ENG_4 0.2292 -0.111 0.1127 -0.0841 0.1832 -0.1004 BBN_KB_ENG_5 0.2263 -0.1155 0.1076 -0.0805 0.1794 -0.1017 CMUML_SF_ENG_1 0.0333 -0.0168 0.0000 0 0.0202 -0.0102 CMUML_SF_ENG_2 0.0256 -0.0046 0.0000 0 0.0155 -0.0028 CMUML_SF_ENG_3 0.0030 -0.003 0.0000 0 0.0018 -0.0018 CMUML_SF_ENG_4 0.0299 -0.0134 0.0000 0 0.0181 -0.0081 CMUML_SF_ENG_5 0.0333 -0.0155 0.0000 0 0.0202 -0.0094 DCD_SF_SF_ENG_1 0.1007 -0.0263 0.0296 -0.0285 0.0726 -0.0271 DCD_SF_SF_ENG_2 0.1160 -0.0247 0.0270 -0.0207 0.0809 -0.0231 doughnutPRIS_SF_ENG_4 0.0859 0.0047 0.0000 0 0.0520 0.0028 doughnutPRIS_SF_ENG_5 0.1284 0 0.0000 0 0.0777 -0.0001 hltcoe_KB_ENG_1 0.2046 -0.0694 0.1108 -0.0811 0.1675 -0.074 hltcoe_KB_ENG_2 0.1992 -0.088 0.0886 -0.0827 0.1555 -0.0859 hltcoe_KB_ENG_3 0.1972 -0.0851 0.0886 -0.0827 0.1543 -0.0842 hltcoe_KB_ENG_4 0.2022 -0.0955 0.1045 -0.0967 0.1636 -0.096 hltcoe_KB_ENG_5 0.2072 -0.0746 0.1108 -0.0811 0.1691 -0.0771 ICTCAS_OKN_KB_ENG_1 0.1240 -0.0151 0.0252 -0.0047 0.0850 -0.011 ICTCAS_OKN_KB_ENG_2 0.1256 -0.0129 0.0198 -0.0008 0.0838 -0.0081 ICTCAS_OKN_KB_ENG_3 0.1002 -0.0092 0.0253 -0.0041 0.0706 -0.0072 ICTCAS_OKN_KB_ENG_4 0.1134 -0.0071 0.0140 -0.0075 0.0741 -0.0072 IRTSX_SF_ENG_1 0.0033 -0.0013 0.0000 0 0.0020 -0.0008 IRTSX_SF_ENG_2 0.0037 -0.0011 0.0000 0 0.0022 -0.0007 IRTSX_SF_ENG_3 0.0046 -0.0033 0.0000 0 0.0028 -0.002 lilian_KB_ENG_1 0.0270 -0.0213 0.0000 0 0.0163 -0.0128 lilian_KB_ENG_2 0.0270 -0.0213 0.0000 0 0.0163 -0.0128 lilian_KB_ENG_3 0.0275 -0.0184 0.0000 0 0.0166 -0.0111 MSR_SF_ENG_1 0.0313 -0.0152 0.0000 0 0.0189 -0.0092 MSR_SF_ENG_2 0.0427 -0.0177 0.0000 0 0.0258 -0.0107 MSR_SF_ENG_3 0.0223 -0.0142 0.0000 0 0.0135 -0.0086 MSR_SF_ENG_4 0.0735 -0.0501 0.0215 -0.0132 0.0529 -0.0354 MSR_SF_ENG_5 0.0210 -0.0027 0.0000 0 0.0127 -0.0017 NAIST_CL_SF_ENG_1 0.0935 -0.0136 0.0199 -0.0088 0.0644 -0.0116 NAIST_CL_SF_ENG_2 0.0935 -0.0136 0.0000 0 0.0566 -0.0082 NAIST_CL_SF_ENG_3 0.0935 -0.0136 0.0000 0 0.0566 -0.0082 RPI_BLENDER_SF_ENG_1 0.1858 -0.0538 0.0418 -0.0078 0.1289 -0.0356 RPI_BLENDER_SF_ENG_2 0.1551 -0.0461 0.0411 -0.0071 0.1101 -0.0307 RPI_BLENDER_SF_ENG_3 0.1541 -0.0451 0.0405 -0.0065 0.1092 -0.0298 SoochowNLP_SF_ENG_1 0.0251 0.0081 0.0000 0 0.0152 0.0049 SoochowNLP_SF_ENG_2 0.0206 0.002 0.0000 0 0.0125 0.0012 SoochowNLP_SF_ENG_3 0.0206 0.002 0.0000 0 0.0125 0.0012 SoochowNLP_SF_ENG_4 0.0206 0.002 0.0000 0 0.0125 0.0012 SoochowNLP_SF_ENG_5 0.0206 0.002 0.0000 0 0.0125 0.0012 Stanford_KB_ENG_1 0.2149 -0.1238 0.1087 -0.0753 0.1730 -0.1047 Stanford_KB_ENG_2 0.1519 -0.0939 0.0700 -0.0674 0.1195 -0.0834 Stanford_KB_ENG_3 0.2553 -0.1178 0.1526 -0.134 0.2147 -0.1241 Stanford_SF_ENG_1 0.1876 -0.01 0.0788 -0.0788 0.1446 -0.0372 Stanford_SF_ENG_2 0.2334 -0.152 0.1125 -0.1068 0.1856 -0.1341 Stanford_SF_ENG_3 0.2640 -0.1516 0.1638 -0.1404 0.2244 -0.1471 Stanford_SF_ENG_4 0.1733 -0.1307 0.0615 -0.0615 0.1291 -0.1033 Stanford_SF_ENG_5 0.1876 -0.0075 0.0379 0.0014 0.1285 -0.004 summa_KB_ENG_1 0.0068 -0.0017 0.0000 0 0.0041 -0.001 summa_KB_ENG_2 0.0202 -0.0132 0.0156 -0.0078 0.0184 -0.0111 summa_KB_ENG_5 0.0116 -0.0047 0.0156 -0.0078 0.0132 -0.0059 UMass_IESL_KB_ENG_1 0.1375 -0.0985 0.0672 -0.062 0.1097 -0.0841 UMass_IESL_KB_ENG_2 0.1240 -0.0861 0.0581 -0.0529 0.0980 -0.073 UMass_IESL_KB_ENG_3 0.1204 -0.0738 0.1093 -0.1015 0.1160 -0.0847 UMass_IESL_KB_ENG_4 0.1266 -0.0765 0.0587 -0.0509 0.0998 -0.0664 UMass_IESL_KB_ENG_5 0.1375 -0.0797 0.0834 -0.0696 0.1161 -0.0756 UMass_IESL_SF_ENG_1 0.1842 -0.0823 0.0102 -0.0011 0.1154 -0.0502 UMass_IESL_SF_ENG_2 0.1648 -0.069 0.0258 -0.0193 0.1099 -0.0494 UMass_IESL_SF_ENG_3 0.1598 -0.0573 0.0864 -0.0446 0.1308 -0.0523 UMass_IESL_SF_ENG_4 0.1624 -0.0668 0.0102 -0.0011 0.1023 -0.0409 UMass_IESL_SF_ENG_5 0.1637 -0.0636 0.0102 -0.0102 0.1031 -0.0425 UNIST_SAIL_SF_ENG_1 0.0263 0.0033 0.0000 0 0.0159 0.002 UNIST_SAIL_SF_ENG_2 0.0270 0.0051 0.0000 0 0.0164 0.003 UNIST_SAIL_SF_ENG_3 0.0307 0.0006 0.0000 0 0.0186 0.0004 UNIST_SAIL_SF_ENG_4 0.0491 -0.014 0.0000 0 0.0297 -0.0085 Average change in F -0.0431914 -0.0290414 -0.0375929