================================================================ TAC 2020 EPIDEMIC QUESTION ANSWERING EVALUATION RESULTS (TASK B) ================================================================ Team ID: IBM Organization(s): IBM Research AI Run: IBM_1 Query Metric Score Median Maximum CQ101 NDNS-Exact 0.2700 0.1503 0.2700 CQ101 NDNS-Partial 0.2869 0.1340 0.2869 CQ101 NDNS-Relaxed 0.2894 0.1456 0.2894 CQ102 NDNS-Exact 0.4319 0.4063 0.7355 CQ102 NDNS-Partial 0.3748 0.3457 0.6011 CQ102 NDNS-Relaxed 0.3807 0.3494 0.6194 CQ103 NDNS-Exact 0.1034 0.1031 0.1767 CQ103 NDNS-Partial 0.0877 0.0875 0.1499 CQ103 NDNS-Relaxed 0.0877 0.0875 0.1499 CQ104 NDNS-Exact 0.1309 0.1333 0.4982 CQ104 NDNS-Partial 0.1309 0.1333 0.4982 CQ104 NDNS-Relaxed 0.1309 0.1333 0.4982 CQ105 NDNS-Exact 0.3303 0.3401 0.4124 CQ105 NDNS-Partial 0.3877 0.3572 0.4059 CQ105 NDNS-Relaxed 0.3084 0.3254 0.3850 CQ106 NDNS-Exact 0.0000 0.1063 0.5912 CQ106 NDNS-Partial 0.0000 0.0962 0.4821 CQ106 NDNS-Relaxed 0.0000 0.0962 0.4821 CQ107 NDNS-Exact 0.3626 0.2447 0.5969 CQ107 NDNS-Partial 0.2951 0.1991 0.4857 CQ107 NDNS-Relaxed 0.2951 0.1991 0.4857 CQ108 NDNS-Exact 0.5330 0.4565 0.7288 CQ108 NDNS-Partial 0.5492 0.4417 0.6084 CQ108 NDNS-Relaxed 0.5492 0.4100 0.6084 CQ109 NDNS-Exact 0.2520 0.3804 0.7500 CQ109 NDNS-Partial 0.2520 0.3867 0.7500 CQ109 NDNS-Relaxed 0.2520 0.3867 0.7500 CQ110 NDNS-Exact 0.4969 0.5001 0.5953 CQ110 NDNS-Partial 0.3977 0.3627 0.4440 CQ110 NDNS-Relaxed 0.3950 0.3654 0.4347 CQ111 NDNS-Exact 0.3245 0.5054 0.7144 CQ111 NDNS-Partial 0.2957 0.4649 0.6510 CQ111 NDNS-Relaxed 0.2957 0.4649 0.6510 CQ112 NDNS-Exact 0.2559 0.2450 0.3337 CQ112 NDNS-Partial 0.2285 0.2175 0.3522 CQ112 NDNS-Relaxed 0.2285 0.2175 0.3522 CQ113 NDNS-Exact 0.2199 0.0793 0.2236 CQ113 NDNS-Partial 0.1847 0.0667 0.1879 CQ113 NDNS-Relaxed 0.1847 0.0667 0.1879 CQ114 NDNS-Exact 0.0563 0.3065 0.7244 CQ114 NDNS-Partial 0.0667 0.2952 0.6859 CQ114 NDNS-Relaxed 0.0667 0.2902 0.6859 CQ115 NDNS-Exact 0.3584 0.4474 0.7055 CQ115 NDNS-Partial 0.3584 0.4699 0.8956 CQ115 NDNS-Relaxed 0.3584 0.4699 0.8956 CQ116 NDNS-Exact 0.1533 0.1467 0.1711 CQ116 NDNS-Partial 0.1264 0.1210 0.1411 CQ116 NDNS-Relaxed 0.1264 0.1210 0.1411 CQ117 NDNS-Exact 0.2117 0.2244 0.5007 CQ117 NDNS-Partial 0.2117 0.2244 0.5007 CQ117 NDNS-Relaxed 0.2117 0.2244 0.5007 CQ118 NDNS-Exact 0.4799 0.4771 0.6880 CQ118 NDNS-Partial 0.3279 0.3276 0.4794 CQ118 NDNS-Relaxed 0.3105 0.3087 0.4540 CQ119 NDNS-Exact 0.0654 0.1554 0.3275 CQ119 NDNS-Partial 0.0656 0.1559 0.3287 CQ119 NDNS-Relaxed 0.0656 0.1559 0.3287 CQ120 NDNS-Exact 0.8264 0.5451 0.8264 CQ120 NDNS-Partial 0.7665 0.5149 0.7665 CQ120 NDNS-Relaxed 0.7665 0.5149 0.7665 CQ121 NDNS-Exact 0.1963 0.2090 0.4916 CQ121 NDNS-Partial 0.1679 0.1776 0.3967 CQ121 NDNS-Relaxed 0.1720 0.1819 0.3927 CQ122 NDNS-Exact 0.3593 0.4885 0.8152 CQ122 NDNS-Partial 0.4087 0.5132 0.8152 CQ122 NDNS-Relaxed 0.4087 0.5132 0.8152 CQ123 NDNS-Exact 0.5215 0.5139 0.7226 CQ123 NDNS-Partial 0.4189 0.4122 0.5432 CQ123 NDNS-Relaxed 0.4149 0.4118 0.5468 CQ124 NDNS-Exact 0.1886 0.3348 0.6871 CQ124 NDNS-Partial 0.1911 0.2711 0.5566 CQ124 NDNS-Relaxed 0.1911 0.2711 0.5566 CQ125 NDNS-Exact 0.0000 0.0000 0.0449 CQ125 NDNS-Partial 0.0000 0.0000 0.0449 CQ125 NDNS-Relaxed 0.0000 0.0000 0.0449 CQ126 NDNS-Exact 0.1620 0.2838 0.2972 CQ126 NDNS-Partial 0.1401 0.2407 0.2743 CQ126 NDNS-Relaxed 0.1401 0.2407 0.2725 CQ127 NDNS-Exact 0.1834 0.1922 0.6145 CQ127 NDNS-Partial 0.1757 0.1703 0.5328 CQ127 NDNS-Relaxed 0.1718 0.1703 0.5328 CQ128 NDNS-Exact 0.2874 0.2830 0.5150 CQ128 NDNS-Partial 0.2632 0.2661 0.4694 CQ128 NDNS-Relaxed 0.2620 0.2655 0.4694 CQ129 NDNS-Exact 0.0567 0.0387 0.3125 CQ129 NDNS-Partial 0.0467 0.0318 0.2572 CQ129 NDNS-Relaxed 0.0467 0.0318 0.2572 CQ130 NDNS-Exact 0.1836 0.3557 0.6022 CQ130 NDNS-Partial 0.2493 0.3675 0.6022 CQ130 NDNS-Relaxed 0.2493 0.3675 0.6022 MEAN NDNS-Exact 0.2667 0.3142 0.4143 MEAN NDNS-Partial 0.2485 0.2858 0.3662 MEAN NDNS-Relaxed 0.2453 0.2845 0.3675 ******************************************************* Run: IBM_2 Query Metric Score Median Maximum CQ101 NDNS-Exact 0.2559 0.1503 0.2700 CQ101 NDNS-Partial 0.2813 0.1340 0.2869 CQ101 NDNS-Relaxed 0.2784 0.1456 0.2894 CQ102 NDNS-Exact 0.4176 0.4063 0.7355 CQ102 NDNS-Partial 0.3612 0.3457 0.6011 CQ102 NDNS-Relaxed 0.3660 0.3494 0.6194 CQ103 NDNS-Exact 0.1028 0.1031 0.1767 CQ103 NDNS-Partial 0.0872 0.0875 0.1499 CQ103 NDNS-Relaxed 0.0872 0.0875 0.1499 CQ104 NDNS-Exact 0.1672 0.1333 0.4982 CQ104 NDNS-Partial 0.1672 0.1333 0.4982 CQ104 NDNS-Relaxed 0.1672 0.1333 0.4982 CQ105 NDNS-Exact 0.3416 0.3401 0.4124 CQ105 NDNS-Partial 0.4059 0.3572 0.4059 CQ105 NDNS-Relaxed 0.3190 0.3254 0.3850 CQ106 NDNS-Exact 0.0000 0.1063 0.5912 CQ106 NDNS-Partial 0.0000 0.0962 0.4821 CQ106 NDNS-Relaxed 0.0000 0.0962 0.4821 CQ107 NDNS-Exact 0.3636 0.2447 0.5969 CQ107 NDNS-Partial 0.2959 0.1991 0.4857 CQ107 NDNS-Relaxed 0.2959 0.1991 0.4857 CQ108 NDNS-Exact 0.5333 0.4565 0.7288 CQ108 NDNS-Partial 0.5494 0.4417 0.6084 CQ108 NDNS-Relaxed 0.5494 0.4100 0.6084 CQ109 NDNS-Exact 0.2466 0.3804 0.7500 CQ109 NDNS-Partial 0.2466 0.3867 0.7500 CQ109 NDNS-Relaxed 0.2466 0.3867 0.7500 CQ110 NDNS-Exact 0.5454 0.5001 0.5953 CQ110 NDNS-Partial 0.3991 0.3627 0.4440 CQ110 NDNS-Relaxed 0.4007 0.3654 0.4347 CQ111 NDNS-Exact 0.3245 0.5054 0.7144 CQ111 NDNS-Partial 0.2957 0.4649 0.6510 CQ111 NDNS-Relaxed 0.2957 0.4649 0.6510 CQ112 NDNS-Exact 0.2568 0.2450 0.3337 CQ112 NDNS-Partial 0.2294 0.2175 0.3522 CQ112 NDNS-Relaxed 0.2294 0.2175 0.3522 CQ113 NDNS-Exact 0.2236 0.0793 0.2236 CQ113 NDNS-Partial 0.1879 0.0667 0.1879 CQ113 NDNS-Relaxed 0.1879 0.0667 0.1879 CQ114 NDNS-Exact 0.0578 0.3065 0.7244 CQ114 NDNS-Partial 0.0685 0.2952 0.6859 CQ114 NDNS-Relaxed 0.0685 0.2902 0.6859 CQ115 NDNS-Exact 0.6298 0.4474 0.7055 CQ115 NDNS-Partial 0.8198 0.4699 0.8956 CQ115 NDNS-Relaxed 0.8198 0.4699 0.8956 CQ116 NDNS-Exact 0.1710 0.1467 0.1711 CQ116 NDNS-Partial 0.1410 0.1210 0.1411 CQ116 NDNS-Relaxed 0.1410 0.1210 0.1411 CQ117 NDNS-Exact 0.2372 0.2244 0.5007 CQ117 NDNS-Partial 0.2372 0.2244 0.5007 CQ117 NDNS-Relaxed 0.2372 0.2244 0.5007 CQ118 NDNS-Exact 0.4836 0.4771 0.6880 CQ118 NDNS-Partial 0.3302 0.3276 0.4794 CQ118 NDNS-Relaxed 0.3128 0.3087 0.4540 CQ119 NDNS-Exact 0.0240 0.1554 0.3275 CQ119 NDNS-Partial 0.0241 0.1559 0.3287 CQ119 NDNS-Relaxed 0.0241 0.1559 0.3287 CQ120 NDNS-Exact 0.8216 0.5451 0.8264 CQ120 NDNS-Partial 0.7620 0.5149 0.7665 CQ120 NDNS-Relaxed 0.7620 0.5149 0.7665 CQ121 NDNS-Exact 0.2216 0.2090 0.4916 CQ121 NDNS-Partial 0.1874 0.1776 0.3967 CQ121 NDNS-Relaxed 0.1919 0.1819 0.3927 CQ122 NDNS-Exact 0.3701 0.4885 0.8152 CQ122 NDNS-Partial 0.4195 0.5132 0.8152 CQ122 NDNS-Relaxed 0.4195 0.5132 0.8152 CQ123 NDNS-Exact 0.5194 0.5139 0.7226 CQ123 NDNS-Partial 0.4173 0.4122 0.5432 CQ123 NDNS-Relaxed 0.4138 0.4118 0.5468 CQ124 NDNS-Exact 0.1886 0.3348 0.6871 CQ124 NDNS-Partial 0.1887 0.2711 0.5566 CQ124 NDNS-Relaxed 0.1887 0.2711 0.5566 CQ125 NDNS-Exact 0.0000 0.0000 0.0449 CQ125 NDNS-Partial 0.0000 0.0000 0.0449 CQ125 NDNS-Relaxed 0.0000 0.0000 0.0449 CQ126 NDNS-Exact 0.2347 0.2838 0.2972 CQ126 NDNS-Partial 0.2017 0.2407 0.2743 CQ126 NDNS-Relaxed 0.2017 0.2407 0.2725 CQ127 NDNS-Exact 0.1788 0.1922 0.6145 CQ127 NDNS-Partial 0.1703 0.1703 0.5328 CQ127 NDNS-Relaxed 0.1663 0.1703 0.5328 CQ128 NDNS-Exact 0.3186 0.2830 0.5150 CQ128 NDNS-Partial 0.2918 0.2661 0.4694 CQ128 NDNS-Relaxed 0.2905 0.2655 0.4694 CQ129 NDNS-Exact 0.0704 0.0387 0.3125 CQ129 NDNS-Partial 0.0579 0.0318 0.2572 CQ129 NDNS-Relaxed 0.0579 0.0318 0.2572 CQ130 NDNS-Exact 0.1563 0.3557 0.6022 CQ130 NDNS-Partial 0.2129 0.3675 0.6022 CQ130 NDNS-Relaxed 0.2129 0.3675 0.6022 MEAN NDNS-Exact 0.2821 0.3142 0.4143 MEAN NDNS-Partial 0.2679 0.2858 0.3662 MEAN NDNS-Relaxed 0.2644 0.2845 0.3675 ******************************************************* Run: IBM_3 Query Metric Score Median Maximum CQ101 NDNS-Exact 0.2435 0.1503 0.2700 CQ101 NDNS-Partial 0.2478 0.1340 0.2869 CQ101 NDNS-Relaxed 0.2572 0.1456 0.2894 CQ102 NDNS-Exact 0.3951 0.4063 0.7355 CQ102 NDNS-Partial 0.3302 0.3457 0.6011 CQ102 NDNS-Relaxed 0.3328 0.3494 0.6194 CQ103 NDNS-Exact 0.0972 0.1031 0.1767 CQ103 NDNS-Partial 0.0824 0.0875 0.1499 CQ103 NDNS-Relaxed 0.0824 0.0875 0.1499 CQ104 NDNS-Exact 0.2346 0.1333 0.4982 CQ104 NDNS-Partial 0.2346 0.1333 0.4982 CQ104 NDNS-Relaxed 0.2346 0.1333 0.4982 CQ105 NDNS-Exact 0.1804 0.3401 0.4124 CQ105 NDNS-Partial 0.2160 0.3572 0.4059 CQ105 NDNS-Relaxed 0.1684 0.3254 0.3850 CQ106 NDNS-Exact 0.0417 0.1063 0.5912 CQ106 NDNS-Partial 0.0340 0.0962 0.4821 CQ106 NDNS-Relaxed 0.0340 0.0962 0.4821 CQ107 NDNS-Exact 0.2357 0.2447 0.5969 CQ107 NDNS-Partial 0.1918 0.1991 0.4857 CQ107 NDNS-Relaxed 0.1918 0.1991 0.4857 CQ108 NDNS-Exact 0.4181 0.4565 0.7288 CQ108 NDNS-Partial 0.4702 0.4417 0.6084 CQ108 NDNS-Relaxed 0.4069 0.4100 0.6084 CQ109 NDNS-Exact 0.1927 0.3804 0.7500 CQ109 NDNS-Partial 0.2645 0.3867 0.7500 CQ109 NDNS-Relaxed 0.2645 0.3867 0.7500 CQ110 NDNS-Exact 0.4751 0.5001 0.5953 CQ110 NDNS-Partial 0.3642 0.3627 0.4440 CQ110 NDNS-Relaxed 0.3550 0.3654 0.4347 CQ111 NDNS-Exact 0.3420 0.5054 0.7144 CQ111 NDNS-Partial 0.3116 0.4649 0.6510 CQ111 NDNS-Relaxed 0.3116 0.4649 0.6510 CQ112 NDNS-Exact 0.2342 0.2450 0.3337 CQ112 NDNS-Partial 0.2095 0.2175 0.3522 CQ112 NDNS-Relaxed 0.2095 0.2175 0.3522 CQ113 NDNS-Exact 0.0953 0.0793 0.2236 CQ113 NDNS-Partial 0.0801 0.0667 0.1879 CQ113 NDNS-Relaxed 0.0801 0.0667 0.1879 CQ114 NDNS-Exact 0.2825 0.3065 0.7244 CQ114 NDNS-Partial 0.2775 0.2952 0.6859 CQ114 NDNS-Relaxed 0.2675 0.2902 0.6859 CQ115 NDNS-Exact 0.7055 0.4474 0.7055 CQ115 NDNS-Partial 0.8956 0.4699 0.8956 CQ115 NDNS-Relaxed 0.8956 0.4699 0.8956 CQ116 NDNS-Exact 0.1621 0.1467 0.1711 CQ116 NDNS-Partial 0.1337 0.1210 0.1411 CQ116 NDNS-Relaxed 0.1337 0.1210 0.1411 CQ117 NDNS-Exact 0.3707 0.2244 0.5007 CQ117 NDNS-Partial 0.3707 0.2244 0.5007 CQ117 NDNS-Relaxed 0.3707 0.2244 0.5007 CQ118 NDNS-Exact 0.4743 0.4771 0.6880 CQ118 NDNS-Partial 0.3272 0.3276 0.4794 CQ118 NDNS-Relaxed 0.3069 0.3087 0.4540 CQ119 NDNS-Exact 0.1554 0.1554 0.3275 CQ119 NDNS-Partial 0.1559 0.1559 0.3287 CQ119 NDNS-Relaxed 0.1559 0.1559 0.3287 CQ120 NDNS-Exact 0.4900 0.5451 0.8264 CQ120 NDNS-Partial 0.4732 0.5149 0.7665 CQ120 NDNS-Relaxed 0.4732 0.5149 0.7665 CQ121 NDNS-Exact 0.3001 0.2090 0.4916 CQ121 NDNS-Partial 0.2434 0.1776 0.3967 CQ121 NDNS-Relaxed 0.2493 0.1819 0.3927 CQ122 NDNS-Exact 0.4462 0.4885 0.8152 CQ122 NDNS-Partial 0.4956 0.5132 0.8152 CQ122 NDNS-Relaxed 0.4956 0.5132 0.8152 CQ123 NDNS-Exact 0.4774 0.5139 0.7226 CQ123 NDNS-Partial 0.3661 0.4122 0.5432 CQ123 NDNS-Relaxed 0.3612 0.4118 0.5468 CQ124 NDNS-Exact 0.1297 0.3348 0.6871 CQ124 NDNS-Partial 0.1401 0.2711 0.5566 CQ124 NDNS-Relaxed 0.1401 0.2711 0.5566 CQ125 NDNS-Exact 0.0449 0.0000 0.0449 CQ125 NDNS-Partial 0.0449 0.0000 0.0449 CQ125 NDNS-Relaxed 0.0449 0.0000 0.0449 CQ126 NDNS-Exact 0.2835 0.2838 0.2972 CQ126 NDNS-Partial 0.2743 0.2407 0.2743 CQ126 NDNS-Relaxed 0.2725 0.2407 0.2725 CQ127 NDNS-Exact 0.1981 0.1922 0.6145 CQ127 NDNS-Partial 0.1844 0.1703 0.5328 CQ127 NDNS-Relaxed 0.1795 0.1703 0.5328 CQ128 NDNS-Exact 0.3360 0.2830 0.5150 CQ128 NDNS-Partial 0.3140 0.2661 0.4694 CQ128 NDNS-Relaxed 0.3063 0.2655 0.4694 CQ129 NDNS-Exact 0.1563 0.0387 0.3125 CQ129 NDNS-Partial 0.1286 0.0318 0.2572 CQ129 NDNS-Relaxed 0.1286 0.0318 0.2572 CQ130 NDNS-Exact 0.1295 0.3557 0.6022 CQ130 NDNS-Partial 0.1795 0.3675 0.6022 CQ130 NDNS-Relaxed 0.1795 0.3675 0.6022 MEAN NDNS-Exact 0.2776 0.3142 0.4143 MEAN NDNS-Partial 0.2680 0.2858 0.3662 MEAN NDNS-Relaxed 0.2630 0.2845 0.3675 *******************************************************