================================================================ TAC 2020 EPIDEMIC QUESTION ANSWERING EVALUATION RESULTS (TASK A) ================================================================ Team ID: IBM Organization(s): IBM Research AI Run: IBM_1 Query Metric Score Median Maximum EQ101 NDNS-Exact 0.4571 0.4974 0.6811 EQ101 NDNS-Partial 0.4836 0.4186 0.5638 EQ101 NDNS-Relaxed 0.4836 0.4186 0.5638 EQ102 NDNS-Exact 0.3945 0.3410 0.5418 EQ102 NDNS-Partial 0.3604 0.3115 0.4950 EQ102 NDNS-Relaxed 0.3604 0.3115 0.4950 EQ103 NDNS-Exact 0.1423 0.2500 0.3667 EQ103 NDNS-Partial 0.1231 0.2078 0.3048 EQ103 NDNS-Relaxed 0.1231 0.2078 0.3048 EQ104 NDNS-Exact 0.6266 0.4219 0.6273 EQ104 NDNS-Partial 0.6397 0.4080 0.6397 EQ104 NDNS-Relaxed 0.5977 0.4080 0.5977 EQ105 NDNS-Exact 0.2932 0.2766 0.7001 EQ105 NDNS-Partial 0.3075 0.2784 0.6617 EQ105 NDNS-Relaxed 0.3075 0.2784 0.6617 EQ106 NDNS-Exact 0.5595 0.4169 0.6451 EQ106 NDNS-Partial 0.5105 0.3377 0.5332 EQ106 NDNS-Relaxed 0.5105 0.3377 0.5225 EQ107 NDNS-Exact 0.1281 0.1240 0.1839 EQ107 NDNS-Partial 0.1181 0.1144 0.1695 EQ107 NDNS-Relaxed 0.1181 0.1144 0.1695 EQ108 NDNS-Exact 0.3226 0.3949 0.6287 EQ108 NDNS-Partial 0.2762 0.3380 0.5382 EQ108 NDNS-Relaxed 0.2762 0.3380 0.5382 EQ109 NDNS-Exact 0.6000 0.6309 1.0000 EQ109 NDNS-Partial 0.7500 0.7500 1.0000 EQ109 NDNS-Relaxed 0.7500 0.7500 1.0000 EQ110 NDNS-Exact 0.3048 0.3887 0.5470 EQ110 NDNS-Partial 0.2770 0.3022 0.4270 EQ110 NDNS-Relaxed 0.2668 0.2992 0.4210 EQ111 NDNS-Exact 0.4418 0.4480 0.5874 EQ111 NDNS-Partial 0.3653 0.3620 0.5094 EQ111 NDNS-Relaxed 0.3691 0.3688 0.5236 EQ112 NDNS-Exact 0.4661 0.5012 0.5953 EQ112 NDNS-Partial 0.4034 0.4338 0.5153 EQ112 NDNS-Relaxed 0.4034 0.4338 0.5153 EQ113 NDNS-Exact 0.1866 0.3610 0.5062 EQ113 NDNS-Partial 0.2166 0.3511 0.4923 EQ113 NDNS-Relaxed 0.2166 0.3511 0.4923 EQ114 NDNS-Exact 0.1558 0.3145 0.6719 EQ114 NDNS-Partial 0.1505 0.3039 0.6492 EQ114 NDNS-Relaxed 0.1505 0.3039 0.6492 EQ115 NDNS-Exact 0.0900 0.1533 0.2688 EQ115 NDNS-Partial 0.0945 0.1461 0.2561 EQ115 NDNS-Relaxed 0.0916 0.1466 0.2570 EQ116 NDNS-Exact 0.2252 0.2327 0.3331 EQ116 NDNS-Partial 0.1992 0.2058 0.2946 EQ116 NDNS-Relaxed 0.1992 0.2058 0.2946 EQ117 NDNS-Exact 0.4462 0.3511 0.5322 EQ117 NDNS-Partial 0.4695 0.3715 0.6085 EQ117 NDNS-Relaxed 0.4454 0.3715 0.6085 EQ118 NDNS-Exact 0.2345 0.5646 0.6908 EQ118 NDNS-Partial 0.2145 0.4360 0.5335 EQ118 NDNS-Relaxed 0.1926 0.4360 0.5335 EQ119 NDNS-Exact 0.3403 0.2868 0.4027 EQ119 NDNS-Partial 0.2592 0.2263 0.3067 EQ119 NDNS-Relaxed 0.2592 0.2244 0.3067 EQ120 NDNS-Exact 0.4729 0.5027 0.7176 EQ120 NDNS-Partial 0.4459 0.4740 0.6767 EQ120 NDNS-Relaxed 0.4513 0.4798 0.6849 EQ121 NDNS-Exact 0.2991 0.3903 0.4689 EQ121 NDNS-Partial 0.3214 0.3819 0.4589 EQ121 NDNS-Relaxed 0.3214 0.3819 0.4589 EQ122 NDNS-Exact 0.3352 0.2465 0.3862 EQ122 NDNS-Partial 0.2963 0.2078 0.3409 EQ122 NDNS-Relaxed 0.2815 0.2092 0.3244 EQ123 NDNS-Exact 0.4411 0.5079 0.6317 EQ123 NDNS-Partial 0.3416 0.3934 0.4892 EQ123 NDNS-Relaxed 0.3424 0.3943 0.4904 EQ124 NDNS-Exact 0.4640 0.4694 0.6163 EQ124 NDNS-Partial 0.3989 0.4035 0.5298 EQ124 NDNS-Relaxed 0.3989 0.4035 0.5298 EQ125 NDNS-Exact 0.5293 0.3250 0.8570 EQ125 NDNS-Partial 0.4448 0.2731 0.7201 EQ125 NDNS-Relaxed 0.4448 0.2731 0.7201 EQ126 NDNS-Exact 0.1309 0.2966 0.6099 EQ126 NDNS-Partial 0.1057 0.2396 0.4927 EQ126 NDNS-Relaxed 0.1063 0.2408 0.4952 EQ127 NDNS-Exact 0.3250 0.4034 0.5870 EQ127 NDNS-Partial 0.2901 0.3684 0.5060 EQ127 NDNS-Relaxed 0.2919 0.3737 0.5160 EQ128 NDNS-Exact 0.4274 0.3416 0.4318 EQ128 NDNS-Partial 0.3329 0.2551 0.3393 EQ128 NDNS-Relaxed 0.3283 0.2555 0.3331 EQ129 NDNS-Exact 0.3100 0.1448 0.3360 EQ129 NDNS-Partial 0.2638 0.1205 0.2868 EQ129 NDNS-Relaxed 0.2638 0.1205 0.2868 EQ130 NDNS-Exact 0.4360 0.4404 0.6946 EQ130 NDNS-Partial 0.4605 0.4535 0.6946 EQ130 NDNS-Relaxed 0.4605 0.4535 0.6946 MEAN NDNS-Exact 0.3529 0.3802 0.4207 MEAN NDNS-Partial 0.3307 0.3377 0.3699 MEAN NDNS-Relaxed 0.3271 0.3387 0.3709 ******************************************************* Run: IBM_2 Query Metric Score Median Maximum EQ101 NDNS-Exact 0.5286 0.4974 0.6811 EQ101 NDNS-Partial 0.5036 0.4186 0.5638 EQ101 NDNS-Relaxed 0.5036 0.4186 0.5638 EQ102 NDNS-Exact 0.5348 0.3410 0.5418 EQ102 NDNS-Partial 0.4886 0.3115 0.4950 EQ102 NDNS-Relaxed 0.4886 0.3115 0.4950 EQ103 NDNS-Exact 0.1399 0.2500 0.3667 EQ103 NDNS-Partial 0.1213 0.2078 0.3048 EQ103 NDNS-Relaxed 0.1213 0.2078 0.3048 EQ104 NDNS-Exact 0.4189 0.4219 0.6273 EQ104 NDNS-Partial 0.4586 0.4080 0.6397 EQ104 NDNS-Relaxed 0.4586 0.4080 0.5977 EQ105 NDNS-Exact 0.2785 0.2766 0.7001 EQ105 NDNS-Partial 0.2873 0.2784 0.6617 EQ105 NDNS-Relaxed 0.2873 0.2784 0.6617 EQ106 NDNS-Exact 0.6451 0.4169 0.6451 EQ106 NDNS-Partial 0.5332 0.3377 0.5332 EQ106 NDNS-Relaxed 0.5225 0.3377 0.5225 EQ107 NDNS-Exact 0.1717 0.1240 0.1839 EQ107 NDNS-Partial 0.1583 0.1144 0.1695 EQ107 NDNS-Relaxed 0.1583 0.1144 0.1695 EQ108 NDNS-Exact 0.4114 0.3949 0.6287 EQ108 NDNS-Partial 0.3522 0.3380 0.5382 EQ108 NDNS-Relaxed 0.3522 0.3380 0.5382 EQ109 NDNS-Exact 0.6000 0.6309 1.0000 EQ109 NDNS-Partial 0.7500 0.7500 1.0000 EQ109 NDNS-Relaxed 0.7500 0.7500 1.0000 EQ110 NDNS-Exact 0.3258 0.3887 0.5470 EQ110 NDNS-Partial 0.2931 0.3022 0.4270 EQ110 NDNS-Relaxed 0.2828 0.2992 0.4210 EQ111 NDNS-Exact 0.4542 0.4480 0.5874 EQ111 NDNS-Partial 0.3914 0.3620 0.5094 EQ111 NDNS-Relaxed 0.4004 0.3688 0.5236 EQ112 NDNS-Exact 0.4799 0.5012 0.5953 EQ112 NDNS-Partial 0.4154 0.4338 0.5153 EQ112 NDNS-Relaxed 0.4154 0.4338 0.5153 EQ113 NDNS-Exact 0.2798 0.3610 0.5062 EQ113 NDNS-Partial 0.3214 0.3511 0.4923 EQ113 NDNS-Relaxed 0.3214 0.3511 0.4923 EQ114 NDNS-Exact 0.2362 0.3145 0.6719 EQ114 NDNS-Partial 0.2282 0.3039 0.6492 EQ114 NDNS-Relaxed 0.2282 0.3039 0.6492 EQ115 NDNS-Exact 0.1984 0.1533 0.2688 EQ115 NDNS-Partial 0.1975 0.1461 0.2561 EQ115 NDNS-Relaxed 0.1949 0.1466 0.2570 EQ116 NDNS-Exact 0.2288 0.2327 0.3331 EQ116 NDNS-Partial 0.2024 0.2058 0.2946 EQ116 NDNS-Relaxed 0.2024 0.2058 0.2946 EQ117 NDNS-Exact 0.5255 0.3511 0.5322 EQ117 NDNS-Partial 0.6085 0.3715 0.6085 EQ117 NDNS-Relaxed 0.6085 0.3715 0.6085 EQ118 NDNS-Exact 0.2231 0.5646 0.6908 EQ118 NDNS-Partial 0.2185 0.4360 0.5335 EQ118 NDNS-Relaxed 0.2037 0.4360 0.5335 EQ119 NDNS-Exact 0.2760 0.2868 0.4027 EQ119 NDNS-Partial 0.2102 0.2263 0.3067 EQ119 NDNS-Relaxed 0.2102 0.2244 0.3067 EQ120 NDNS-Exact 0.5168 0.5027 0.7176 EQ120 NDNS-Partial 0.4873 0.4740 0.6767 EQ120 NDNS-Relaxed 0.4932 0.4798 0.6849 EQ121 NDNS-Exact 0.2943 0.3903 0.4689 EQ121 NDNS-Partial 0.3167 0.3819 0.4589 EQ121 NDNS-Relaxed 0.3167 0.3819 0.4589 EQ122 NDNS-Exact 0.3862 0.2465 0.3862 EQ122 NDNS-Partial 0.3409 0.2078 0.3409 EQ122 NDNS-Relaxed 0.3244 0.2092 0.3244 EQ123 NDNS-Exact 0.3891 0.5079 0.6317 EQ123 NDNS-Partial 0.3014 0.3934 0.4892 EQ123 NDNS-Relaxed 0.3021 0.3943 0.4904 EQ124 NDNS-Exact 0.4640 0.4694 0.6163 EQ124 NDNS-Partial 0.3989 0.4035 0.5298 EQ124 NDNS-Relaxed 0.3989 0.4035 0.5298 EQ125 NDNS-Exact 0.3454 0.3250 0.8570 EQ125 NDNS-Partial 0.2903 0.2731 0.7201 EQ125 NDNS-Relaxed 0.2903 0.2731 0.7201 EQ126 NDNS-Exact 0.1331 0.2966 0.6099 EQ126 NDNS-Partial 0.1075 0.2396 0.4927 EQ126 NDNS-Relaxed 0.1081 0.2408 0.4952 EQ127 NDNS-Exact 0.3448 0.4034 0.5870 EQ127 NDNS-Partial 0.3102 0.3684 0.5060 EQ127 NDNS-Relaxed 0.3119 0.3737 0.5160 EQ128 NDNS-Exact 0.4266 0.3416 0.4318 EQ128 NDNS-Partial 0.3318 0.2551 0.3393 EQ128 NDNS-Relaxed 0.3277 0.2555 0.3331 EQ129 NDNS-Exact 0.3360 0.1448 0.3360 EQ129 NDNS-Partial 0.2868 0.1205 0.2868 EQ129 NDNS-Relaxed 0.2868 0.1205 0.2868 EQ130 NDNS-Exact 0.4240 0.4404 0.6946 EQ130 NDNS-Partial 0.4465 0.4535 0.6946 EQ130 NDNS-Relaxed 0.4465 0.4535 0.6946 MEAN NDNS-Exact 0.3672 0.3802 0.4207 MEAN NDNS-Partial 0.3453 0.3377 0.3699 MEAN NDNS-Relaxed 0.3439 0.3387 0.3709 ******************************************************* Run: IBM_3 Query Metric Score Median Maximum EQ101 NDNS-Exact 0.3709 0.4974 0.6811 EQ101 NDNS-Partial 0.3915 0.4186 0.5638 EQ101 NDNS-Relaxed 0.3915 0.4186 0.5638 EQ102 NDNS-Exact 0.5418 0.3410 0.5418 EQ102 NDNS-Partial 0.4950 0.3115 0.4950 EQ102 NDNS-Relaxed 0.4950 0.3115 0.4950 EQ103 NDNS-Exact 0.2212 0.2500 0.3667 EQ103 NDNS-Partial 0.2005 0.2078 0.3048 EQ103 NDNS-Relaxed 0.2005 0.2078 0.3048 EQ104 NDNS-Exact 0.5557 0.4219 0.6273 EQ104 NDNS-Partial 0.5380 0.4080 0.6397 EQ104 NDNS-Relaxed 0.5180 0.4080 0.5977 EQ105 NDNS-Exact 0.2597 0.2766 0.7001 EQ105 NDNS-Partial 0.2695 0.2784 0.6617 EQ105 NDNS-Relaxed 0.2695 0.2784 0.6617 EQ106 NDNS-Exact 0.5232 0.4169 0.6451 EQ106 NDNS-Partial 0.4270 0.3377 0.5332 EQ106 NDNS-Relaxed 0.4238 0.3377 0.5225 EQ107 NDNS-Exact 0.0452 0.1240 0.1839 EQ107 NDNS-Partial 0.0416 0.1144 0.1695 EQ107 NDNS-Relaxed 0.0416 0.1144 0.1695 EQ108 NDNS-Exact 0.3022 0.3949 0.6287 EQ108 NDNS-Partial 0.2723 0.3380 0.5382 EQ108 NDNS-Relaxed 0.2587 0.3380 0.5382 EQ109 NDNS-Exact 0.6000 0.6309 1.0000 EQ109 NDNS-Partial 0.7500 0.7500 1.0000 EQ109 NDNS-Relaxed 0.7500 0.7500 1.0000 EQ110 NDNS-Exact 0.3222 0.3887 0.5470 EQ110 NDNS-Partial 0.2948 0.3022 0.4270 EQ110 NDNS-Relaxed 0.2895 0.2992 0.4210 EQ111 NDNS-Exact 0.5874 0.4480 0.5874 EQ111 NDNS-Partial 0.5094 0.3620 0.5094 EQ111 NDNS-Relaxed 0.5236 0.3688 0.5236 EQ112 NDNS-Exact 0.5225 0.5012 0.5953 EQ112 NDNS-Partial 0.4523 0.4338 0.5153 EQ112 NDNS-Relaxed 0.4523 0.4338 0.5153 EQ113 NDNS-Exact 0.2798 0.3610 0.5062 EQ113 NDNS-Partial 0.3214 0.3511 0.4923 EQ113 NDNS-Relaxed 0.3214 0.3511 0.4923 EQ114 NDNS-Exact 0.4637 0.3145 0.6719 EQ114 NDNS-Partial 0.4480 0.3039 0.6492 EQ114 NDNS-Relaxed 0.4480 0.3039 0.6492 EQ115 NDNS-Exact 0.2328 0.1533 0.2688 EQ115 NDNS-Partial 0.2289 0.1461 0.2561 EQ115 NDNS-Relaxed 0.2255 0.1466 0.2570 EQ116 NDNS-Exact 0.2046 0.2327 0.3331 EQ116 NDNS-Partial 0.1810 0.2058 0.2946 EQ116 NDNS-Relaxed 0.1810 0.2058 0.2946 EQ117 NDNS-Exact 0.3509 0.3511 0.5322 EQ117 NDNS-Partial 0.3923 0.3715 0.6085 EQ117 NDNS-Relaxed 0.3923 0.3715 0.6085 EQ118 NDNS-Exact 0.4160 0.5646 0.6908 EQ118 NDNS-Partial 0.3813 0.4360 0.5335 EQ118 NDNS-Relaxed 0.3655 0.4360 0.5335 EQ119 NDNS-Exact 0.2417 0.2868 0.4027 EQ119 NDNS-Partial 0.2070 0.2263 0.3067 EQ119 NDNS-Relaxed 0.2070 0.2244 0.3067 EQ120 NDNS-Exact 0.5438 0.5027 0.7176 EQ120 NDNS-Partial 0.5127 0.4740 0.6767 EQ120 NDNS-Relaxed 0.5189 0.4798 0.6849 EQ121 NDNS-Exact 0.2889 0.3903 0.4689 EQ121 NDNS-Partial 0.3114 0.3819 0.4589 EQ121 NDNS-Relaxed 0.3114 0.3819 0.4589 EQ122 NDNS-Exact 0.2195 0.2465 0.3862 EQ122 NDNS-Partial 0.2023 0.2078 0.3409 EQ122 NDNS-Relaxed 0.1951 0.2092 0.3244 EQ123 NDNS-Exact 0.3091 0.5079 0.6317 EQ123 NDNS-Partial 0.2496 0.3934 0.4892 EQ123 NDNS-Relaxed 0.2400 0.3943 0.4904 EQ124 NDNS-Exact 0.4640 0.4694 0.6163 EQ124 NDNS-Partial 0.3989 0.4035 0.5298 EQ124 NDNS-Relaxed 0.3989 0.4035 0.5298 EQ125 NDNS-Exact 0.2895 0.3250 0.8570 EQ125 NDNS-Partial 0.2433 0.2731 0.7201 EQ125 NDNS-Relaxed 0.2433 0.2731 0.7201 EQ126 NDNS-Exact 0.1237 0.2966 0.6099 EQ126 NDNS-Partial 0.0999 0.2396 0.4927 EQ126 NDNS-Relaxed 0.1005 0.2408 0.4952 EQ127 NDNS-Exact 0.3969 0.4034 0.5870 EQ127 NDNS-Partial 0.3677 0.3684 0.5060 EQ127 NDNS-Relaxed 0.3710 0.3737 0.5160 EQ128 NDNS-Exact 0.4318 0.3416 0.4318 EQ128 NDNS-Partial 0.3393 0.2551 0.3393 EQ128 NDNS-Relaxed 0.3331 0.2555 0.3331 EQ129 NDNS-Exact 0.1611 0.1448 0.3360 EQ129 NDNS-Partial 0.1553 0.1205 0.2868 EQ129 NDNS-Relaxed 0.1494 0.1205 0.2868 EQ130 NDNS-Exact 0.3524 0.4404 0.6946 EQ130 NDNS-Partial 0.4041 0.4535 0.6946 EQ130 NDNS-Relaxed 0.4041 0.4535 0.6946 MEAN NDNS-Exact 0.3541 0.3802 0.4207 MEAN NDNS-Partial 0.3362 0.3377 0.3699 MEAN NDNS-Relaxed 0.3340 0.3387 0.3709 *******************************************************