====================================================
TAC KBP 2017 ENGLISH SLOT FILLING EVALUATION RESULTS
====================================================


Team ID:  dws_uma
Organization:  University of Mannheim


*************************************************************

Run ID:  dws_uma_SF_ENG_1
Did the run access the live Web during the evaluation window:  No
Did the run extract relations from the Cold Start source corpus: Yes
Did the run generate meaningful confidence values: Yes

Slot Filling Evaluation (Queries involve ONLY SF slots):

Scores based on P/R/F1; only k=1 justification allowed:
Metric                   RunID            Hop Prec   Recall F1    
SF-ALL-Micro             dws_uma_SF_ENG_1 0   0.1644 0.0311 0.0523
SF-ALL-Micro             dws_uma_SF_ENG_1 1   0.1481 0.0219 0.0381
SF-ALL-Micro             dws_uma_SF_ENG_1 ALL 0.1612 0.0289 0.0490
SF-ALL-Macro             dws_uma_SF_ENG_1 0   0.0430 0.0349 0.0333
SF-ALL-Macro             dws_uma_SF_ENG_1 1   0.0305 0.0244 0.0254
SF-ALL-Macro             dws_uma_SF_ENG_1 ALL 0.0381 0.0307 0.0302
LDC-MAX-ALL-Micro        dws_uma_SF_ENG_1 0   0.1368 0.0305 0.0499
LDC-MAX-ALL-Micro        dws_uma_SF_ENG_1 1   0.1154 0.0190 0.0326
LDC-MAX-ALL-Micro        dws_uma_SF_ENG_1 ALL 0.1329 0.0279 0.0461
LDC-MAX-ALL-Macro        dws_uma_SF_ENG_1 0   0.0434 0.0353 0.0327
LDC-MAX-ALL-Macro        dws_uma_SF_ENG_1 1   0.0283 0.0208 0.0220
LDC-MAX-ALL-Macro        dws_uma_SF_ENG_1 ALL 0.0375 0.0297 0.0285
LDC-MEAN-ALL-Macro       dws_uma_SF_ENG_1 0   0.0424 0.0351 0.0324
LDC-MEAN-ALL-Macro       dws_uma_SF_ENG_1 1   0.0283 0.0208 0.0220
LDC-MEAN-ALL-Macro       dws_uma_SF_ENG_1 ALL 0.0369 0.0296 0.0284

*ALL-Macro Prec, Recall and F1 refer to mean-precision, mean-recall and mean-F1.

Scores based on confidence values and Average Precision (AP); only k=1 justification allowed:
Metric                RunID            Hop AP    
SF-ALL-Macro          dws_uma_SF_ENG_1 0   0.0313
SF-ALL-Macro          dws_uma_SF_ENG_1 1   0.0185
SF-ALL-Macro          dws_uma_SF_ENG_1 ALL 0.0290
LDC-MEAN-ALL-Macro    dws_uma_SF_ENG_1 0   0.0309
LDC-MEAN-ALL-Macro    dws_uma_SF_ENG_1 1   0.0154
LDC-MEAN-ALL-Macro    dws_uma_SF_ENG_1 ALL 0.0289 (PRIMARY METRIC)

*ALL-Macro AP refer to mean of corresponding AP values.