Go to the NIST home page Go to the TAC Home Page TAC Banner

Return to TAC Homepage

TAC 2016 Tracks
    Cold Start SF/KB
    EDL
    Ensembling
        Guidelines
        Tools
        Schedule
    Event
    Belief/Sentiment
Data
Schedule
Organizers
Track Registration
Reporting Guidelines
TAC 2016 Workshop




TAC KBP 2016 Validation/Ensembling Track

Overview

The slot filler validation (SFV) task of the Validation/Ensembling track focuses on the refinement of output from English slot filling (SF) systems by either combining information from multiple slot filling systems, or applying more intensive linguistic processing to validate individual candidate slot fillers.

SFV is a diagnostic task that eliminates the need for a team to have a full slot filling system and concentrates on the refinement of existing system(s) output. The input to the SFV system is a set of submission files from Cold Start KB and slot filling runs. The output of the SFV system is a binary classification (Correct/Incorrect) of each candidate slot filler in each SF run.

For the SFV Ensemble task, the evaluation considers the SFV run as a single ensemble Cold Start SF run that includes only selected slot fillers from the input Cold Start SF runs; the objective function for the SFV Ensemble task is to maximize the F-score of the ensemble Cold Start run.

Preliminary Schedule

    Preliminary TAC KBP 2016 Schedule
    February 29Track registration opens
    July 15Deadline for registration for track participation
    August - OctoberTrack evaluation windows (varies by track)
    August 1-14EDL First Evaluation Window (EDL1)
    August 15-29Cold Start KB/SF Evaluation Window
    August 22-Sept 7Event Argument Extraction and Linking Evaluation Window
    Sept 1Release EDL1 scores to individual participants
    Sept 12-19EDL Evaluation Window 2 (EDL2)
    Sept 20-Oct 3Event Nugget Detection and Coreference Evaluation Window
    Sept 26-Oct 3Slot Filler Validation/Ensembling Evaluation Window
    Oct 10-17Belief and Sentiment Evaluation Window
    By mid OctoberRelease of individual evaluated results to participants (most tracks)
    October 10Deadline for short system descriptions
    October 18Deadline for workshop presentation proposals
    October 20Notification of acceptance of presentation proposals
    Nov 1Deadline for system reports (workshop notebook version)
    November 14-15TAC 2016 workshop in Gaithersburg, Maryland, USA
    February 2017Deadline for system reports (final proceedings version)

Track Coordinator

Hoa Dang (NIST, hoa.dang@nist.gov)


NIST is an agency of the
U.S. Department of Commerce

privacy policy / security notice / accessibility statement
disclaimer
FOIA

Last updated: Tuesday, 28-Mar-2017 09:22:20 MDT
Comments to: tac-web@nist.gov