Go to the NIST home page Go to the TAC Home Page TAC Banner

Return to TAC Homepage

TAC 2015 Tracks
    Cold Start KBP
    Tri-Lingual EDL
    Event
    Validation/Ensembling
        Guidelines
        Tools
        Schedule
Data
Schedule
Organizers
Track Registration
Reporting Guidelines
TAC 2015 Workshop




TAC KBP 2015 Validation/Ensembling Track

Overview

The slot filler validation (SFV) task of the Validation/Ensembling track focuses on the refinement of output from English slot filling (SF) systems by either combining information from multiple slot filling systems, or applying more intensive linguistic processing to validate individual candidate slot fillers.

SFV is a diagnostic task that eliminates the need for a team to have a full slot filling system and concentrates on the refinement of existing system(s) output. The input to the SFV system is a set of submission files from several slot filling runs (with the run ID anonymized appropriately). The output of the SFV system is a binary classification (Correct/Incorrect) of each candidate slot filler in each SF run.

SFV is motivated by a use case in which the SFV system is used as a component of a full SF system. The evaluation measures the effect of using the SFV output to filter the contributing SF runs. Each contributing SF run will be filtered and rescored in the same way as for the full SF task, and the results compared against the scores for the unfiltered SF run. SFV tries to increase Precision of the contributing SF runs without significantly reducing Recall, and the objective function is to maximize the mean F-score over each of the filtered contributing runs.

Slot Filler Validation can be viewed as a Recognizing Textual Entailment (RTE) task, where the "text" consists of the provenance returned with each candidate slot filler, and the "hypothesis" is a natural language expression of the relation between the target entity and the candidate filler. The text entails the hypothesis if and only if the candidate slot filler is Correct according to the associated provenance.

Preliminary Schedule

    April 15Track registration opens
    mid AprilTrack guidelines posted
    June 30Deadline for registration for track participation
    Sept 21-Oct 12Slot Filler Validation/Ensembling evaluation window
    By mid OctoberRelease of individual evaluated results to participants
    October 20Deadline for short system descriptions
    October 20Deadline for workshop presentation proposals
    October 25Notification of acceptance of presentation proposals
    Nov 8Deadline for system reports (workshop notebook version)
    November 16-17TAC 2015 workshop in Gaithersburg, Maryland, USA
    March 5, 2016Deadline for system reports (final proceedings version)

Track Coordinator

Hoa Dang (NIST, hoa.dang@nist.gov)


NIST is an agency of the
U.S. Department of Commerce

privacy policy / security notice / accessibility statement
disclaimer
FOIA

Last updated: Tuesday, 28-Mar-2017 09:22:18 MDT
Comments to: tac-web@nist.gov