TAC KBP 2015 Validation/Ensembling Track

Overview

The slot filler validation (SFV) task of the Validation/Ensembling track focuses on the refinement of output from English slot filling (SF) systems by either combining information from multiple slot filling systems, or applying more intensive linguistic processing to validate individual candidate slot fillers.

SFV is a diagnostic task that eliminates the need for a team to have a full slot filling system and concentrates on the refinement of existing system(s) output. The input to the SFV system is a set of submission files from several slot filling runs (with the run ID anonymized appropriately). The output of the SFV system is a binary classification (Correct/Incorrect) of each candidate slot filler in each SF run.

SFV is motivated by a use case in which the SFV system is used as a component of a full SF system. The evaluation measures the effect of using the SFV output to filter the contributing SF runs. Each contributing SF run will be filtered and rescored in the same way as for the full SF task, and the results compared against the scores for the unfiltered SF run. SFV tries to increase Precision of the contributing SF runs without significantly reducing Recall, and the objective function is to maximize the mean F-score over each of the filtered contributing runs.

Slot Filler Validation can be viewed as a Recognizing Textual Entailment (RTE) task, where the "text" consists of the provenance returned with each candidate slot filler, and the "hypothesis" is a natural language expression of the relation between the target entity and the candidate filler. The text entails the hypothesis if and only if the candidate slot filler is Correct according to the associated provenance.

Preliminary Schedule

April 15	Track registration opens
mid April	Track guidelines posted
June 30	Deadline for registration for track participation
Sept 21-Oct 12	Slot Filler Validation/Ensembling evaluation window
By mid October	Release of individual evaluated results to participants
October 20	Deadline for short system descriptions
October 20	Deadline for workshop presentation proposals
October 25	Notification of acceptance of presentation proposals
Nov 8	Deadline for system reports (workshop notebook version)
November 16-17	TAC 2015 workshop in Gaithersburg, Maryland, USA
March 5, 2016	Deadline for system reports (final proceedings version)

Track Coordinator