Go to the NIST home page Go to the TAC Home Page TAC Banner

Return to TAC Homepage

TAC 2016 Tracks
    Cold Start SF/KB
        Guidelines
        Tools
        Schedule
    EDL
    Ensembling
    Event
    Belief/Sentiment
Data
Schedule
Organizers
Track Registration
Reporting Guidelines
TAC 2016 Workshop




TAC 2016 Cold Start KBP Track

Overview

The Cold Start KBP track builds a knowledge base from scratch using a given document collection and a predefined schema for the entities and relations that will compose the KB. The current Cold Start schema consists of five entity types (PER, ORG, GPE, LOC, FAC) and binary relation types corresponding to KBP slots (and their inverse slots).

In addition to end-to-end KB construction, KBP 2016 offers two diagnostic tasks: Entity Discovery and Linking (EDL) and Cold Start Slot Filling (SF). The EDL task is to extract names and nominal mentions of specific individual PER, ORG, GPE, LOC, and FAC entities mentioned in the Cold Start document collection, and to link each mention to its KB node (either a node in the TAC reference KB, or a newly created NIL node if it doesn't have a corresponding KB entry). The Cold Start slot filling task is to search the same document collection to fill in values for specific slots for specific entities.

In 2016, EDL and Cold Start SF/KB systems will operate on the same source corpus of approximately 90K English, Chinese, and Spanish documents consisting of newswire and discussion forum posts. In order to enable slot filling teams to also participate in the full KB construction task, two EDL evaluation windows will be offered and staged so that teams attempting to construct a KB will be given the output of EDL systems participating in the first EDL evaluation window.

The Cold Start KBP track evaluates systems along two dimensions:

  1. Slot Filling: Cold Start KB Construction systems and Cold Start SF systems are evaluated on the slot filling dimension using the same set of slot filling queries.
  2. Entity Discovery and Linking: Cold Start KB Construction systems and EDL systems are evaluated on the entity discovery and linking dimension using the entity mentions and links in the same set of "core" documents.

EDL as a standalone task is organized under the Entity Discovery and Linking Track, and is described fully on the EDL 2016 home page. Below, we focus on the relationship between EDL and Cold Start SF/KB Construction.

Tasks

Given a collection of documents, the Cold Start KB Construction system must find all names or nominal mentions of specific, individual PER, ORG, GPE, LOC, and FAC entities in the collection, create a KB node for each entity, and link each name/nominal mention to the correct entity node in the KB. All Cold Start relations that are attested in the document collection must be included in the KB, along with a confidence value and provenance consisting of document(s) and offsets for the subject, predicate, and object of the relation. For relations whose objects are themselves Cold Start PER, ORG, or GPE entities (such as per:siblings or org:subsidiaries), systems must link to the KB node representing the correct entity; relations whose objects are strings (such as per:title or org:website) will use strings to represent the object. If a KB includes multiple instances of the same subject-predicate-object triple (but with different provenance information), the triple with the highest confidence value will be evaluated, and additional triples with lower confidence value may be evaluated as resources permit.

Systems partipating in the Cold Start KB Construction task are evaluated on two dimensions: slot-filling, and entity discovery and linking.

  • Entity Discovery and Linking Evaluation: approximately 500 "core" documents from the 90K input documents will be selected, and their entity mentions and entity nodes will be evaluated using the clustering metric of the KBP 2016 Entity Discovery and Linking Track.
  • Slot Filling Evaluation: An evaluation query contains a named entity mention in the document collection (an "entry point"), and a sequence of one or more relation types. The entry point selects a single corresponding entity node in the KB, and the sequence of relation types is followed to arrive at a set of terminal slot fillers at the end of the sequence. The terminal slot fillers are then assessed and scored as in the traditional English slot filling task. For example, a typical query may ask "What are the ages of the siblings of the Bart Simpson mentioned in Document 42?" or "What schools are attended by the children of the Marge Simpson mentioned in Document 9?" Such "two-hop" queries will verify that the knowledge base is well-formed in a way that goes beyond basic entity linking and slot filling. Each evaluation query may have multiple entry points (i.e., multiple mentions of the same entity), in order to mitigate cascaded errors caused by submitted KBs that are not able to link every name mention to a KB entity node.

The Cold Start Slot Filling (SF) task removes the requirement that an entire text collection must be processed. Instead, Cold Start SF participants will receive the evaluation queries, and need only produce those entities and relations that would be found by the queries. A TAC slot filling system can easily be applied to this task by running initially from each evaluation query entry point, then recursively applying the system to the identified slot fillers.

The 2016 Cold Start SF/KB Construction tasks are identical to the 2015 tasks, with the following changes:

  1. In addition to English, the source corpus will include Chinese and Spanish documents. For the cross-lingual Cold Start SF/KB Construction tasks, systems may return slot fillers and provenance from any combination of English, Chinese, and Spanish documents. Additionally, 3 diagnostic monolingual versions of these tasks will be offered (one for each language).
  2. In addition to PER, ORG, and GPE entity types, KB Construction systems must return mentions of LOC and FAC entities (although the slot inventory will not be modified to include LOC and FAC entities).
  3. In addition to name mentions, KB Construction systems must extract and link all nominal mentions of specific individual PER, ORG, GPE, LOC, and FAC entities.
  4. Cold Start SF/KB Construction systems may return a nominal mention as a filler if no name mention is available in the source corpus.

Preliminary Schedule

    Preliminary TAC KBP 2016 Schedule
    February 29Track registration opens
    July 15Deadline for registration for track participation
    August - OctoberTrack evaluation windows (varies by track)
    August 1-14EDL First Evaluation Window (EDL1)
    August 15-29Cold Start KB/SF Evaluation Window
    August 22-Sept 7Event Argument Extraction and Linking Evaluation Window
    Sept 1Release EDL1 scores to individual participants
    Sept 12-19EDL Evaluation Window 2 (EDL2)
    Sept 20-Oct 3Event Nugget Detection and Coreference Evaluation Window
    Sept 26-Oct 3Slot Filler Validation/Ensembling Evaluation Window
    Oct 10-17Belief and Sentiment Evaluation Window
    By mid OctoberRelease of individual evaluated results to participants (most tracks)
    October 10Deadline for short system descriptions
    October 18Deadline for workshop presentation proposals
    October 20Notification of acceptance of presentation proposals
    Nov 1Deadline for system reports (workshop notebook version)
    November 14-15TAC 2016 workshop in Gaithersburg, Maryland, USA
    February 2017Deadline for system reports (final proceedings version)

Track Coordinators

Hoa Dang (National Institute of Standards and Technology, hoa.dang@nist.gov)
Shahzad Rajput (National Institute of Standards and Technology, shahzad.rajput@nist.gov)


NIST is an agency of the
U.S. Department of Commerce

privacy policy / security notice / accessibility statement
disclaimer
FOIA

Last updated: Friday, 17-Feb-2017 17:25:47 EST
Comments to: tac-web@nist.gov