Go to the NIST home page Go to the TAC Home Page TAC Banner

Return to TAC Homepage

TAC 2016 Tracks
    Cold Start SF/KB
    EDL
    Ensembling
    Event
    Belief/Sentiment
Data
Schedule
Organizers
Track Registration
Reporting Guidelines
TAC 2016 Workshop




TAC Knowledge Base Population (KBP) 2016

Evaluation: February-November, 2016
Workshop: November 14-15, 2016

Conducted by:
U.S. National Institute of Standards and Technology (NIST)

With support from:
U.S. Department of Defense

Overview

The Text Analysis Conference (TAC) is a series of evaluation workshops organized to encourage research in Natural Language Processing and related applications, by providing a large test collection, common evaluation procedures, and a forum for organizations to share their results. TAC comprises sets of tasks known as "tracks," each of which focuses on a particular subproblem of NLP. TAC tracks focus on end-user tasks, but also include component evaluations situated within the context of end-user tasks.

The goal of TAC Knowledge Base Population (KBP) is to develop and evaluate technologies for populating knowledge bases (KBs) from unstructured text.

The end-to-end Cold Start KBP task is to build a KB from scratch, using a predefined KB schema and a collection of unstructured text. The current KB schema consists of entities that can be a specific individual person (PER), organization (ORG), geopolitical entity (GPE), location (LOC), or facility (FAC); and predefined attributes (a.k.a "slots") for those entities. The submitted Cold Start KBs are evaluated by applying a set of slot filling evaluation queries to each KB and assessing the correctness of the relations found.

In addition to end-to-end Cold Start KB construction, two diagnostic tasks and evaluations are offered:

  1. Entity Discovery and Linking (EDL): The EDL task is to extract name and nominal mentions of specific individual PER, ORG, GPE, LOC, and FAC entities mentioned in the Cold Start document collection, and to link each mention to its KB node (either a node in the TAC reference KB, or a newly created NIL node if it doesn't have a corresponding KB entry).
  2. Slot Filling (SF): The slot filling task is to search the Cold Start document collection to fill in values for specific slots for specific entities.

The slot filler validation task in the validation/ensembling track focuses on the refinement of output from slot filling systems by either combining information from multiple slot filling systems, or applying more intensive linguistic processing to validate candidate slot fillers.

The event track in KBP 2016 aims to extract information about events from unstructured text, such that the information would be suitable as input into a structured KB. The track includes Event Nugget (EN) tasks to detect and link events, and Event Argument (EA) tasks to extract event arguments and link arguments that belong to the same event.

Finally, the Belief and Sentiment track detects belief and sentiment of an entity toward another entity, relation, or event.

In 2016, all tasks will be multi-lingual (English, Chinese, and Spanish); Cold Start EDL, SF and KB construction tasks will also be cross-lingual. Participants are encouraged to participate in all three languages for each task, although diagnostic scores will also be provided for each individual language. The source corpus for all KBP 2016 tasks will comprise the same set of up to 90K English, Chinese, and Spanish newswire and discussion forum documents. Of these, approximately 500 "core" documents will be annotated with entities, relations, and events (ERE) according to the guidelines for Rich ERE, and will be used to evaluate system responses for the EDL, Event, and Belief and Sentiment tracks. Post-submission assessment procedures will be used to evaluate system responses for the Cold Start SF/KB Construction tasks, and cross-document event frames in the Event Argument Extraction and Linking task.

Tracks

  • Cold Start KBP
    The Cold Start KBP track builds a knowledge base from scratch using a given document collection and a predefined schema for the entities and relations that will comprise the KB. In addition to an end-to-end KB Construction task, Cold Start KBP includes a Slot Filling (SF) task to fill in values for predefined slots (attributes) for a given entity.
    Track home page: http://tac.nist.gov/2016/KBP/ColdStart/
    Track coordinators: Hoa Dang (hoa.dang@nist.gov) and Shahzad Rajput (shahzad.rajput@nist.gov)

  • Validation/Ensembling Track
    The Validation/Ensembling track focuses on the refinement of output from slot filling systems by either combining information from multiple slot filling systems, or applying more intensive linguistic processing to validate individual candidate slot fillers.
    Track home page: http://tac.nist.gov/2016/KBP/SFValidation/
    Track coordinator: Hoa Dang (hoa.dang@nist.gov)

  • Entity Discovery and Linking (EDL)
    The Entity Discovery and Linking (EDL) track aims to extract entity mentions from a source collection of textual documents in multiple languages (English, Chinese, and Spanish), and link them to an existing Knowledge Base (KB); an EDL system is also required to cluster mentions for those entities that don't have corresponding KB entries.
    Track home page: http://nlp.cs.rpi.edu/kbp/2016/
    Track coordinator: Heng Ji (jih@rpi.edu)

  • Event Track
    The goal of the Event track is to extract information about events such that the information would be suitable as input to a knowledge base. The track includes Event Nugget (EN) tasks to detect and link events, and Event Argument (EA) tasks to extract event arguments and link arguments that belong to the same event.
    Track home page: http://tac.nist.gov/2016/KBP/Event/
    Event Nugget coordinators: Eduard Hovy (ehovy@andrew.cmu.edu) and Teruko Mitamura (teruko@cs.cmu.edu)
    Event Argument coordinator: Marjorie Freedman (mfreedma@bbn.com)

  • Belief/Sentiment Track
    The Belief and Sentiment track detects belief and sentiment of an entity toward another entity, relation, or event.
    Track home page: http://www.cs.columbia.edu/~rambow/best-eval-2016/
    Track coordinator: Owen Rambow (rambow@ccls.columbia.edu)

Preliminary Schedule

    Preliminary TAC KBP 2016 Schedule
    February 29Track registration opens
    July 15Deadline for registration for track participation
    August - OctoberTrack evaluation windows (varies by track)
    August 1-14EDL First Evaluation Window (EDL1)
    August 15-29Cold Start KB/SF Evaluation Window
    August 22-Sept 7Event Argument Extraction and Linking Evaluation Window
    Sept 1Release EDL1 scores to individual participants
    Sept 12-19EDL Evaluation Window 2 (EDL2)
    Sept 20-Oct 3Event Nugget Detection and Coreference Evaluation Window
    Sept 26-Oct 3Slot Filler Validation/Ensembling Evaluation Window
    Oct 10-17Belief and Sentiment Evaluation Window
    By mid OctoberRelease of individual evaluated results to participants (most tracks)
    October 10Deadline for short system descriptions
    October 18Deadline for workshop presentation proposals
    October 20Notification of acceptance of presentation proposals
    Nov 1Deadline for system reports (workshop notebook version)
    November 14-15TAC 2016 workshop in Gaithersburg, Maryland, USA
    February 2017Deadline for system reports (final proceedings version)

Organizing Committee

Hoa Trang Dang (U.S. National Institute of Standards and Techonology)
Jason Duncan (MITRE)
Joe Ellis (Linguistic Data Consortium)
Marjorie Freedman (BBN Technologies)
Ralph Grishman (New York University)
Eduard Hovy (Carnegie Mellon University)
Heng Ji (Rensselaer Polytechnic Institute)
James Mayfield (Johns Hopkins University)
Teruko Mitamura (Carnegie Mellon University)
Boyan Onyshkevych (U.S. Department of Defense)
Shahzad Rajput (U.S. National Institute of Standards and Techonology)
Owen Rambow (Columbia University)
Zhiyi Song (Linguistic Data Consortium)
Stephanie Strassel (Linguistic Data Consortium)


NIST is an agency of the
U.S. Department of Commerce

privacy policy / security notice / accessibility statement
disclaimer
FOIA

Last updated: Friday, 17-Feb-2017 17:24:59 EST
Comments to: tac-web@nist.gov