TAC Knowledge Base Population (KBP) 2016

Overview

The Text Analysis Conference (TAC) is a series of evaluation workshops organized to encourage research in Natural Language Processing and related applications, by providing a large test collection, common evaluation procedures, and a forum for organizations to share their results. TAC comprises sets of tasks known as "tracks," each of which focuses on a particular subproblem of NLP. TAC tracks focus on end-user tasks, but also include component evaluations situated within the context of end-user tasks.

The goal of TAC Knowledge Base Population (KBP) is to develop and evaluate technologies for populating knowledge bases (KBs) from unstructured text.

The end-to-end Cold Start KBP task is to build a KB from scratch, using a predefined KB schema and a collection of unstructured text. The current KB schema consists of entities that can be a specific individual person (PER), organization (ORG), geopolitical entity (GPE), location (LOC), or facility (FAC); and predefined attributes (a.k.a "slots") for those entities. The submitted Cold Start KBs are evaluated by applying a set of slot filling evaluation queries to each KB and assessing the correctness of the relations found.

In addition to end-to-end Cold Start KB construction, two diagnostic tasks and evaluations are offered:

1. Entity Discovery and Linking (EDL): The EDL task is to extract name and nominal mentions of specific individual PER, ORG, GPE, LOC, and FAC entities mentioned in the Cold Start document collection, and to link each mention to its KB node (either a node in the TAC reference KB, or a newly created NIL node if it doesn't have a corresponding KB entry).
2. Slot Filling (SF): The slot filling task is to search the Cold Start document collection to fill in values for specific slots for specific entities.

The slot filler validation task in the validation/ensembling track focuses on the refinement of output from slot filling systems by either combining information from multiple slot filling systems, or applying more intensive linguistic processing to validate candidate slot fillers.

The event track in KBP 2016 aims to extract information about events from unstructured text, such that the information would be suitable as input into a structured KB. The track includes Event Nugget (EN) tasks to detect and link events, and Event Argument (EA) tasks to extract event arguments and link arguments that belong to the same event.

Finally, the Belief and Sentiment track detects belief and sentiment of an entity toward another entity, relation, or event.

In 2016, all tasks will be multi-lingual (English, Chinese, and Spanish); Cold Start EDL, SF and KB construction tasks will also be cross-lingual. Participants are encouraged to participate in all three languages for each task, although diagnostic scores will also be provided for each individual language. The source corpus for all KBP 2016 tasks will comprise the same set of up to 90K English, Chinese, and Spanish newswire and discussion forum documents. Of these, approximately 500 "core" documents will be annotated with entities, relations, and events (ERE) according to the guidelines for Rich ERE, and will be used to evaluate system responses for the EDL, Event, and Belief and Sentiment tracks. Post-submission assessment procedures will be used to evaluate system responses for the Cold Start SF/KB Construction tasks, and cross-document event frames in the Event Argument Extraction and Linking task.

Tracks

• Cold Start KBP
  The Cold Start KBP track builds a knowledge base from scratch using a given document collection and a predefined schema for the entities and relations that will comprise the KB. In addition to an end-to-end KB Construction task, Cold Start KBP includes a Slot Filling (SF) task to fill in values for predefined slots (attributes) for a given entity.
  Track home page: http://tac.nist.gov/2016/KBP/ColdStart/
  Track coordinators: Hoa Dang (hoa.dang@nist.gov) and Shahzad Rajput (shahzad.rajput@nist.gov)
  


• Validation/Ensembling Track
  The Validation/Ensembling track focuses on the refinement of output from slot filling systems by either combining information from multiple slot filling systems, or applying more intensive linguistic processing to validate individual candidate slot fillers.
  Track home page: http://tac.nist.gov/2016/KBP/SFValidation/
  Track coordinator: Hoa Dang (hoa.dang@nist.gov)
  


• Entity Discovery and Linking (EDL)
  The Entity Discovery and Linking (EDL) track aims to extract entity mentions from a source collection of textual documents in multiple languages (English, Chinese, and Spanish), and link them to an existing Knowledge Base (KB); an EDL system is also required to cluster mentions for those entities that don't have corresponding KB entries.
  Track home page: http://nlp.cs.rpi.edu/kbp/2016/
  Track coordinator: Heng Ji (jih@rpi.edu)
  


• Event Track
  The goal of the Event track is to extract information about events such that the information would be suitable as input to a knowledge base. The track includes Event Nugget (EN) tasks to detect and link events, and Event Argument (EA) tasks to extract event arguments and link arguments that belong to the same event.
  Track home page: http://tac.nist.gov/2016/KBP/Event/
  Event Nugget coordinators: Eduard Hovy (ehovy@andrew.cmu.edu) and Teruko Mitamura (teruko@cs.cmu.edu)
  Event Argument coordinator: Marjorie Freedman (mfreedma@bbn.com)
  


• Belief/Sentiment Track
  The Belief and Sentiment track detects belief and sentiment of an entity toward another entity, relation, or event.
  Track home page: http://www.cs.columbia.edu/~rambow/best-eval-2016/
  Track coordinator: Owen Rambow (rambow@ccls.columbia.edu)
  

Preliminary Schedule

Preliminary TAC KBP 2016 Schedule
February 29	Track registration opens
July 15	Deadline for registration for track participation
August - October	Track evaluation windows (varies by track)
August 1-14	EDL First Evaluation Window (EDL1)
August 15-29	Cold Start KB/SF Evaluation Window
August 22-Sept 7	Event Argument Extraction and Linking Evaluation Window
Sept 1	Release EDL1 scores to individual participants
Sept 12-19	EDL Evaluation Window 2 (EDL2)
Sept 20-Oct 3	Event Nugget Detection and Coreference Evaluation Window
Sept 26-Oct 3	Slot Filler Validation/Ensembling Evaluation Window
Oct 10-17	Belief and Sentiment Evaluation Window
By mid October	Release of individual evaluated results to participants (most tracks)
October 10	Deadline for short system descriptions
October 18	Deadline for workshop presentation proposals
October 20	Notification of acceptance of presentation proposals
Nov 1	Deadline for system reports (workshop notebook version)
November 14-15	TAC 2016 workshop in Gaithersburg, Maryland, USA
February 2017	Deadline for system reports (final proceedings version)

Organizing Committee