TAC Knowledge Base Population (KBP) 2015

Overview

The Text Analysis Conference (TAC) is a series of evaluation workshops organized to encourage research in Natural Language Processing and related applications, by providing a large test collection, common evaluation procedures, and a forum for organizations to share their results. TAC comprises sets of tasks known as "tracks," each of which focuses on a particular subproblem of NLP. TAC tracks focus on end-user tasks, but also include component evaluations situated within the context of end-user tasks.

The goal of TAC Knowledge Base Population (KBP) is to develop and evaluate technologies for populating knowledge bases (KBs) from unstructured text.

The Cold Start KBP task is to build a KB from scratch, using a predefined KB schema and a collection of unstructured text. The current KB schema consists of named entities that can be a person (PER), organization (ORG), or geopolitical entity (GPE); and predefined attributes (a.k.a "slots") for those named entities. In addition to end-to-end KB construction, the Cold Start track offers two diagnostic tasks: Entity Discovery (ED) and Slot Filling (SF). The Cold Start entity discovery task is to create a KB node for each PER, ORG, and GPE entity mentioned by name in the Cold Start collection of text, and to link all named mentions of the entity to its KB node. The Cold Start slot filling task is to search the same document collection to fill in values for specific slots for specific entities.

The Tri-lingual Entity Discovery and Linking (EDL) track aims to extract entity mentions from a source collection of textual documents in multiple languages (English, Chinese and Spanish), and link them to an existing Knowledge Base (KB). An EDL system is also required to cluster mentions for those NIL entities that don't have corresponding KB entries.

The event track in KBP 2015 aims to extract information about events from unstructured text, such that the information would be suitable as input into a structured KB.

Finally, the slot filler validation task in the validation/ensembling track focuses on the refinement of output from slot filling systems by either combining information from multiple slot filling systems, or applying more intensive linguistic processing to validate candidate slot fillers.

Tracks

• Cold Start KBP Track
  The Cold Start KBP track builds a knowledge base from scratch using a given document collection and a predefined schema for the entities and relations that will comprise the KB. Cold Start KBP includes an Entity Discovery (ED) task to discover entities and cluster their named mentions, and a Slot Filling (SF) task to fill in values for predefined slots (attributes) for a given entity.
  Track home page: http://tac.nist.gov/2015/KBP/ColdStart/
  Track coordinators: James Mayfield (jamesmayfield@gmail.com) and Ralph Grishman (grishman@cs.nyu.edu)
  


• Tri-Lingual Entity Discovery and Linking Track (EDL)
  The Tri-Lingual Entity Discovery and Linking (EDL) track aims to extract entity mentions from a source collection of textual documents in multiple languages (English, Chinese, and Spanish), and link them to an existing Knowledge Base (KB); an EDL system is also required to cluster mentions for those entities that don't have corresponding KB entries.
  Track home page: http://nlp.cs.rpi.edu/kbp/2015/
  Track coordinator: Heng Ji (jih@rpi.edu)
  


• Event Track
  The goal of the Event track is to extract information about events such that the information would be suitable as input to a knowledge base. The track includes Event Nugget (EN) tasks to detect and link events, and Event Argument (EA) tasks to extract event arguments and link arguments that belong to the same event.
  Track home page: http://tac.nist.gov/2015/KBP/Event/
  Event Nugget coordinators: Eduard Hovy (ehovy@andrew.cmu.edu) and Teruko Mitamura (teruko@cs.cmu.edu)
  Event Argument coordinator: Marjorie Freedman (mfreedma@bbn.com)
  


• Validation/Ensembling Track
  The Validation/Ensembling track focuses on the refinement of output from slot filling systems by either combining information from multiple slot filling systems, or applying more intensive linguistic processing to validate individual candidate slot fillers.
  Track home page: http://tac.nist.gov/2015/KBP/SFValidation/
  Track coordinator: Hoa Dang (hoa.dang@nist.gov)
  

Preliminary Schedule

Preliminary TAC KBP 2015 Schedule
April 15	Track registration opens
Mid April	Initial track guidelines posted
June 30	Deadline for registration for track participation
August - October	Track evaluation windows (varies by track)
August 3-31	Cold Start KBP evaluation window
August 17-24	Event (Argument Extraction and Linking) evaluation window
August 31-Sept 7	Event (Argument Verification and Linking) evaluation window
Sept 8-21	Event (Nugget Detection) evaluation window
Sept 8-21	Event (Nugget Detection and Coreference) evaluation window
Sept 22-29	Event (Nugget Coreference) evaluation window
Sept 21-Oct 12	Slot Filler Validation/Ensembling evaluation window
Sept 28-Oct 12	Tri-Lingual EDL evaluation window
Oct 14-19	Diagnostic Tri-Lingual EL evaluation window
By mid October	Release of individual evaluated results to participants (most tracks)
October 20	Deadline for short system descriptions
October 20	Deadline for workshop presentation proposals
October 25	Notification of acceptance of presentation proposals
Nov 8	Deadline for system reports (workshop notebook version)
November 16-17	TAC 2015 workshop in Gaithersburg, Maryland, USA
March 5, 2016	Deadline for system reports (final proceedings version)

Organizing Committee