Go to the NIST home page Go to the TAC Home Page TAC Banner

Return to TAC Homepage

TAC 2015 Tracks
    Cold Start KBP
    Tri-Lingual EDL
    Event
    Validation/Ensembling
Data
Schedule
Organizers
Track Registration
Reporting Guidelines
TAC 2015 Workshop




TAC Knowledge Base Population (KBP) 2015

Evaluation: February-November, 2015
Workshop: November 16-17, 2015

Conducted by:
U.S. National Institute of Standards and Technology (NIST)

With support from:
U.S. Department of Defense

Overview

The Text Analysis Conference (TAC) is a series of evaluation workshops organized to encourage research in Natural Language Processing and related applications, by providing a large test collection, common evaluation procedures, and a forum for organizations to share their results. TAC comprises sets of tasks known as "tracks," each of which focuses on a particular subproblem of NLP. TAC tracks focus on end-user tasks, but also include component evaluations situated within the context of end-user tasks.

The goal of TAC Knowledge Base Population (KBP) is to develop and evaluate technologies for populating knowledge bases (KBs) from unstructured text.

The Cold Start KBP task is to build a KB from scratch, using a predefined KB schema and a collection of unstructured text. The current KB schema consists of named entities that can be a person (PER), organization (ORG), or geopolitical entity (GPE); and predefined attributes (a.k.a "slots") for those named entities. In addition to end-to-end KB construction, the Cold Start track offers two diagnostic tasks: Entity Discovery (ED) and Slot Filling (SF). The Cold Start entity discovery task is to create a KB node for each PER, ORG, and GPE entity mentioned by name in the Cold Start collection of text, and to link all named mentions of the entity to its KB node. The Cold Start slot filling task is to search the same document collection to fill in values for specific slots for specific entities.

The Tri-lingual Entity Discovery and Linking (EDL) track aims to extract entity mentions from a source collection of textual documents in multiple languages (English, Chinese and Spanish), and link them to an existing Knowledge Base (KB). An EDL system is also required to cluster mentions for those NIL entities that don't have corresponding KB entries.

The event track in KBP 2015 aims to extract information about events from unstructured text, such that the information would be suitable as input into a structured KB.

Finally, the slot filler validation task in the validation/ensembling track focuses on the refinement of output from slot filling systems by either combining information from multiple slot filling systems, or applying more intensive linguistic processing to validate candidate slot fillers.

Tracks

  • Cold Start KBP Track
    The Cold Start KBP track builds a knowledge base from scratch using a given document collection and a predefined schema for the entities and relations that will comprise the KB. Cold Start KBP includes an Entity Discovery (ED) task to discover entities and cluster their named mentions, and a Slot Filling (SF) task to fill in values for predefined slots (attributes) for a given entity.
    Track home page: http://tac.nist.gov/2015/KBP/ColdStart/
    Track coordinators: James Mayfield (jamesmayfield@gmail.com) and Ralph Grishman (grishman@cs.nyu.edu)

  • Tri-Lingual Entity Discovery and Linking Track (EDL)
    The Tri-Lingual Entity Discovery and Linking (EDL) track aims to extract entity mentions from a source collection of textual documents in multiple languages (English, Chinese, and Spanish), and link them to an existing Knowledge Base (KB); an EDL system is also required to cluster mentions for those entities that don't have corresponding KB entries.
    Track home page: http://nlp.cs.rpi.edu/kbp/2015/
    Track coordinator: Heng Ji (jih@rpi.edu)

  • Event Track
    The goal of the Event track is to extract information about events such that the information would be suitable as input to a knowledge base. The track includes Event Nugget (EN) tasks to detect and link events, and Event Argument (EA) tasks to extract event arguments and link arguments that belong to the same event.
    Track home page: http://tac.nist.gov/2015/KBP/Event/
    Event Nugget coordinators: Eduard Hovy (ehovy@andrew.cmu.edu) and Teruko Mitamura (teruko@cs.cmu.edu)
    Event Argument coordinator: Marjorie Freedman (mfreedma@bbn.com)

  • Validation/Ensembling Track
    The Validation/Ensembling track focuses on the refinement of output from slot filling systems by either combining information from multiple slot filling systems, or applying more intensive linguistic processing to validate individual candidate slot fillers.
    Track home page: http://tac.nist.gov/2015/KBP/SFValidation/
    Track coordinator: Hoa Dang (hoa.dang@nist.gov)

Preliminary Schedule

    Preliminary TAC KBP 2015 Schedule
    April 15Track registration opens
    Mid AprilInitial track guidelines posted
    June 30Deadline for registration for track participation
    August - OctoberTrack evaluation windows (varies by track)
    August 3-31Cold Start KBP evaluation window
    August 17-24Event (Argument Extraction and Linking) evaluation window
    August 31-Sept 7Event (Argument Verification and Linking) evaluation window
    Sept 8-21Event (Nugget Detection) evaluation window
    Sept 8-21Event (Nugget Detection and Coreference) evaluation window
    Sept 22-29Event (Nugget Coreference) evaluation window
    Sept 21-Oct 12Slot Filler Validation/Ensembling evaluation window
    Sept 28-Oct 12Tri-Lingual EDL evaluation window
    Oct 14-19Diagnostic Tri-Lingual EL evaluation window
    By mid OctoberRelease of individual evaluated results to participants (most tracks)
    October 20Deadline for short system descriptions
    October 20Deadline for workshop presentation proposals
    October 25Notification of acceptance of presentation proposals
    Nov 8Deadline for system reports (workshop notebook version)
    November 16-17TAC 2015 workshop in Gaithersburg, Maryland, USA
    March 5, 2016Deadline for system reports (final proceedings version)

Organizing Committee

Hoa Trang Dang (U.S. National Institute of Standards and Techonology)
Jason Duncan (MITRE)
Joe Ellis (Linguistic Data Consortium)
Marjorie Freedman (BBN Technologies)
Ralph Grishman (New York University)
Eduard Hovy (Carnegie Mellon University)
Heng Ji (Rensselaer Polytechnic Institute)
James Mayfield (Johns Hopkins University)
Teruko Mitamura (Carnegie Mellon University)
Boyan Onyshkevych (U.S. Department of Defense)
Shahzad Rajput (U.S. National Institute of Standards and Techonology)
Zhiyi Song (Linguistic Data Consortium)
Stephanie Strassel (Linguistic Data Consortium)


NIST is an agency of the
U.S. Department of Commerce

privacy policy / security notice / accessibility statement
disclaimer
FOIA

Last updated: Wednesday, 21-Sep-2016 14:48:08 EDT
Comments to: tac-web@nist.gov