
TAC 2016 Tracks
Cold Start SF/KB
EDL
Ensembling
Event
Belief/Sentiment
Data
Schedule
Organizers
Track Registration
Reporting Guidelines
TAC 2016 Workshop

|
|
TAC Knowledge Base Population (KBP) 2016
Evaluation: February-November, 2016
Workshop: November 14-15, 2016
Conducted by:
U.S. National Institute of Standards and Technology (NIST)
With support from:
U.S. Department of Defense
Overview
The Text Analysis Conference (TAC) is a series of evaluation workshops
organized to encourage research in Natural Language Processing and
related applications, by providing a large test collection, common
evaluation procedures, and a forum for organizations to share their
results. TAC comprises sets of tasks known as "tracks," each of which
focuses on a particular subproblem of NLP. TAC tracks focus on
end-user tasks, but also include component evaluations situated within
the context of end-user tasks.
The goal of TAC Knowledge Base Population (KBP) is to develop and
evaluate technologies for populating knowledge bases (KBs) from
unstructured text.
The end-to-end Cold Start KBP task is to build a KB from scratch,
using a predefined KB schema and a collection of unstructured
text. The current KB schema consists of entities that can be a
specific individual person (PER), organization (ORG), geopolitical
entity (GPE), location (LOC), or facility (FAC); and predefined
attributes (a.k.a "slots") for those entities. The submitted Cold
Start KBs are evaluated by applying a set of slot filling evaluation
queries to each KB and assessing the correctness of the relations
found.
In addition to end-to-end Cold Start KB construction,
two diagnostic tasks and evaluations are offered:
- Entity Discovery and Linking (EDL): The EDL task is to extract
name and nominal mentions of specific individual PER, ORG, GPE, LOC,
and FAC entities mentioned in the Cold Start document collection, and to link
each mention to its KB node (either a node in the TAC reference KB, or
a newly created NIL node if it doesn't have a corresponding KB
entry).
- Slot Filling (SF): The slot filling task is to search the Cold Start
document collection to fill in values for specific slots for specific
entities.
The slot filler validation task in the validation/ensembling
track focuses on the refinement of output from slot filling systems by
either combining information from multiple slot filling systems, or
applying more intensive linguistic processing to validate candidate
slot fillers.
The event track in KBP 2016 aims to extract information about
events from unstructured text, such that the information would be
suitable as input into a structured KB. The track includes Event
Nugget (EN) tasks to detect and link events, and Event Argument (EA)
tasks to extract event arguments and link arguments that belong to the
same event.
Finally, the Belief and Sentiment track detects belief and sentiment of an entity toward another entity, relation, or event.
In 2016, all tasks will be multi-lingual (English, Chinese, and
Spanish); Cold Start EDL, SF and KB construction tasks will also be
cross-lingual. Participants are encouraged to participate in all three
languages for each task, although diagnostic scores will also be
provided for each individual language. The source corpus for all KBP
2016 tasks will comprise the same set of up to 90K English, Chinese,
and Spanish newswire and discussion forum documents. Of these,
approximately 500 "core" documents will be annotated with entities,
relations, and events (ERE) according to the guidelines for Rich ERE, and will
be used to evaluate system responses for the EDL, Event, and Belief
and Sentiment tracks. Post-submission assessment procedures will be
used to evaluate system responses for the Cold Start SF/KB
Construction tasks, and cross-document event frames in the Event
Argument Extraction and Linking task.
Tracks
- Cold Start KBP
The Cold Start KBP track builds a knowledge base from scratch using
a given document collection and a predefined schema for the entities
and relations that will comprise the KB. In addition to an end-to-end
KB Construction task, Cold Start KBP includes a Slot
Filling (SF) task to fill in values for predefined slots
(attributes) for a given entity.
Track home page: http://tac.nist.gov/2016/KBP/ColdStart/
Track coordinators: Hoa Dang (hoa.dang@nist.gov) and Shahzad Rajput (shahzad.rajput@nist.gov)
- Validation/Ensembling Track
The Validation/Ensembling track focuses on the refinement of output from slot filling systems by either combining information from
multiple slot filling systems, or applying more intensive linguistic processing to validate individual candidate slot fillers.
Track home page: http://tac.nist.gov/2016/KBP/SFValidation/
Track coordinator: Hoa Dang (hoa.dang@nist.gov)
- Entity Discovery and Linking (EDL)
The Entity Discovery and Linking (EDL) track aims to extract entity mentions from a source collection of textual documents in multiple languages (English, Chinese, and Spanish), and link them to an existing Knowledge Base (KB); an EDL system is also required to cluster mentions for those entities that don't have corresponding KB entries.
Track home page: http://nlp.cs.rpi.edu/kbp/2016/
Track coordinator: Heng Ji (jih@rpi.edu)
- Event Track
The goal of the Event track is to extract information about events such that the information would be suitable as input to a knowledge base. The track includes Event Nugget (EN) tasks to detect and link events, and Event Argument (EA) tasks to extract event arguments and link arguments that belong to the same event.
Track home page: http://tac.nist.gov/2016/KBP/Event/
Event Nugget coordinators: Eduard Hovy (ehovy@andrew.cmu.edu) and Teruko Mitamura (teruko@cs.cmu.edu)
Event Argument coordinator: Marjorie Freedman (mfreedma@bbn.com)
- Belief/Sentiment Track
The Belief and Sentiment track detects belief and sentiment of an entity toward another entity, relation, or event.
Track home page: http://www.cs.columbia.edu/~rambow/best-eval-2016/
Track coordinator: Owen Rambow (rambow@ccls.columbia.edu)
Preliminary Schedule
Preliminary TAC KBP 2016 Schedule |
February 29 | Track registration opens |
July 15 | Deadline for registration for track participation |
August - October | Track evaluation windows (varies by track) |
August 1-14 | EDL First Evaluation Window (EDL1) |
August 15-29 | Cold Start KB/SF Evaluation Window |
August 22-Sept 7 | Event Argument Extraction and Linking Evaluation Window |
Sept 1 | Release EDL1 scores to individual participants |
Sept 12-19 | EDL Evaluation Window 2 (EDL2) |
Sept 20-Oct 3 | Event Nugget Detection and Coreference Evaluation Window |
Sept 26-Oct 3 | Slot Filler Validation/Ensembling Evaluation Window |
Oct 10-17 | Belief and Sentiment Evaluation Window |
By mid October | Release of individual evaluated results to participants (most tracks) |
October 10 | Deadline for short system descriptions |
October 18 | Deadline for workshop presentation proposals |
October 20 | Notification of acceptance of presentation proposals |
Nov 1 | Deadline for system reports (workshop notebook version) |
November 14-15 | TAC 2016 workshop in Gaithersburg, Maryland, USA |
February 2017 | Deadline for system reports (final proceedings version) |
Organizing Committee
Hoa Trang Dang (U.S. National Institute of Standards and Techonology)
Jason Duncan (MITRE)
Joe Ellis (Linguistic Data Consortium)
Marjorie Freedman (BBN Technologies)
Ralph Grishman (New York University)
Eduard Hovy (Carnegie Mellon University)
Heng Ji (Rensselaer Polytechnic Institute)
James Mayfield (Johns Hopkins University)
Teruko Mitamura (Carnegie Mellon University)
Boyan Onyshkevych (U.S. Department of Defense)
Shahzad Rajput (U.S. National Institute of Standards and Techonology)
Owen Rambow (Columbia University)
Zhiyi Song (Linguistic Data Consortium)
Stephanie Strassel (Linguistic Data Consortium)
|