TAC 2015 Tracks
Cold Start KBP
TAC 2015 Workshop
TAC KBP 2015 Event Track
The goal of the TAC KBP Event track is to extract information about
events such that the information would be suitable as input to a
knowledge base. The track includes Event Nugget (EN) tasks to
detect and link events, and Event Argument (EA) tasks to
extract event arguments and link arguments that belong to the same
event. In 2015, there are 5 sub-tasks of the TAC KBP Event Track:
- EN Task 1: Event Nugget Detection: This task aims to identify the explicit mentioning of Events in text for English. Participating systems must identify all relevant Event Mention instances within each sentence. Every instance of a mention of the relevant Event types/subtypes taken from the Rich ERE Annotation guidelines must be identified. In addition, systems must identify three REALIS values (ACTUAL, GENERIC, OTHER), which are also described in the Rich ERE guidelines.
- EN Task 2: Event Nugget Detection and Coreference: In addition to the Event Nugget Detection task described in the EN Task 1, this task also aims to identify Full Event Coreference links at the same time. Full Event Coreference is identified when two or more Event Nuggets refer to the same event. This notion is described as Event Hoppers in the Rich ERE Annotation Guidelines. The Full Event Coreference links do not include subevents relations.
- EN Task 3: Event Nugget Coreference: This task is to identify Full Event Coreference links, given the annotated Event Nuggets in the text.
- EA Task 1: Event Argument Extraction and Linking: Systems in this task aim to extract information about entities (and times) and the role they play in an event and then link those arguments that participate in the same event. Participating systems will extract tuples that include (EventType, Role, Argument) and group these tuples so that, for example the date, location and thing purchased of the same transaction event appear in the same frame. EventType and Role will be drawn from an externally specified ontology that is based on ACE 2005 and Rich ERE. Arguments will be strings from within a document representing the canonical (most-specific) name or description of the entity.
- EA Task 2: Event Argument Verification and Linking: This task is designed to allow participants to explore approaches to event argument linking without building their own event argument extraction systems. It also allows participants to explore approaches that improve the precision of extracted event arguments via system combination and/or verification algorithms. The output of this task is identical to that of EA Task 1 (Event Argument Extraction and Linking). Participants in this task will receive unlinked argument extraction output from all EA Task 1 participants.
For all 5 subtasks of the Event Track, systems will operate over a
100-500 document English corpus that is a mix of newswire and
discussion forum documents. A discussion forum document may contain
multiple posts. The corpus will be manually and automatically filtered
to ensure at least a few instances of all event-types. The
discussion-forum posts will be automatically filtered to identify
those posts that are not simply reposts of newswire documents. Very
long discussion-forum threads will be truncated.
The input documents for the EN tasks will be a subset of the input
documents for EA.
Participants can submit to any number and combination of the 5 tasks
in the Event track. Because input documents are shared across the 5
tasks, participants must refrain from examining any of the input
documents before they have finished submitting results for all of the
Event tasks in which they are participating.
|Preliminary TAC KBP 2015 Schedule|
|April 15||Track registration opens|
|Mid April||Initial track guidelines posted|
|June 30||Deadline for registration for track participation|
|August 17-24||Event (Argument Extraction and Linking) evaluation window|
|August 31-Sept 7||Event (Argument Verification and Linking) evaluation window|
|Sept 8-21||Event (Nugget Detection) evaluation window|
|Sept 8-21||Event (Nugget Detection and Coreference) evaluation window|
|Sept 22-29||Event (Nugget Coreference) evaluation window|
|By mid October||Release of individual evaluated results to participants (varies by track)|
|October 20||Deadline for short system descriptions|
|October 20||Deadline for workshop presentation proposals|
|October 25||Notification of acceptance of presentation proposals|
|Nov 8||Deadline for system reports (workshop notebook version)|
|November 16-17||TAC 2015 workshop in Gaithersburg, Maryland, USA|
|March 5, 2016||Deadline for system reports (final proceedings version)|
While the ACE 2005 event annotation is being provided to all participants, this task diverges from ACE in some cases. One example of divergence is the addition of correct answers derived through inference/world knowledge. This evaluation will treat as correct some cases that were explicitly excluded in ACE 2005.
Event Arguments: Marjorie Freedman (BBN, firstname.lastname@example.org)
Event Nuggets: Teruko Mitamura (CMU, email@example.com) and Eduard Hovy (CMU, firstname.lastname@example.org)