TAC KBP 2015 Event Track

Overview

The goal of the TAC KBP Event track is to extract information about events such that the information would be suitable as input to a knowledge base. The track includes Event Nugget (EN) tasks to detect and link events, and Event Argument (EA) tasks to extract event arguments and link arguments that belong to the same event. In 2015, there are 5 sub-tasks of the TAC KBP Event Track:

• EN Task 1: Event Nugget Detection: This task aims to identify the explicit mentioning of Events in text for English. Participating systems must identify all relevant Event Mention instances within each sentence. Every instance of a mention of the relevant Event types/subtypes taken from the Rich ERE Annotation guidelines must be identified. In addition, systems must identify three REALIS values (ACTUAL, GENERIC, OTHER), which are also described in the Rich ERE guidelines.


• EN Task 2: Event Nugget Detection and Coreference: In addition to the Event Nugget Detection task described in the EN Task 1, this task also aims to identify Full Event Coreference links at the same time. Full Event Coreference is identified when two or more Event Nuggets refer to the same event. This notion is described as Event Hoppers in the Rich ERE Annotation Guidelines. The Full Event Coreference links do not include subevents relations.


• EN Task 3: Event Nugget Coreference: This task is to identify Full Event Coreference links, given the annotated Event Nuggets in the text.


• EA Task 1: Event Argument Extraction and Linking: Systems in this task aim to extract information about entities (and times) and the role they play in an event and then link those arguments that participate in the same event. Participating systems will extract tuples that include (EventType, Role, Argument) and group these tuples so that, for example the date, location and thing purchased of the same transaction event appear in the same frame. EventType and Role will be drawn from an externally specified ontology that is based on ACE 2005 and Rich ERE. Arguments will be strings from within a document representing the canonical (most-specific) name or description of the entity.


• EA Task 2: Event Argument Verification and Linking: This task is designed to allow participants to explore approaches to event argument linking without building their own event argument extraction systems. It also allows participants to explore approaches that improve the precision of extracted event arguments via system combination and/or verification algorithms. The output of this task is identical to that of EA Task 1 (Event Argument Extraction and Linking). Participants in this task will receive unlinked argument extraction output from all EA Task 1 participants.

For all 5 subtasks of the Event Track, systems will operate over a 100-500 document English corpus that is a mix of newswire and discussion forum documents. A discussion forum document may contain multiple posts. The corpus will be manually and automatically filtered to ensure at least a few instances of all event-types. The discussion-forum posts will be automatically filtered to identify those posts that are not simply reposts of newswire documents. Very long discussion-forum threads will be truncated.

The input documents for the EN tasks will be a subset of the input documents for EA.

Participants can submit to any number and combination of the 5 tasks in the Event track. Because input documents are shared across the 5 tasks, participants must refrain from examining any of the input documents before they have finished submitting results for all of the Event tasks in which they are participating.

Preliminary Schedule

Preliminary TAC KBP 2015 Schedule
April 15	Track registration opens
Mid April	Initial track guidelines posted
June 30	Deadline for registration for track participation
August 17-24	Event (Argument Extraction and Linking) evaluation window
August 31-Sept 7	Event (Argument Verification and Linking) evaluation window
Sept 8-21	Event (Nugget Detection) evaluation window
Sept 8-21	Event (Nugget Detection and Coreference) evaluation window
Sept 22-29	Event (Nugget Coreference) evaluation window
By mid October	Release of individual evaluated results to participants (varies by track)
October 20	Deadline for short system descriptions
October 20	Deadline for workshop presentation proposals
October 25	Notification of acceptance of presentation proposals
Nov 8	Deadline for system reports (workshop notebook version)
November 16-17	TAC 2015 workshop in Gaithersburg, Maryland, USA
March 5, 2016	Deadline for system reports (final proceedings version)

References

• ACE 2005 Training Data: http://catalog.ldc.upenn.edu/LDC2006T06

While the ACE 2005 event annotation is being provided to all participants, this task diverges from ACE in some cases. One example of divergence is the addition of correct answers derived through inference/world knowledge. This evaluation will treat as correct some cases that were explicitly excluded in ACE 2005.

Track Coordinators