TAC KBP 2015 Event Argument Tasks

Overview

The two Event Argument (EA) tasks of the TAC KBP Event track focus on extracting event arguments and linking arguments that belong to the same event.

Systems in this task aim to extract information about entities (and times) and the role they play in an event and then link those arguments that participate in the same event. Participating systems will extract tuples that include (EventType, Role, Argument) and group these tuples so that, for example the date, location and thing purchased of the same transaction event appear in the same frame. EventType and Role will be drawn from an externally specified ontology that is based on ACE 2005 and Rich ERE. Arguments will be strings from within a document representing the canonical (most-specific) name or description of the entity.

The input to this task will be unannotated documents. The output will be a set of event argument extraction assertions and links between these assertions indicating participation in the same event. The official metric will measure argument extraction and linking together. Diagnostic metrics will include measures of only event argument extraction quality (F1 as in 2014) and other measures. A simple baseline event argument linking strategy which links all event arguments where the event type/subtype are the same together has been implemented by BBN and will be distributed via github to all participants.

This task is designed to allow participants to explore approaches to event argument linking without building their own event argument extraction systems. It also allows participants to explore approaches that improve the precision of extracted event arguments via system combination and/or verification algorithms. The output of this task is identical to that of EA Task 1 (Event Argument Extraction and Linking). Participants in this task will receive unlinked argument extraction output from all EA Task 1 participants.

The input to this task will be the event argument extraction output of all systems that participate in EA Task 1. The output of this task and official metric will be the same as EA Task 1. Participants in this task will have training data in the form of the assessments from the 2014 Event Argument Extraction task, but there will not be training data created specifically connected to the 2015 EA Task 1 submissions. Participants in this task are expected to verify event argument extractions (i.e. remove incorrect event argument extractions), but will not be allowed to add additional event argument extractions.

Preliminary Schedule

Preliminary TAC KBP 2015 Schedule
April 15	Track registration opens
Mid April	Initial track guidelines posted
June 30	Deadline for registration for track participation
August 17-24	Event (Argument Extraction and Linking) evaluation window
August 31-Sept 7	Event (Argument Verification and Linking) evaluation window
By mid October	Release of individual evaluated results to participants (varies by track)
October 20	Deadline for short system descriptions
October 20	Deadline for workshop presentation proposals
October 25	Notification of acceptance of presentation proposals
Nov 8	Deadline for system reports (workshop notebook version)
November 16-17	TAC 2015 workshop in Gaithersburg, Maryland, USA
March 5, 2016	Deadline for system reports (final proceedings version)

References

• ACE 2005 Training Data: http://catalog.ldc.upenn.edu/LDC2006T06

While the ACE 2005 event annotation is being provided to all participants, this task diverges from ACE in some cases. One example of divergence is the addition of correct answers derived through inference/world knowledge. This evaluation will treat as correct some cases that were explicitly excluded in ACE 2005.

Task Coordinator