TAC KBP 2014 Event Track

Overview

The goal of the TAC KBP Event track is to extract information about events such that the information would be suitable as input to a knowledge base. The Event Argument Extraction task aims to extract information about entities (and times) and the role they play in an event. Participating systems will extract tuples that include (EventType, Role, Argument). EventType and Role will be drawn from an externally specified ontology that is based on ACE 2005. Arguments will be strings from within a document representing the canonical (most-specific) name or description of the entity.

While this task does not require the reification of events (or linking the different arguments of an event), systems developed for this task will support KB-queries that link entities to participation in an event-- for example "List ORGANIZATIONs with a PURCHASER role". When combined with other KBP technologies (i.e. slot-filling and entity-linking), more complex, multi-hop queries are possible-- for example, "List ORGANIZATIONSs with MEMBERs who have served as an ATTACKER".

This is a new task in 2014 and will be evaluated in English only. In 2015, we expect to extend to additional TAC languages and add an evaluation of a system's ability to reify events and connect their arguments.

Task

Systems will be given a 500-1,000 document corpus and asked to extract instances of arguments that play a role in some event. The corpus will be a mix of newswire and discussion forum documents. A discussion forum document may contain multiple posts. The corpus will be manually and automatically filtered to ensure at least a few instances of all event-types. The discussion-forum posts will be automatically filtered to identify those posts that are not simply reposts of newswire documents. Very long discussion-forum threads will be truncated.

Systems will be evaluated on F-Score of (EventType, Role, NormalizedArgumentString, Realis) over the pooled assessments. The assessment pool for each document will include a human-participant. The human-participants submission will be assessed in the same manner as the system submissions.

Preliminary Schedule

March	Track guidelines posted. ACE 2005 data released to participants as they sign up.
April 1	LDC releases sample documents (~50) to participants in preparation for pilot assessment
April 15	Participants (optionally) submit pilot output for pilot assessment. Pilot may be automatic, manual, or some combination. While participation is not required, participants are encouraged to submit output to better understand this new task (e.g. receive feedback on the assessment of inference, nominal extents, realis).
May 15	LDC releases the full set of pilot assessment to participants who submitted pilot output.
June 15	Deadline for registration for track participation
August 11-18	Event Argument Extraction evaluation window
October	Release of individual evaluated results to participants
October 7	Deadline for short system descriptions
October 7	Deadline for workshop presentation proposals
mid October	Notification of acceptance of presentation proposals
Nov 1 Nov 9	Deadline for system reports (workshop notebook version)
November 17-18	TAC 2014 workshop in Gaithersburg, Maryland, USA
February 15, 2015	Deadline for system reports (final proceedings version)

References

• ACE 2005 Training Data: http://catalog.ldc.upenn.edu/LDC2006T06

While the ACE 2005 event annotation is being provided to all participants, this task diverges from ACE in some cases. One example of divergence is the addition of correct answers derived through inference/world knowledge. This evaluation will treat as correct some cases that were explicitly excluded in ACE 2005.

Track Coordinator