TAC KBP 2013 Sentiment Slot Filling Track

Overview

The overall goal of the Sentiment Slot Filling track is to assess the quality of detectors for scoped and attributed sentiment. This track will supplement existing Entity Linking, Slot Filling, and Cold Start tasks with information about opinions held by people and other entities. The task will be constrained in 2013 to address only positive and negative evaluative sentiment, attributed to a KBP entity, and held about another KBP entity.

Definitions:

• Evaluative sentiment: for the purposes of this task, evaluative sentiment refers to positive or negative sentiment expressed about an entity.
• Scoped sentiment: refers to sentiment as it applies to a specific entity; for the purpose of this task, only sentiments referring to KBP entity types (PERSON, ORGANIZATION, GPE) are relevant, and only if they apply to an entity as a whole. Sentiment that scopes over only one attribute of an entity is not relevant (e.g., "I love the Ravens" is relevant, while "The Ravens had a horrible season this year" is not).
• Attributed sentiment: applies to cases where it is clear who holds the sentiment, and that sentiment holder is a valid KBP entity. The sentiment holder may be the author of the document, and is assumed for Blog or other similar text, but sentiment is generally attributed to a third party in newswire reporting.

The document collection is expected to be primarily blogs with discussion posts, although news articles may also be included. Documents will be formatted in the same way as the documents for the other Knowledge Base Population tracks.

The task is for systems to find all cases where the query entity (sentiment holder) is documented as holding a positive or negative sentiment about another entity (sentiment target). The sentiment slot filling task is a special case of slot filling in which the slots indicate that the query entity has expressed either a positive or negative sentiment toward the entity that is the slot filler. Alternatively, the slots may indicate that the query entity is the target of either a positive or negative sentiment expressed by the slot filler. The slot names in 2013 are limited to the following four sentiment types:

• pos-towards: query entity holds positive sentiment towards filler entity ("likes", "is hopeful about", etc.).
• pos-from: query entity is target of positive sentiment from filler entity ("is liked by", "was hoped for by", etc.).
• neg-towards: query entity holds negative sentiment towards filler entity ("dislikes", "is skeptical about", etc.).
• neg-from: query entity is target of negative sentiment from filler entity ("is disliked by", etc.).

Input and output format and requirements for the sentiment slot filling task are the same as for the English slot filling task, except that slots for the sentiment task are limited to the four sentiment types defined above. In addition to identifying positive and negative sentiment between two KBP entities, systems must be able detect when two entity mentions refer to the same entity, so as to avoid returning redundant slot fillers for a given query entity and sentiment slot. Evaluation will be on slot filler P/R/F1, as for the Engish slot filling task.

Schedule

May	Track guidelines posted
By early June	Distribution of document collections
mid June	Release of Sentiment Slot Filling sample data
June 15	Deadline for registration for track participation
July 29	Release of Sentiment Slot Filling training data
August 26 - September 5	Extended Sentiment Slot Filling evaluation window
September 30	Deadline for short system descriptions
September 30	Deadline for workshop presentation proposals
mid October	Notification of acceptance of presentation proposals
By late October	Release of individual evaluated results to participants
November 8	Deadline for system reports (workshop notebook version)
November 18-19	TAC KBP 2013 workshop in Gaithersburg, Maryland, USA
February 15, 2014	Deadline for system reports (final proceedings version)

Track Coordinator