Go to the NIST home page Go to the TAC Home Page TAC Banner

Return to TAC Homepage

Track Home
Schedule
Guidelines
KBP Data
Tools

KBP 2013 Home




TAC KBP 2013 Sentiment Slot Filling Track

Overview

The overall goal of the Sentiment Slot Filling track is to assess the quality of detectors for scoped and attributed sentiment. This track will supplement existing Entity Linking, Slot Filling, and Cold Start tasks with information about opinions held by people and other entities. The task will be constrained in 2013 to address only positive and negative evaluative sentiment, attributed to a KBP entity, and held about another KBP entity.

Definitions:

  • Evaluative sentiment: for the purposes of this task, evaluative sentiment refers to positive or negative sentiment expressed about an entity.
  • Scoped sentiment: refers to sentiment as it applies to a specific entity; for the purpose of this task, only sentiments referring to KBP entity types (PERSON, ORGANIZATION, GPE) are relevant, and only if they apply to an entity as a whole. Sentiment that scopes over only one attribute of an entity is not relevant (e.g., "I love the Ravens" is relevant, while "The Ravens had a horrible season this year" is not).
  • Attributed sentiment: applies to cases where it is clear who holds the sentiment, and that sentiment holder is a valid KBP entity. The sentiment holder may be the author of the document, and is assumed for Blog or other similar text, but sentiment is generally attributed to a third party in newswire reporting.

The document collection is expected to be primarily blogs with discussion posts, although news articles may also be included. Documents will be formatted in the same way as the documents for the other Knowledge Base Population tracks.

The task is for systems to find all cases where the query entity (sentiment holder) is documented as holding a positive or negative sentiment about another entity (sentiment target). The sentiment slot filling task is a special case of slot filling in which the slots indicate that the query entity has expressed either a positive or negative sentiment toward the entity that is the slot filler. Alternatively, the slots may indicate that the query entity is the target of either a positive or negative sentiment expressed by the slot filler. The slot names in 2013 are limited to the following four sentiment types:

  • pos-towards: query entity holds positive sentiment towards filler entity ("likes", "is hopeful about", etc.).
  • pos-from: query entity is target of positive sentiment from filler entity ("is liked by", "was hoped for by", etc.).
  • neg-towards: query entity holds negative sentiment towards filler entity ("dislikes", "is skeptical about", etc.).
  • neg-from: query entity is target of negative sentiment from filler entity ("is disliked by", etc.).

Input and output format and requirements for the sentiment slot filling task are the same as for the English slot filling task, except that slots for the sentiment task are limited to the four sentiment types defined above. In addition to identifying positive and negative sentiment between two KBP entities, systems must be able detect when two entity mentions refer to the same entity, so as to avoid returning redundant slot fillers for a given query entity and sentiment slot. Evaluation will be on slot filler P/R/F1, as for the Engish slot filling task.

Schedule

    MayTrack guidelines posted
    By early JuneDistribution of document collections
    mid JuneRelease of Sentiment Slot Filling sample data
    June 15Deadline for registration for track participation
    July 29Release of Sentiment Slot Filling training data
    August 26 - September 5Extended Sentiment Slot Filling evaluation window
    September 30Deadline for short system descriptions
    September 30Deadline for workshop presentation proposals
    mid OctoberNotification of acceptance of presentation proposals
    By late OctoberRelease of individual evaluated results to participants
    November 8Deadline for system reports (workshop notebook version)
    November 18-19TAC KBP 2013 workshop in Gaithersburg, Maryland, USA
    February 15, 2014Deadline for system reports (final proceedings version)

Track Coordinator

Margaret Mitchell (Johns Hopkins University, m.mitchell@jhu.edu)


NIST is an agency of the
U.S. Department of Commerce

privacy policy / security notice / accessibility statement
disclaimer
FOIA

Last updated: Tuesday, 28-Mar-2017 11:22:12 EDT
Comments to: tac-web@nist.gov