
TAC 2011 Tracks
KBP
RTE
Summarization
Call for Participation
Reporting Guidelines
TAC 2011 Workshop

|
|
Second Call for Participation
Text Analysis Conference (TAC 2011)
Track Evaluations: February - October 2011
Workshop: November 14-15, 2011
** Track Registration Deadline extended to: June 10, 2011 **
Conducted by:
National Institute of Standards and Technology (NIST)
With support from:
U.S. Department of Defense
INTRODUCTION
The Text Analysis Conference (TAC) is a series of evaluations and
workshops organized to encourage research in Natural Language
Processing and related applications, by providing a large test
collection, common evaluation procedures, and a forum for
organizations to share their results. TAC comprises multiple tracks,
each of which focuses on a particular subproblem of NLP. TAC tracks
aim to improve end-user tasks, but also include diagnostic and
component evaluations situated within the context of end-user tasks.
You are invited to participate in TAC 2011. NIST will provide test
data for each track, and track participants will run their NLP systems
on the data and return their results to NIST for evaluation. TAC 2011
has three tracks:
- Knowledge Base Population (KBP)
- Recognizing Textual Entailment (RTE-7)
- Summarization
Organizations may choose to participate in any or all of the tracks.
The annual conference culminates in a November workshop at NIST in
Gaithersburg, Maryland, USA. All results submitted to NIST are
archived on the TAC web site, and all evaluations of submitted results
are included in the conference proceedings. Dissemination of TAC work
and results other than in the conference proceedings is welcomed, but
the conditions of participation specifically preclude any advertising
claims based on TAC results.
TRACKS and TASKS
TAC 2011 has three tracks:
- Knowledge Base Population Track (KBP)
Web page: http://nlp.cs.qc.cuny.edu/kbp/2011/ (archived here)
Track Coordinators:
- Heng Ji (hengji@cs.qc.cuny.edu)
- Ralph Grishman (grishman@cs.nyu.edu)
The goal of the KBP track is to promote research in automated
systems that discover information about named entities as found in a
large corpus and incorporate this information into a given knowlege
base (namely, a KB derived from Wikipedia). The KBP track comprises
the following tasks:
- Entity-Linking Task: Given a name (of a Person, Organization, or
Geopolitical Entity) and a document containing that name,
determine the KB node for the named entity, adding a new node for
the entity if it is not already in the KB. Two variants of the
entity-linking task are offered: English-only, and cross-lingual
(both English and Chinese documents).
- Slot-Filling Task: Given a named entity and a pre-defined set of
attributes ("slots") for the entity type, augment a KB node for
that entity by extracting all new learnable slot values for the
entity as found in a large corpus of documents.
- Temporal Slot-Filling Task: Same as the regular slot-filling task,
but also specify time intervals for each extracted slot value. In
addition to a full temporal slot-filling task, a diagnostic
temporal task is being offered, in which systems are provided with
documents and correct slot values and only have to specify the
temporal information.
- Recognizing Textual Entailment Track (RTE-7)
Web page: http://tac.nist.gov/2011/RTE/
Track Coordinators:
- Danilo Giampiccolo (giampiccolo@celct.it)
- Luisa Bentivogli (bentivo@fbk.eu)
The goal of the RTE Track is to develop systems that recognize when
one piece of text entails another. RTE-7 pursues the direction of
recognizing entailment in larger contexts -- a whole document or set
of documents. RTE-7 comprises the following tasks:
- Main and Novelty-Detection Tasks: Determine whether a given
sentence -- in the context of an entire document -- entails a
given Hypothesis.
- Knowledge Base Population Validation Task: Determine whether a
given document entails a given TAC KBP relation (e.g., "X is
married to Y").
- Summarization Track
Web page: http://tac.nist.gov/2011/Summarization/
Track Coordinators:
- Karolina Owczarzak (karolina.owczarzak@nist.gov)
- Hoa Trang Dang (hoa.dang@nist.gov)
Multiling Pilot Coordinators:
- George Giannakopoulos (ggianna@iit.demokritos.gr)
- Ilias Zavitsanos (izavits@iit.demokritos.gr)
The goal of the Summarization Track is to develop systems that
produce coherent summaries of text. The Summarization track
comprises the following tasks:
- Guided Summarization Task: Produce short, coherent summaries of
news articles falling into predefined categories, guided by
predefined aspects for each category.
- Automatically Evaluating Summaries of Peers (AESOP) Task:
Automatically score a summary for a given metric, including
content (Pyramid score), overall responsiveness, and overall
readabilty.
- Multiling Pilot: Develop and apply partially or fully language-independent
summarization algorithms to multiple languages, including Arabic,
Czech, English, French, Greek, Hindi, and Hebrew.
TRACK REGISTRATION
Organizations wishing to participate in any of the TAC 2011 tasks are
invited to register online by June 10, 2011 at:
Track participants are advised to register and submit all required
agreement forms as soon as possible in order to receive timely access
to track resources. Participants should also make sure they are
subscribed to the mailing lists for the tracks for which they are
registered (see instructions on the individual track home pages).
Registration for a track does not commit you to participating in the
track, but is helpful to know for planning. Late registration will be
permitted only if resources allow.
Any questions about conference participation may be sent to the TAC
project manager: tac-manager@nist.gov.
WORKSHOP
The TAC 2011 workshop will be held November 14-15, 2011, in
Gaithersburg, Maryland, USA. The TAC workshop is a forum both for
presentation of results (including failure analyses and system
comparisons), and for more lengthy system presentations describing
techniques used, experiments run on the data, and other issues of
interest to researchers in NLP.
SCHEDULE
TAC 2011 Schedule |
June 10 | Deadline for registration for track participation |
July - September | Deadlines for results submission (varies by track) |
By September 25 | Release of individual evaluated results to participants |
September 25 | Deadline for workshop presentation proposals |
October 12 | Notification of acceptance of presentation proposals |
October 25 | Deadline for system reports (workshop notebook version) |
November 14-15 | TAC 2011 workshop in Gaithersburg, Maryland, USA |
February 2012 | Deadline for system reports (final proceedings version) |
|