TAC 2008 Tracks
QA
RTE
Summarization
Call for Participation
Reporting Guidelines
TAC 2008 Workshop
|
|
TAC 2008: Call for Participation
Track Evaluations: February - October 2008
Workshop: November 17-19, 2008
With support from:
Intelligence Advanced Research Projects Activity (IARPA)
The Text Analysis Conference (TAC) is a new series of evaluation
workshops organized to encourage research in Natural Language
Processing and related applications, by providing a large test
collection, common evaluation procedures, and a forum for organizations
to share their results. TAC grew out of the Document Understanding Conference (DUC)
and the Question Answering Track of the Text REtrieval Conference (TREC). TAC
comprises a set of tasks known as "tracks," each of which focuses on a
particular subproblem of NLP.
You are invited to participate in TAC 2008. NIST will provide
test data for each track, and track participants will run their NLP
systems on the data and return their results to NIST for evaluation.
Organizations may choose to participate in any or all of the tracks.
The annual conference culminates in a November workshop at NIST in
Gaithersburg, Maryland. Dissemination of TAC work and results other
than in the (publicly available) conference proceedings is welcomed,
but the conditions of participation specifically preclude any
advertising claims based on TAC results. All results submitted to
NIST are published in the proceedings and are archived on the TAC web
site.
Task Description
TAC 2008 will have three tracks: Question Answering, Recognizing
Textual Entailment, and Summarization.
Each track is a continuation
of an evaluation series previously organized under different
frameworks, though specific tasks in a track may differ from previous
years.
Below is a brief summary of the tracks. The exact definition of the
tasks to be performed in each TAC 2008 track is still being
formulated. Track discussion takes place on the track mailing list.
To be added to a track mailing list, follow the instructions given
in the track web page for contacting the mailing list. For questions about the
track, send mail to the track coordinator (or post the question to the
track mailing list once you join).
- Question Answering
The goal of the QA Track is to develop systems that search large document collections to retrieve precise answers to questions, rather than entire documents.
Track coordinator: Hoa Trang Dang, hoa.dang@nist.gov
Track web page: http://tac.nist.gov/2008/qa/
Mailing list: trec-qa@nist.gov
- Recognizing Textual Entailment
The goal of the RTE Track is to develop systems that recognize when one piece of text entails another.
Track coordinator: Danilo Giampiccolo, giampiccolo@celct.it
Track web page: http://tac.nist.gov/2008/rte/
Mailing list: rte@nist.gov
- Summarization
The goal of the Summarization Track is to develop systems that produce short, coherent summaries of text.
Track coordinator: Hoa Trang Dang, hoa.dang@nist.gov
Track web page: http://tac.nist.gov/2008/summarization/
Mailing list: duc_list@nist.gov
Conference Format
TAC 2008 culminates in a November workshop that is used as a forum
both for presentation of results (including failure analyses and
system comparisons), and for more lengthy system presentations
describing techniques used, experiments run on the data, and other
issues of interest to researchers in NLP. Track participants
interested in giving a presentation during the workshop should submit
a 500-word abstract in September describing the experiments they
performed. As there is a limited amount of time for oral
presentations, the TAC advisory committee will use the abstracts to
determine which participants are asked to speak and which will present
in a poster session.
Data
For most tracks, training and test materials are available from NIST;
some tracks may use special collections that are available from other
organizations for a fee. Many of the existing TREC/DUC
data (documents, topics, and judgments) are available for
training purposes and may also be used in some of the tracks.
All documents are typical of those seen in a real-world situation
(i.e., there will not be arcane vocabulary, but there may be
missing pieces of text or typographical errors). For most tracks,
the judgments against which each system's output will be
scored will be made by experienced assessors based on manual analysis and
the pooled output of all track participants.
Registration for TAC 2008 Tracks
Organizations wishing to participate in any of the TAC 2008 tracks
should register
online by March 4, 2008. Registration for a track does not
commit you to participating in the track, but is helpful to know for
planning. Late registration will be permitted only if resources
allow. Any questions about conference participation should be sent to
the TAC project manager: tac-manager@nist.gov.
Schedule
Tentative TAC 2008 Schedule |
March 4 | Deadline for registration for track participation |
Beginning March 4 | Submit signed User Agreements to NIST |
mid June - early September | Deadlines for results submission for most tracks |
September 16 (estimated) | Deadline for workshop presentation proposals |
September 16 (estimated) | Deadline for proposals for future tracks/tasks |
By early October | Release of individual evaluated results to participants |
mid October | Deadline for participants' notebook papers |
November 17-19 | TAC 2008 workshop in Gaithersburg, Maryland, USA |
early February 2009 | Deadline for participants' final proceedings papers |
Advisory Committee
John Burger, MITRE
John Conroy, IDA/CCS
Ido Dagan, Bar Ilan University
Hoa Trang Dang, NIST (chair)
Maarten de Rijke, University of Amsterdam
Bonnie Dorr, University of Maryland
Donna Harman, NIST
Andy Hickl, Language Computer Corporation
Ed Hovy, ISI/USC
Boris Katz, Massachusetts Institute of Technology
Bernardo Magnini, ITC-irst
Kathy McKeown, Columbia University
Ani Nenkova, University of Pennsylvania
Drago Radev, University of Michigan
Lucy Vanderwende, Microsoft Research
Ellen Voorhees, NIST
Ralph Weischedel, BBN
|