TAC 2008: Call for Participation

Conducted by:
National Institute of Standards and Technology (NIST)

With support from:
Intelligence Advanced Research Projects Activity (IARPA)

The Text Analysis Conference (TAC) is a new series of evaluation workshops organized to encourage research in Natural Language Processing and related applications, by providing a large test collection, common evaluation procedures, and a forum for organizations to share their results. TAC grew out of the Document Understanding Conference (DUC) and the Question Answering Track of the Text REtrieval Conference (TREC). TAC comprises a set of tasks known as "tracks," each of which focuses on a particular subproblem of NLP.

You are invited to participate in TAC 2008. NIST will provide test data for each track, and track participants will run their NLP systems on the data and return their results to NIST for evaluation. Organizations may choose to participate in any or all of the tracks. The annual conference culminates in a November workshop at NIST in Gaithersburg, Maryland. Dissemination of TAC work and results other than in the (publicly available) conference proceedings is welcomed, but the conditions of participation specifically preclude any advertising claims based on TAC results. All results submitted to NIST are published in the proceedings and are archived on the TAC web site.

Task Description

TAC 2008 will have three tracks: Question Answering, Recognizing Textual Entailment, and Summarization. Each track is a continuation of an evaluation series previously organized under different frameworks, though specific tasks in a track may differ from previous years.

Below is a brief summary of the tracks. The exact definition of the tasks to be performed in each TAC 2008 track is still being formulated. Track discussion takes place on the track mailing list. To be added to a track mailing list, follow the instructions given in the track web page for contacting the mailing list. For questions about the track, send mail to the track coordinator (or post the question to the track mailing list once you join).

• Question Answering
  The goal of the QA Track is to develop systems that search large document collections to retrieve precise answers to questions, rather than entire documents.
  Track coordinator: Hoa Trang Dang, hoa.dang@nist.gov
  Track web page: http://tac.nist.gov/2008/qa/
  Mailing list: trec-qa@nist.gov
  
• Recognizing Textual Entailment
  The goal of the RTE Track is to develop systems that recognize when one piece of text entails another.
  Track coordinator: Danilo Giampiccolo, giampiccolo@celct.it
  Track web page: http://tac.nist.gov/2008/rte/
  Mailing list: rte@nist.gov
  
• Summarization
  The goal of the Summarization Track is to develop systems that produce short, coherent summaries of text.
  Track coordinator: Hoa Trang Dang, hoa.dang@nist.gov
  Track web page: http://tac.nist.gov/2008/summarization/
  Mailing list: duc_list@nist.gov
  

Conference Format

TAC 2008 culminates in a November workshop that is used as a forum both for presentation of results (including failure analyses and system comparisons), and for more lengthy system presentations describing techniques used, experiments run on the data, and other issues of interest to researchers in NLP. Track participants interested in giving a presentation during the workshop should submit a 500-word abstract in September describing the experiments they performed. As there is a limited amount of time for oral presentations, the TAC advisory committee will use the abstracts to determine which participants are asked to speak and which will present in a poster session.

Data

For most tracks, training and test materials are available from NIST; some tracks may use special collections that are available from other organizations for a fee. Many of the existing TREC/DUC data (documents, topics, and judgments) are available for training purposes and may also be used in some of the tracks.

All documents are typical of those seen in a real-world situation (i.e., there will not be arcane vocabulary, but there may be missing pieces of text or typographical errors). For most tracks, the judgments against which each system's output will be scored will be made by experienced assessors based on manual analysis and the pooled output of all track participants.

Registration for TAC 2008 Tracks

Organizations wishing to participate in any of the TAC 2008 tracks should register online by March 4, 2008. Registration for a track does not commit you to participating in the track, but is helpful to know for planning. Late registration will be permitted only if resources allow. Any questions about conference participation should be sent to the TAC project manager: tac-manager@nist.gov.

• Registration to Participate in TAC 2008 Tracks

Schedule

Tentative TAC 2008 Schedule
March 4	Deadline for registration for track participation
Beginning March 4	Submit signed User Agreements to NIST
mid June - early September	Deadlines for results submission for most tracks
September 16 (estimated)	Deadline for workshop presentation proposals
September 16 (estimated)	Deadline for proposals for future tracks/tasks
By early October	Release of individual evaluated results to participants
mid October	Deadline for participants' notebook papers
November 17-19	TAC 2008 workshop in Gaithersburg, Maryland, USA
early February 2009	Deadline for participants' final proceedings papers

Advisory Committee

John Burger, MITRE
John Conroy, IDA/CCS
Ido Dagan, Bar Ilan University
Hoa Trang Dang, NIST (chair)
Maarten de Rijke, University of Amsterdam
Bonnie Dorr, University of Maryland
Donna Harman, NIST
Andy Hickl, Language Computer Corporation
Ed Hovy, ISI/USC
Boris Katz, Massachusetts Institute of Technology
Bernardo Magnini, ITC-irst
Kathy McKeown, Columbia University
Ani Nenkova, University of Pennsylvania
Drago Radev, University of Michigan
Lucy Vanderwende, Microsoft Research
Ellen Voorhees, NIST
Ralph Weischedel, BBN