TAC 2008 Tracks

TAC 2008 has three tracks: Question Answering, Recognizing Textual Entailment, and Summarization. Organizations may choose to participate in any or all of the tracks. Each track is a continuation of an evaluation series previously organized under different frameworks, though specific tasks in a track may differ from previous years. Below is a brief summary of the tracks. The exact definition of the tasks to be performed in each track for TAC 2008 is formulated and discussed on the track mailing list. Applying to participate in a TAC track does not automatically add you to the track's mailing list; if you are not already on the mailing list, you must subscribe yourself by following the instructions for contacting that mailing list as given in the track web page. If you were previously subscribed to the mailing list, you do not have to re-subscribe; track mailing lists are for anyone interested in the problem, rather than specifically for TAC participants, and thus carry over from year to year. For questions about the track, send mail to the track coordinator (or post the question to the track mailing list once you join).

• Question Answering
  The goal of the QA Track is to develop systems that search large document collections to retrieve precise answers to questions, rather than entire documents.
  Track coordinator: Hoa Trang Dang, [email protected]
  Track web page: http://tac.nist.gov/2008/qa/
  Mailing list: [email protected]
  
• Recognizing Textual Entailment
  The goal of the RTE Track is to develop systems that recognize when one piece of text entails another.
  Track coordinator: Danilo Giampiccolo, [email protected]
  Track web page: http://tac.nist.gov/2008/rte/
  Mailing list: [email protected]
  
• Summarization
  The goal of the Summarization Track is to develop systems that produce short, coherent summaries of text.
  Track coordinator: Hoa Trang Dang, [email protected]
  Track web page: http://tac.nist.gov/2008/summarization/
  Mailing list: [email protected]
  

Each track has its own set of deadlines for submitting results, which are given in the final track guidelines. Results are generally due by very early August to give the assessors time to judge the results. Tracks and tasks with smaller assessing constraints (such as the RTE Track) may have later deadlines. For most tracks, the judgments against which each system's output are scored are made by experienced assessors based on manual analysis and the output of all track participants.

All results submitted to TAC are archived on the TAC web site and published in the (publicly available) conference proceedings. Dissemination of TAC work and results other than in the conference proceedings is welcomed, but the conditions of participation specifically preclude any advertising claims based on TAC results. Track participants must abide by the Policies and Guidelines for Track Participants.

Data Access and User Agreement Forms

Whenever possible, data that are required for the tracks are distributed through the track web pages; data are mailed as physical disks only if they cannot be made available for download. For most tracks, test (and any training) materials are distributed by NIST at no cost to track participants. However, some tracks may use special collections that are only available from other organizations for a fee; the individual track guidelines will specify instructions for obtaining such data.

Most data is kept in password-protected areas of the TAC web site. Password-protected areas include the current year's test data and evaluation results, and any data that is copyrighted. Access to TAC 2008 data and results is restricted to registered TAC 2008 participants who have submitted all the required User Agreement forms. Each password-protected area of the web site will indicate which signed User Agreements are required for access. Each participating team receives a Team ID and Team Password upon registration. Once the required User Agreement forms have been received for that team, NIST activates the Team ID and Password to give access to the corresponding password-protected web pages.

Track participants should follow the instructions given in the Data User Agreements page for submitting the "Agreement Concerning Dissemination of TAC Results" and any additional User Agreements that are required for the particular tracks of interest. When submitting forms, in addition to the information requested in the Data User Agreements page, please include the email address of the main TAC contact person for the team, and a comment saying that the form is from a TAC 2008 registered participant. NIST needs a signed physical copy of the agreements before the team can be given any data.

Supplementary Data

Much of the past data from TREC and DUC are available from NIST and may be useful for system development for some of the TAC tracks. The data include test topics/judgments (on the TREC and DUC web sites) and document disks. In addition, various collections of web documents that have been used in TREC are not available from NIST, but are available for a fee from the University of Glasgow (see http://ir.dcs.gla.ac.uk/test_collections/).

Beginning March 4, registered TAC participants may request disks containing entire document collections that might be useful (but are not required) for some of the tracks. The document disks that are available from NIST include the TIPSTER set (disks 1-3), the TREC set (disks 4-5), the AQUAINT set, and the AQUAINT-2 set. All documents are typical of those seen in a real-world situation (i.e., there will not be arcane vocabulary, but there may be missing pieces of text or typographical errors). Participants should follow the instructions in the Data User Agreements page for submitting the "Agreement Concerning Dissemination of TAC Results" and any additional User Agreements that are required for the particular disks that are being requested. NIST will ship the CD-ROMs free of charge to participants who have returned all required forms. No disks will be shipped before March 4.

Schedule

BACK to General Information about Tracks