TAC 2008 Question Answering Track

• Track coordinator: Hoa Trang Dang, hoa.dang@nist.gov
• Opinion QA Task Guidelines
• Opinion QA Test Data (Available as Past TAC Data)
• Opinion QA Evaluation Results (Available as Past TAC Data)

The goal of the QA Track is to develop systems that search large document collections and retrieve precise answers to questions (rather than entire documents). The TAC QA track evolved from the TREC QA track. Please visit the TREC website for additional information about Question Answering at TREC, including an open archive of TREC QA data and instructions on how to obtain additional detailed Past TREC Results for individual submitted runs.

The 2008 QA Track will focus on finding answers to short series of opinion questions, where each series of 2-4 questions will be about a particular target. There will be two types of questions, each with its own evaluation metric.

1. rigid lists:
• given a question such as:
• "Name US senators who support tax reform."
• "Which countries would like to build nuclear power plants?"
• "Which rock bands do college students like?"
• return: exact strings containing a list item
• evaluation: F-measure combining precision and recall
2. squishy lists:
• given a question such as:
• "What criticisms do US senators have against the current tax system?"
• "Why do countries want to have nuclear power plants?"
• "What do people like about Ikea?"
• return: strings containing an answer to the question
• evaluation: nugget pyramid evaluation used for "Other" questions in the TREC 2006-2007 QA track. NIST may experiment with and report on various additional nugget-based methods of scoring.

Answers must be retrieved from documents in the Blog06 corpus. Participants should obtain the Blog06 collection directly from the University of Glasgow. (See http://ir.dcs.gla.ac.uk/test_collections/.) As a service to those participants who do not wish to run their own search engines over the entire Blog06 corpus, NIST will distribute the top 50 documents for each target (as retrieved by a Lucene-based search engine) at not cost to track participants; participants wishing access to these top documents must submit the Blog06 User Agreement to NIST (even if the group has already submitted the agreement to the University of Glasgow to obtain the entire collection).

To improve the quality and completeness of the pool of answers in the QA track, NIST will accept manual runs in addition to fully automatic ones. A submission is considered to be a manual run if it involves any human intervention in response to the particular test data, whether it is simply to put the questions into a canonical form for an otherwise fully automatic QA system, or whether it entails a person performing the entire question answering task. At the time of submission, participants must fully describe the extent and nature of any manual intervention. Participants who submit a manual run must also submit a fully automatic run.

Joint Blog Task The TAC 2008 QA task is part of a larger information retrieval, analysis, and synthesis task involving the TREC Blog Track and the TAC Summarization Track. A subset of the targets from the QA Track will be the basis for TREC Blog Track topics, where the task in TREC will be to retrieve documents containing opinions about the target; the TREC Blog Track homepage contains additional information about the document retrieval task in the Blog Track. To support an additional level of text analysis, the answer strings and supporting documents that are returned by TAC QA participants will be given to TAC Summarization participants, who will create fluent summaries of the answers to the questions from the QA track.

Mailing List

The mailing list for the Question Answering Track is trec-qa@nist.gov. The list is used to discuss and define the task guidelines for the track, as well as for general discussion related to QA and its evaluation. To subscribe, send a message to listproc@email.nist.gov such that the body consists of the line:
            subscribe trec-qa <FirstName> <LastName>
In order for your messages to get posted to the list, you must send them from the email address used when you subscribed to the list. For additional information on how to use mailing lists hosted at NIST, send a message to listproc@email.nist.gov such that the body consists of the line:
            HELP

Schedule

BACK to All TAC 2008 Tracks