TAC 2008 Summarization Track

• Track coordinator: Hoa Trang Dang, hoa.dang@nist.gov


• Update Summarization Task Guidelines
• Update Summarization Test Data:
• Test topic statements (Available as Past TAC Data)
• Test document sets (Available as Past TAC Data)
• Update Summarization Evaluation Results (Available as Past TAC Data)
• Update Summarization Evaluation: "summary" Excel sheets (Available as Past TAC Data)


• Opinion Summarization Pilot Guidelines
• Opinion Summarization Test Data:
• Test targets/questions/documents (Available as Past TAC Data)
• Test document sets (Available as Past TAC Data)
• Optional answer-snippets (Available as Past TAC Data)
• Opinion Summarization Evaluation Results (Available as Past TAC Data)

The goal of the Summarization Track is to develop systems that produce short, coherent summaries of text. The TAC Summarization Track continues the efforts of the Document Understanding Conference; please visit the DUC website for additional information about past summarization tasks and results.

1. Update Task: Piloted in DUC 2007, the update summarization task is to write a short (~ 100-word) summary of a set of newswire articles, under the assumption that the user has already read a given set of earlier articles. The summaries will be evaluated for readability and content (based on Columbia University's Pyramid Method). Test documents will come from the AQUAINT-2 collection, and task participants must submit the AQUAINT-2 User Agreement to NIST.
2. Opinion Pilot: The pilot task will be to write summaries of opinions from blogs. Participants will be given the questions from the TAC QA Track and the text snippets output by QA systems, and will produce short coherent summaries of the answers to the questions, either from the text snippets themselves, or from the associated documents. Summaries will be assessed for both readability and content. Evaluation of content will be based on the Nugget Pyramid Method used in the QA Track. Test documents will come from the Blog06 collection, and task participants must submit the Blog06 User Agreement to NIST.

It is anticipated that NIST will distribute all required test data (including documents for summarization) at no cost to track participants via password-protected web pages. Participants in the Opinion Summarization Pilot should subscribe to the mailing lists for both the Question Answering and the Summarization Tracks.

Mailing List

The mailing list for the Summarization Track is duc_list@nist.gov. The list is used to discuss and define the task guidelines for the track, as well as for general discussion related to summarization and its evaluation. To subscribe, send a message to listproc@email.nist.gov such that the body consists of the line:
            subscribe duc_list <FirstName> <LastName>
In order for your messages to get posted to the list, you must send them from the email address used when you subscribed to the list. For additional information on how to use mailing lists hosted at NIST, send a message to listproc@email.nist.gov such that the body consists of the line:
            HELP

Schedule

BACK to All TAC 2008 Tracks