Overview

There has been a growing recognition of the importance of community-wide evaluations for research in information technologies. The Text Analysis Conference is a series of workshops that provides the infrastructure for large-scale evaluation of Natural Language Processing technology.

Mission

TAC's mission is to support research within the Natural Language Processing community by providing the infrastructure necessary for large-scale evaluation of NLP methodologies. TAC's primary purpose is not competitive benchmarking; the emphasis is on advancing the state of the art through evaluation results. In particular, the TAC workshop series has the following goals:

• to promote research in NLP based on large common test collections;
• to improve evaluation methodologies and measures for NLP;
• to build a series of test collections that evolve to anticipate the evaluation needs of modern NLP systems;
• to increase communication among industry, academia, and government by creating an open forum for the exchange of research ideas;
• to speed the transfer of technology from research labs into commercial products by demonstrating substantial improvements in NLP methodologies on real-world problems.

Organization

TAC is organized by the Retrieval Group of the Information Access Division (IAD) in the Information Technology Laboratory at the National Institute of Standards and Technology (NIST). Initiated in 2008, TAC grew out of NIST's Document Understanding Conference (DUC) for text summarization, and the Question Answering Track of the Text Retrieval Conference (TREC). TAC is sponsored by NIST and other U.S. government agencies and is overseen by an Advisory Committee consisting of representatives from government, industry, and academia.

TAC Cycle

A TAC cycle consists of a set tracks, areas of focus in which particular NLP tasks are defined. The tracks serve several purposes. First, tracks act as incubators to experiment with new research areas; the first running of a track often defines what the problem really is, and a track creates the necessary infrastructure (test collections, evaluation methodology, etc.) to support research on its tasks. The tracks also demonstrate the robustness of core NLP technology in that the same techniques are frequently appropriate for a variety of tasks. Finally, the tracks make TAC attractive to a broader community by providing tasks that match the research interests of more groups. The TAC advisory committee selects the set of tracks that will be run in a given year of TAC based on track proposals.

The TAC cycle culminates with a workshop that is a forum for participants to share their experiences and plan future tasks/evaluations. The workshop is usually held at NIST during the last full week before the U.S. Thanksgiving holiday. Attendance at TAC workshops is open to the public, but workshop attendees must pre-register in order to enter the NIST campus.

Additional Information

• TAC Advisory Committee
• TAC Policies and Guidelines for Track Participants