Information for TAC 2009 Track Participants

TAC 2009 has three tracks:

• Knowledge Base Population (KBP)
  Track Coordinator: Paul McNamee (paul.mcnamee@jhuapl.edu)
  Web page: http://pmcnamee.net/kbp.html
  Mailing list: tac-kbp@nist.gov
  
• Recognizing Textual Entailment (RTE)
  Track Coordinators: Danilo Giampiccolo (giampiccolo@celct.it) and Luisa Bentivogli (bentivo@fbk.eu)
  Web page: http://tac.nist.gov/2009/RTE/
  Mailing list: rte@nist.gov
  
• Summarization
  Track Coordinator: Hoa Trang Dang (hoa.dang@nist.gov)
  Web page: http://tac.nist.gov/2009/Summarization/
  Mailing list: duc_list@nist.gov
  

The exact definition of the tasks to be performed in each track for TAC 2009 is formulated and discussed on the track mailing list. Registering to participate in a TAC track does not automatically add you to the track's mailing list; if you are not already on the mailing list, you must subscribe yourself by following the instructions for contacting that mailing list as given in the track web page. If you were previously subscribed to the mailing list, you do not have to re-subscribe; track mailing lists are for anyone interested in the track, rather than specifically for TAC participants, and thus carry over from year to year. For questions about the track, send mail to the track coordinator (or post the question to the track mailing list once you join).

Each track has its own set of deadlines for submitting results, which will be given in the final track guidelines. Most results are due by very early August to give assessors time to judge the results. Tracks and tasks with smaller assessing constraints (such as the RTE Track) may have later deadlines. For most tracks, the judgments against which each system's output will be scored will be made by experienced assessors based on manual analysis and the output of all track participants.

Track participants must abide by the Policies and Guidelines for Track Participants. All results submitted to TAC are archived on the TAC web site and all evaluations are published in the conference proceedings. Dissemination of TAC work and results other than in the conference proceedings is welcomed, but the conditions of participation specifically preclude any advertising claims based on TAC results.

Data and User Agreements

Whenever possible, data that are required for the tracks are distributed through the track web pages; data are mailed as physical disks only if they cannot be made available for download. Test materials for most tracks are distributed by NIST at no cost to track participants. However, some tracks may use special collections that are available from other organizations for a fee; the individual track guidelines will specify instructions for obtaining such data.

Most data is kept in password-protected areas of the TAC web site. Password-protected areas include the current year's test data and evaluation results, and any data that is copyrighted. Access to TAC 2009 data is restricted to registered TAC 2009 participants who have submitted all the required User Agreement forms. Each password-protected area of the web site will indicate which signed User Agreements are required for access. Each participating team will provide a Team ID and receive a Team Password upon registration. Once the required User Agreement forms have been received for that team, NIST will activate the Team ID and Password to give access to the corresponding password-protected web pages. Teams that have participated in past TAC cycles must register and obtain a new password for 2009. Teams must also submit new (current) User Agreement forms for 2009.

Track participants should follow the instructions given in the User Agreements page for submitting the "Agreement Concerning Dissemination of TAC Results" and any additional User Agreements that are required for the particular tracks of interest. When submitting forms, in addition to the information requested in the User Agreements page, please include the email address of the main TAC 2009 contact person for the team, and a comment saying that the form is from a TAC 2009 registered participant. NIST needs a signed physical copy of the agreements before the team can be given any data.

Much of the past data from TREC, DUC and TAC are available from NIST and may be useful for system development for some of the TAC tracks. Beginning March 3, registered TAC participants may also request disks containing entire document collections that might be useful (but not required) for some of the tracks. These document disks consist of the TIPSTER set (disks 1-3) and the TREC set (disks 4-5). Participants should follow the instructions in the User Agreements page for submitting the "Agreement Concerning Dissemination of TAC Results" and any additional User Agreements that are required for the particular disks that are being requested. NIST will ship the CD-ROMs at no cost to participants who have returned all required forms. No disks will be shipped before March 3.

Schedule

TAC 2009 Schedule
Beginning March 3	Submit signed User Agreements to NIST
May 31	Deadline for registration for track participation
July - early September	Deadlines for results submission for most tracks
September 25	Deadline for workshop presentation proposals
September 25	Deadline for proposals for TAC 2010 tracks/tasks
By early October	Release of individual evaluated results to participants
October 7	Notification of acceptance of presentation proposals
October 22	Deadline for participants' notebook papers, copyright release forms
November 16-17	TAC 2009 workshop in Gaithersburg, Maryland, USA
February 25, 2010	Extended Deadline for participants' final proceedings papers

BACK to General Information about Tracks