TAC Policies and Guidelines for Track Participants

TAC Policies

1. Dissemination of TAC results is permitted only in accordance with the "Agreement Concerning Dissemination of TAC Results". To participate in TAC, you must abide by the conditions set forth in the Agreement. The use of TAC results (or the sponsors' names) in advertising is not only contrary to the spirit of TAC, but is likely to result in difficulties for the sponsoring agencies and jeopardize the existence of the conference. If you feel that your organization cannot adhere to these guidelines, then you should withdraw from TAC immediately.
2. If you submit results to any tracks, they may not subsequently be withdrawn. All results that are submitted to NIST will be evaluated in the conference proceedings and archived on the TAC web site.
3. Each team that submits results is required to write a paper for the proceedings that describes how the runs were produced (to the extent that intellectual property concerns allow). Failure to provide a paper for the proceedings may result in the team members being denied participation in future TAC evaluations. A draft version of the proceedings papers is distributed as a notebook to workshop attendees. The actual (final) proceedings are published on the TAC web site after the workshop. Having a notebook version of papers gives workshop attendees something concrete to discuss during the meeting itself, while the final proceedings give participants the opportunity to revise their papers based on findings from workshop discussion.
4. Unless specifically invited by the TAC Advisory Committee, only teams that submit results for evaluation may contribute a paper for the proceedings. TAC proceedings papers must be on the subject of the participant's track submission(s); the TAC Advisory Committee reserves the right to exclude non-responsive papers. However, papers do not go through any other review process. Participants are encouraged to expand on the work reported in their TAC proceedings papers and submit the results to peer-reviewed conferences, journals, and other venues.
5. Participants must abide by the guidelines for producing results. In general, teams are expected to do no training or customizing of their systems with respect to the current year's test data.
6. Participants must not share their Team Passwords with others outside the team. The primary purpose of passwords is to protect the copyrights of data providers, and unauthorized access to data jeopordizes NIST's ability to obtain future data for TAC.
7. Registration to participate in a TAC track does not obligate a team to submit results for that track. However, if a team has not submitted any results once the last TAC evaluation window has closed for the year, the team's TAC password for that year will be deactivated, and the team will not be able to obtain additional resources such as gold standard annotations, assessments, and evaluation results for the test data until such resources are made available as training data to registered TAC participants in subsequent evaluation years. Teams are discouraged from registering for a TAC track if they do not intend to submit any results for evaluation; repeatedly registering without submitting any results for TAC may result in the team members being denied participation in future TAC evaluations.

TAC Guidelines

Each track has its own set of guidelines. The guidelines give a precise definition of the task(s) to be performed, a due date for the results, and any special instructions for that track. Guidelines are generally posted on the track web page in the spring. In addition, there are some generic guidelines that tracks assume unless explicitly stated otherwise.

1. System data structures (such as dictionaries, indices, thesauri, etc., whether constructed by hand or automatically) can be built using existing documents and past topics and judgments, but these structures may not be modified in response to the new test topics. For example, you can't add topic words that are not in your dictionary, nor may the system data structures be based in any way on the results of retrieving documents for the test topics and having a human look at the retrieved documents. Most of the tasks in TAC represent the real-world problem of an ordinary user seeking information. If an ordinary user couldn't make the change to the system, you should not make it after receiving the test topics. A corollary of this rule is that your system may not be tuned to the new TAC topics.
2. Each track will specify the format of submitting results for its tasks. NIST will release a routine that checks for common errors in the result files including invalid document numbers and improper file formats. This routine will be made available to participants to check their runs for errors prior to submitting them. Submitting runs is an automatic process done through a web form, and runs that contain errors cannot be processed.