PASCAL Recognizing Textual Entailment Challenge (RTE-5) at TAC 2009

• Track coordinators:
  • Luisa Bentivogli, CELCT and FBK, Italy (bentivo@fbk.eu)
  • Danilo Giampiccolo, CELCT, Italy (giampiccolo@celct.it)
• RTE-5 Call for Participation
• Past RTE Data
• RTE Resource Pool


• RTE-5 Main Task Guidelines
• RTE-5 Main Task Development Set (Available as Past TAC Data)
• RTE-5 Main Task Test Set (Available as Past TAC Data)
• RTE-5 Main Task Annotated Test Set (Available as Past TAC Data)
• RTE-5 Main Task Runs (Available as Past TAC Data)


• RTE-5 Search Pilot Guidelines
• RTE-5 Pilot Development Set (Available as Past TAC Data)
• RTE-5 Pilot Test Set (Available as Past TAC Data)
• RTE-5 Pilot Annotated Test Set (Available as Past TAC Data)

Introduction

Given two text fragments called 'Text' and 'Hypothesis', Textual Entailment Recognition is the task of determining whether the meaning of the Hypothesis is entailed (can be inferred) from the Text. The goal of the first RTE Challenge was to provide the NLP community with a benchmark to test progress in recognizing textual entailment, and to compare the achievements of different groups. Since its inception in 2004, the PASCAL RTE Challenges have promoted research in textual entailment recognition as a generic task that captures major semantic inference needs across many natural language processing applications, such as Question Answering (QA), Information Retrieval (IR), Information Extraction (IE), and multi-document summarization.

After the first three highly successful PASCAL RTE Challenges, RTE became a track at the 2008 Text Analysis Conference, which brought it together with communities working on NLP applications (specifically, QA and summarization). The interaction has provided the opportunity to apply RTE systems to specific applications and to move the RTE task towards more realistic application scenarios.

What's new in RTE-5

The main RTE-5 task is similar to the RTE-4 task, with the following changes:

• The average length of the Texts will be higher.
• Texts will come from a variety of sources and will not be edited from their source documents. Thus, systems will be asked to handle real text that may include typographical errors and ungrammatical sentences.
• A development set will be released.
• The textual entailment recognition task would be based on only three application settings: QA, IE, and IR.

In addition to the main task (Textual Entailment Recognition), RTE-5 will offer a new Textual Entailment Search pilot that is situated in the summarization application setting, where the task would be to find all Texts in a set of documents that entail a given Hypothesis.

Schedule

Mailing List

The mailing list for the RTE Track is rte@nist.gov. The list is used to discuss and define the task guidelines for the track, as well as for general discussion related to textual entailment and its evaluation. To subscribe, send a message to listproc@email.nist.gov such that the body consists of the line:
            subscribe rte <FirstName> <LastName>
In order for your messages to get posted to the list, you must send them from the email address used when you subscribed to the list. To unsubscribe, send a message from the subscribed email address to listproc@email.nist.gov such that the body consists of the line:
            unsubscribe rte
For additional information on how to use mailing lists hosted at NIST, send a message to listproc@email.nist.gov such that the body consists of the line:
            HELP

Organizing Committee

Luisa Bentivogli, CELCT and FBK, Italy
Ido Dagan, Bar Ilan University, Israel
Hoa Trang Dang, NIST, USA
Danilo Giampiccolo, CELCT, Italy
Bernardo Magnini, FBK, Italy

BACK to Information for All TAC 2009 Track Participants