PASCAL Recognizing Textual Entailment Challenge (RTE-5) at TAC 2009
Introduction
Given two text fragments called 'Text' and 'Hypothesis', Textual
Entailment Recognition is the task of determining whether the
meaning of the Hypothesis is entailed (can be inferred) from the Text.
The goal of the first RTE Challenge was to provide the NLP community
with a benchmark to test progress in recognizing textual entailment,
and to compare the achievements of different groups. Since its
inception in 2004, the PASCAL RTE Challenges have promoted research in
textual entailment recognition as a generic task that captures major
semantic inference needs across many natural language processing
applications, such as Question Answering (QA), Information Retrieval
(IR), Information Extraction (IE), and multi-document summarization.
After the first three highly successful PASCAL RTE Challenges, RTE
became a track at the 2008 Text Analysis Conference, which brought it
together with communities working on NLP applications (specifically,
QA and summarization). The interaction has provided the opportunity
to apply RTE systems to specific applications and to move the RTE task
towards more realistic application scenarios.
What's new in RTE-5
The main RTE-5 task is similar to the RTE-4 task, with the following changes:
- The average length of the Texts will be higher.
- Texts will come from a variety of sources and will not be edited from their source documents. Thus, systems will be asked to handle real text that may include typographical errors and ungrammatical sentences.
- A development set will be released.
- The textual entailment recognition task would be based on only three application settings:
QA, IE, and IR.
In addition to the main task (Textual Entailment Recognition),
RTE-5 will offer a new Textual Entailment Search pilot
that is situated in the summarization application setting, where the
task would be to find all Texts in a set of documents that entail a
given Hypothesis.
Schedule
RTE-5 Schedule |
April 3 | Release of Development Set (Search Pilot) |
May 29 | Release of Development Set (Main Task) |
May 31 | Deadline for TAC 2009 track registration |
September 2 | Release of Test Set (Main Task, Search Pilot) |
September 9 | Deadline for task submissions (Main Task, Search Pilot) |
September 18 | Release of individual evaluated results (Main Task, Search Pilot) |
September 25 | Deadline for TAC 2009 workshop presentation proposals |
October 22 | Deadline for systems' reports |
Mailing List
The mailing list for the RTE Track is [email protected]. The list is used to discuss and define the task guidelines for the track, as well as for general discussion related to textual entailment and its evaluation. To subscribe, send a message to [email protected] such that the body consists of the line:
subscribe rte <FirstName> <LastName>
In order for your messages to get posted to the list, you must send
them from the email address used when you subscribed to the list.
To unsubscribe, send a message from the subscribed email address to [email protected] such that the body consists of the line:
unsubscribe rte
For additional information on how to use mailing lists hosted at NIST,
send a message to [email protected] such
that the body consists of the line:
HELP
Organizing Committee
Luisa Bentivogli, CELCT and FBK, Italy
Ido Dagan, Bar Ilan University, Israel
Hoa Trang Dang, NIST, USA
Danilo Giampiccolo, CELCT, Italy
Bernardo Magnini, FBK, Italy