TAC 2011 Knowledge Base Population (KBP2011) TrackEntity Linking Scoring by Javier Artiles (CUNY, javart@gmail.com)In KBP2011, we have a new requirement for entity linking, which is to cluster NIL queries. For a set of query names with source documents, an entity linking system is required to: (1) judge whether each query can be linked to any KB node; (2) Cluster all queries with NIL KB entries into clusters. Ultimately the system output can be viewed as a collection of various clusters; some clusters are labeled as KB node IDs. At the same time the answer key can also be viewed as a different collection of clusters. Therefore we will apply a modified B-Cubed metric (called B-Cubed+) to evaluate these clusters. Entity Linking Scorer: el_scorer.py v0.6
Temporal Slot Filling ScoringV0.3 Download: (revised by Qi Li, CUNY, liqiearth@gmail.com): handle V labels Archived V0.2: Download (by Ralph Grishman) Download2 (revised by Qi Li, CUNY, liqiearth@gmail.com) : added detailed break down scores for each slot type Slot Filling Scoring (by Ralph Grishman) In 2011 evaluation, a uniform scoring metric will be used, based on
traditional measures of recall, precision, and F-measure, computed from
counts of correct, missing, and spurious responses.
A non-NIL response is correct if it matches a
verified non-NIL entry in the key (the human assessment file);
other non-NIL responses are spurious. A
NIL response where the key has a verified non-NIL
response is considered missing. NIL system responses matching verified
NIL entries in the key are not counted. For
single-valued slots only a single system response will be accepted. For list-valued slots, the verified non-NIL
responses will be grouped into equivalence classes.
Multiple responses to a query must come from
disjoint classes to be counted as correct; other
responses
are
counted as spurious. To run, download SFScore.java (SFScore.java V1.2 ; SFScore.java V0.9) javac SFScore.java
java SFScore response-file key-file [flags ...] where the possible flags are trace -- print a line
with
assessment of each
system response
anydoc -- judge response based only on answer string, ignoring doc id nocase -- ignore case in matching answer string slots=slotfile -- take list of entityId:slot pairs from slotfile (otherwise list of pairs is taken from system response)
The slotfile controls which slots are evaluated; if
you want your system evaluated on all slots for which you generate an
output, the "slots" parameter is not needed. In that case it is
important for your system to generate explicit NILs for slots it cannot
fill. As the key file, you can use one of
the newly produced annotation files, or you can run UpdateSFKey (see
the tools page)
on the 2009 judgments file. The anydoc and nocase flags are
designed to make the scorer more useful for development by supporting
soft match, but will not be used for official scoring.
|