===================================================================== TAC KBP 2015 EVENT ARGUMENT EXTRACTION AND LINKING EVALUATION RESULTS ===================================================================== Team ID: IHMC Organization: Florida Institute for Human and Machine Cognition Run ID: IHMC1 Did the run access the live Web during the evaluation window: No Did the run perform any cross-sentence reasoning: No Did the run use any distributed representations (e.g., of words): No Did the run return meaningful confidence values: No Run ID: IHMC2 Did the run access the live Web during the evaluation window: No Did the run perform any cross-sentence reasoning: No Did the run use any distributed representations (e.g., of words): No Did the run return meaningful confidence values: No Run ID: IHMC3 Did the run access the live Web during the evaluation window: No Did the run perform any cross-sentence reasoning: No Did the run use any distributed representations (e.g., of words): No Did the run return meaningful confidence values: No ************************************************************* ### The following are scores for from the TAC 2015 Event Argument and Linking Evaluation. ### For all scoring breakdowns, the summaries report: Precision, Recall, F1, EAArg Score, and Overall score. ### Details of the scoring and the scoring software can be found on the TAC 2015 EAL webpage. ### ### Scores are reported on the full data set (all_genre) and broken down by genre-- discussion forum only(df) newswire only(nw). ### ### The official score (withRealis) incorporates the correctness of the (ACTUAL, GENERIC, and OTHER) distinction ### and the correctness of canonical argument string resolution. As a diagnostic, we also report (a) a score ### that ignores the realis distinction (neutralizeRealis) and (b) a score that ignores both the realis distinction ### and canonical argument string resolution(neutraliseRealisCoref). ### ### Scores are reported over two data sets. Dataset1 (all_event_types), consists of 81 documents assessed for the ### full TAC EAL event taxonomy as specified in the 2015 evaluation plan. Dataset 2(restricted_event_types), ### consists of 201 documents assessed for only 6 event types (assertions outside of the 6 were ignored). Dataset2 ### includes the documents in Dataset1. Dataset 2 was assessed to allow a more in depth evaluation of event-specific ### performance (and variance across performance by event type). The 6 event types included in Dataset2 are: ### - Transaction.Transfer-Money ### - Movement.Transport-Artifact ### - Life.Marry ### - Contact.Meet ### - Conflict.Demonstrate ### - Conflict.Attack ### ### One participant (ZJU) submitted an submission an offset error. This system output was automatically fixed by BBN (the organizer) and ### the system by ZJU (the participant). Because the modifications were different, both numbers are reported. ### ### One participant (ver-CMU) participated in "verification" version of the task. This system took as its input all ### other system submissions. This submission included the ZJU submission which had broken offsets and ### did not include either BBN's fix or ZJU's fix. Thus it is not comparable to the other systems in task performed. ### ### The LDC submission was produced with an LDC annotator spending 45-60 minutes on the task of extracting arguments ### and grouping them. The low recall of the LDC submission is due at least in part to the time limitation. ### ### While all scores provide interesting diagnostic information, the "official" evaluation metric is Dataset1(all_event_types) on the ### both genres (all_genre) using the official(withRealis) metric. #################################### ###### All Event Types ###### ####### Genre: all_genre ####### ##### Scoring Configuration: withRealis ##### submission P R F1 EAArg EALink Overall IHMC1 10.0 12.9 11.3 1.3 4.6 2.9 IHMC2 6.8 9.9 8.1 0.8 3.6 2.2 IHMC3 6.4 13.3 8.6 0.7 4.4 2.6 ##### Scoring Configuration: neutralizeRealisCoref ##### submission P R F1 EAArg EALink Overall IHMC1 25.1 21.7 23.3 9.3 10.8 10.0 IHMC2 14.0 14.7 14.3 2.6 6.0 4.3 IHMC3 13.6 20.5 16.4 2.6 8.7 5.7 ##### Scoring Configuration: neutralizeRealis ##### submission P R F1 EAArg EALink Overall IHMC1 15.4 14.0 14.7 2.2 5.1 3.6 IHMC2 10.3 11.0 10.6 1.3 4.0 2.7 IHMC3 9.4 14.4 11.4 1.2 4.8 3.0 ####### Genre: df ####### ##### Scoring Configuration: withRealis ##### submission P R F1 EAArg EALink Overall IHMC1 9.9 13.9 11.6 2.3 5.1 3.7 IHMC2 7.3 11.4 8.9 2.2 4.6 3.4 IHMC3 6.6 14.9 9.1 1.9 6.0 4.0 ##### Scoring Configuration: neutralizeRealisCoref ##### submission P R F1 EAArg EALink Overall IHMC1 25.8 22.9 24.3 10.8 12.3 11.5 IHMC2 15.8 16.3 16.0 5.7 6.9 6.3 IHMC3 14.4 21.8 17.3 5.5 10.6 8.0 ##### Scoring Configuration: neutralizeRealis ##### submission P R F1 EAArg EALink Overall IHMC1 16.6 15.5 16.0 3.8 5.7 4.7 IHMC2 12.3 12.9 12.6 3.3 5.1 4.2 IHMC3 10.6 16.4 12.9 3.2 6.5 4.8 ####### Genre: nw ####### ##### Scoring Configuration: withRealis ##### submission P R F1 EAArg EALink Overall IHMC1 10.1 12.2 11.1 0.6 4.3 2.4 IHMC2 6.5 9.0 7.5 0.0 3.0 1.5 IHMC3 6.3 12.4 8.4 0.0 3.5 1.7 ##### Scoring Configuration: neutralizeRealisCoref ##### submission P R F1 EAArg EALink Overall IHMC1 24.6 20.9 22.6 8.4 9.8 9.1 IHMC2 13.0 13.8 13.4 0.8 5.5 3.1 IHMC3 13.1 19.6 15.7 0.9 7.6 4.2 ##### Scoring Configuration: neutralizeRealis ##### submission P R F1 EAArg EALink Overall IHMC1 14.7 13.1 13.9 1.3 4.7 3.0 IHMC2 9.1 9.8 9.4 0.1 3.4 1.7 IHMC3 8.7 13.2 10.5 0.0 3.8 1.9 #################################### ###### Restricted Event Types ###### ####### Genre: all_genre ####### ##### Scoring Configuration: withRealis ##### submission P R F1 EAArg EALink Overall IHMC1 10.3 13.3 11.6 3.3 4.6 3.9 IHMC2 4.9 10.4 6.7 0.9 3.9 2.4 IHMC3 4.7 13.1 6.9 0.5 4.5 2.5 ##### Scoring Configuration: neutralizeRealisCoref ##### submission P R F1 EAArg EALink Overall IHMC1 24.6 21.1 22.7 12.1 8.8 10.5 IHMC2 10.6 16.5 12.9 4.4 6.9 5.6 IHMC3 10.1 20.7 13.6 3.5 9.1 6.3 ##### Scoring Configuration: neutralizeRealis ##### submission P R F1 EAArg EALink Overall IHMC1 16.0 14.1 15.0 5.6 5.0 5.3 IHMC2 7.5 11.9 9.2 1.8 4.4 3.1 IHMC3 7.0 14.5 9.4 1.3 5.0 3.1 ####### Genre: df ####### ##### Scoring Configuration: withRealis ##### submission P R F1 EAArg EALink Overall IHMC1 10.0 12.3 11.0 3.7 4.0 3.8 IHMC2 5.5 10.9 7.3 1.7 4.6 3.2 IHMC3 5.0 13.1 7.2 1.0 5.5 3.3 ##### Scoring Configuration: neutralizeRealisCoref ##### submission P R F1 EAArg EALink Overall IHMC1 24.9 19.0 21.6 11.4 8.3 9.8 IHMC2 13.7 17.5 15.4 6.8 8.0 7.4 IHMC3 12.3 21.0 15.5 5.8 10.8 8.3 ##### Scoring Configuration: neutralizeRealis ##### submission P R F1 EAArg EALink Overall IHMC1 15.9 12.4 13.9 6.0 4.4 5.2 IHMC2 9.6 12.4 10.8 3.5 5.2 4.4 IHMC3 8.3 14.4 10.5 2.4 6.2 4.3 ####### Genre: nw ####### ##### Scoring Configuration: withRealis ##### submission P R F1 EAArg EALink Overall IHMC1 10.5 14.2 12.1 2.9 5.1 4.0 IHMC2 4.5 10.0 6.2 0.2 3.3 1.7 IHMC3 4.5 13.0 6.7 0.0 3.7 1.9 ##### Scoring Configuration: neutralizeRealisCoref ##### submission P R F1 EAArg EALink Overall IHMC1 24.4 22.8 23.6 12.8 9.3 11.0 IHMC2 8.7 15.6 11.2 2.3 6.0 4.2 IHMC3 8.8 20.4 12.3 1.4 7.7 4.5 ##### Scoring Configuration: neutralizeRealis ##### submission P R F1 EAArg EALink Overall IHMC1 16.0 15.5 15.7 5.2 5.5 5.3 IHMC2 6.3 11.3 8.1 0.3 3.6 2.0 IHMC3 6.2 14.6 8.7 0.3 4.0 2.1