===================================================================== TAC KBP 2015 EVENT ARGUMENT EXTRACTION AND LINKING EVALUATION RESULTS ===================================================================== Team ID: ZJU_Insight Organization: Zhejiang University Run ID: ZJU_Insight1 Did the run access the live Web during the evaluation window: No Did the run perform any cross-sentence reasoning: No Did the run use any distributed representations (e.g., of words): No Did the run return meaningful confidence values: No Run ID: ZJU_Insight2 Did the run access the live Web during the evaluation window: No Did the run perform any cross-sentence reasoning: No Did the run use any distributed representations (e.g., of words): No Did the run return meaningful confidence values: No Run ID: ZJU_Insight3 Did the run access the live Web during the evaluation window: No Did the run perform any cross-sentence reasoning: No Did the run use any distributed representations (e.g., of words): No Did the run return meaningful confidence values: No Run ID: ZJU_Insight4 Did the run access the live Web during the evaluation window: No Did the run perform any cross-sentence reasoning: No Did the run use any distributed representations (e.g., of words): No Did the run return meaningful confidence values: No Run ID: ZJU_Insight5 Did the run access the live Web during the evaluation window: No Did the run perform any cross-sentence reasoning: No Did the run use any distributed representations (e.g., of words): No Did the run return meaningful confidence values: No ************************************************************* ### The following are scores for from the TAC 2015 Event Argument and Linking Evaluation. ### For all scoring breakdowns, the summaries report: Precision, Recall, F1, EAArg Score, and Overall score. ### Details of the scoring and the scoring software can be found on the TAC 2015 EAL webpage. ### ### Scores are reported on the full data set (all_genre) and broken down by genre-- discussion forum only(df) newswire only(nw). ### ### The official score (withRealis) incorporates the correctness of the (ACTUAL, GENERIC, and OTHER) distinction ### and the correctness of canonical argument string resolution. As a diagnostic, we also report (a) a score ### that ignores the realis distinction (neutralizeRealis) and (b) a score that ignores both the realis distinction ### and canonical argument string resolution(neutraliseRealisCoref). ### ### Scores are reported over two data sets. Dataset1 (all_event_types), consists of 81 documents assessed for the ### full TAC EAL event taxonomy as specified in the 2015 evaluation plan. Dataset 2(restricted_event_types), ### consists of 201 documents assessed for only 6 event types (assertions outside of the 6 were ignored). Dataset2 ### includes the documents in Dataset1. Dataset 2 was assessed to allow a more in depth evaluation of event-specific ### performance (and variance across performance by event type). The 6 event types included in Dataset2 are: ### - Transaction.Transfer-Money ### - Movement.Transport-Artifact ### - Life.Marry ### - Contact.Meet ### - Conflict.Demonstrate ### - Conflict.Attack ### ### One participant (ZJU) submitted an submission an offset error. This system output was automatically fixed by BBN (the organizer) and ### the system by ZJU (the participant). Because the modifications were different, both numbers are reported. ### ### One participant (ver-CMU) participated in "verification" version of the task. This system took as its input all ### other system submissions. This submission included the ZJU submission which had broken offsets and ### did not include either BBN's fix or ZJU's fix. Thus it is not comparable to the other systems in task performed. ### ### The LDC submission was produced with an LDC annotator spending 45-60 minutes on the task of extracting arguments ### and grouping them. The low recall of the LDC submission is due at least in part to the time limitation. ### ### While all scores provide interesting diagnostic information, the "official" evaluation metric is Dataset1(all_event_types) on the ### both genres (all_genre) using the official(withRealis) metric. #################################### ###### All Event Types ###### ####### Genre: all_genre ####### ##### Scoring Configuration: withRealis ##### submission P R F1 EAArg EALink Overall BBN_fixed-ZJU_Insight1 14.3 14.9 14.6 2.4 5.3 3.8 BBN_fixed-ZJU_Insight2 15.3 13.6 14.4 2.5 4.6 3.5 BBN_fixed-ZJU_Insight3 9.1 15.1 11.4 0.8 5.3 3.1 BBN_fixed-ZJU_Insight4 17.2 11.8 14.0 2.9 3.4 3.1 BBN_fixed-ZJU_Insight5 9.4 13.8 11.2 0.8 4.6 2.7 ZJU_fixed-ZJU_Insight1 14.6 15.4 15.0 2.5 5.7 4.1 ZJU_fixed-ZJU_Insight2 15.7 14.7 15.2 2.7 5.1 3.9 ZJU_fixed-ZJU_Insight3 14.6 15.4 15.0 2.5 5.7 4.1 ZJU_fixed-ZJU_Insight4 16.9 13.4 14.9 3.0 4.3 3.6 ZJU_fixed-ZJU_Insight5 15.7 14.7 15.2 2.7 5.1 3.9 ##### Scoring Configuration: neutralizeRealisCoref ##### submission P R F1 EAArg EALink Overall BBN_fixed-ZJU_Insight1 25.9 26.7 26.3 9.5 11.5 10.5 BBN_fixed-ZJU_Insight2 28.3 24.9 26.5 10.5 10.5 10.5 BBN_fixed-ZJU_Insight3 16.1 27.0 20.2 3.8 11.4 7.6 BBN_fixed-ZJU_Insight4 31.4 21.4 25.5 10.7 8.3 9.5 BBN_fixed-ZJU_Insight5 17.0 25.1 20.3 4.2 10.5 7.4 ZJU_fixed-ZJU_Insight1 25.6 27.0 26.3 9.3 11.4 10.4 ZJU_fixed-ZJU_Insight2 27.6 25.7 26.6 10.3 10.7 10.5 ZJU_fixed-ZJU_Insight3 25.6 27.0 26.3 9.3 11.4 10.4 ZJU_fixed-ZJU_Insight4 29.8 23.5 26.3 10.8 9.2 10.0 ZJU_fixed-ZJU_Insight5 27.6 25.7 26.6 10.3 10.7 10.5 ##### Scoring Configuration: neutralizeRealis ##### submission P R F1 EAArg EALink Overall BBN_fixed-ZJU_Insight1 17.4 18.5 17.9 3.2 6.2 4.7 BBN_fixed-ZJU_Insight2 18.7 17.0 17.8 3.5 5.4 4.5 BBN_fixed-ZJU_Insight3 11.0 18.8 13.9 0.9 6.3 3.6 BBN_fixed-ZJU_Insight4 20.8 14.6 17.2 3.9 4.0 4.0 BBN_fixed-ZJU_Insight5 11.5 17.3 13.8 1.0 5.5 3.2 ZJU_fixed-ZJU_Insight1 17.6 19.0 18.3 3.4 6.5 5.0 ZJU_fixed-ZJU_Insight2 18.8 18.0 18.4 3.8 6.0 4.9 ZJU_fixed-ZJU_Insight3 17.6 19.0 18.3 3.4 6.5 5.0 ZJU_fixed-ZJU_Insight4 20.1 16.4 18.1 3.8 4.9 4.4 ZJU_fixed-ZJU_Insight5 18.8 18.0 18.4 3.8 6.0 4.9 ####### Genre: df ####### ##### Scoring Configuration: withRealis ##### submission P R F1 EAArg EALink Overall BBN_fixed-ZJU_Insight1 14.8 15.6 15.2 2.9 6.3 4.6 BBN_fixed-ZJU_Insight2 15.9 14.4 15.1 3.1 5.4 4.3 BBN_fixed-ZJU_Insight3 9.1 15.8 11.5 0.8 6.4 3.6 BBN_fixed-ZJU_Insight4 17.3 12.7 14.6 3.1 4.3 3.7 BBN_fixed-ZJU_Insight5 9.4 14.5 11.4 1.0 5.5 3.3 ZJU_fixed-ZJU_Insight1 15.3 16.2 15.7 3.2 6.4 4.8 ZJU_fixed-ZJU_Insight2 16.4 15.4 15.9 3.4 5.8 4.6 ZJU_fixed-ZJU_Insight3 15.3 16.2 15.7 3.2 6.4 4.8 ZJU_fixed-ZJU_Insight4 17.3 14.0 15.5 3.7 4.6 4.2 ZJU_fixed-ZJU_Insight5 16.4 15.4 15.9 3.4 5.8 4.6 ##### Scoring Configuration: neutralizeRealisCoref ##### submission P R F1 EAArg EALink Overall BBN_fixed-ZJU_Insight1 26.3 26.9 26.6 10.9 13.4 12.2 BBN_fixed-ZJU_Insight2 28.4 25.0 26.6 11.6 12.3 12.0 BBN_fixed-ZJU_Insight3 16.0 27.2 20.1 3.7 13.5 8.6 BBN_fixed-ZJU_Insight4 31.1 22.1 25.8 11.7 10.2 10.9 BBN_fixed-ZJU_Insight5 16.6 25.3 20.0 4.2 12.4 8.3 ZJU_fixed-ZJU_Insight1 26.9 27.7 27.3 11.5 13.7 12.6 ZJU_fixed-ZJU_Insight2 28.6 26.2 27.3 11.9 12.9 12.4 ZJU_fixed-ZJU_Insight3 26.9 27.7 27.3 11.5 13.7 12.6 ZJU_fixed-ZJU_Insight4 30.1 23.9 26.6 12.1 11.0 11.6 ZJU_fixed-ZJU_Insight5 28.6 26.2 27.3 11.9 12.9 12.4 ##### Scoring Configuration: neutralizeRealis ##### submission P R F1 EAArg EALink Overall BBN_fixed-ZJU_Insight1 17.7 18.7 18.2 3.7 7.2 5.4 BBN_fixed-ZJU_Insight2 18.9 17.2 18.0 4.0 6.1 5.0 BBN_fixed-ZJU_Insight3 11.0 19.1 14.0 0.9 7.2 4.0 BBN_fixed-ZJU_Insight4 20.5 15.1 17.4 3.6 4.7 4.2 BBN_fixed-ZJU_Insight5 11.3 17.5 13.7 1.0 6.2 3.6 ZJU_fixed-ZJU_Insight1 18.3 19.4 18.8 4.1 7.2 5.7 ZJU_fixed-ZJU_Insight2 19.3 18.2 18.7 4.3 6.5 5.4 ZJU_fixed-ZJU_Insight3 18.3 19.4 18.8 4.1 7.2 5.7 ZJU_fixed-ZJU_Insight4 20.3 16.6 18.3 4.4 5.2 4.8 ZJU_fixed-ZJU_Insight5 19.3 18.2 18.7 4.3 6.5 5.4 ####### Genre: nw ####### ##### Scoring Configuration: withRealis ##### submission P R F1 EAArg EALink Overall BBN_fixed-ZJU_Insight1 13.9 14.4 14.1 2.0 4.7 3.3 BBN_fixed-ZJU_Insight2 14.9 13.1 13.9 2.1 4.0 3.1 BBN_fixed-ZJU_Insight3 9.0 14.7 11.2 0.8 4.7 2.8 BBN_fixed-ZJU_Insight4 17.1 11.3 13.6 2.8 2.9 2.8 BBN_fixed-ZJU_Insight5 9.4 13.5 11.1 0.7 4.2 2.4 ZJU_fixed-ZJU_Insight1 14.1 15.0 14.5 2.0 5.3 3.6 ZJU_fixed-ZJU_Insight2 15.2 14.3 14.7 2.3 4.8 3.5 ZJU_fixed-ZJU_Insight3 14.1 15.0 14.5 2.0 5.3 3.6 ZJU_fixed-ZJU_Insight4 16.6 13.1 14.6 2.5 4.1 3.3 ZJU_fixed-ZJU_Insight5 15.2 14.3 14.7 2.3 4.8 3.5 ##### Scoring Configuration: neutralizeRealisCoref ##### submission P R F1 EAArg EALink Overall BBN_fixed-ZJU_Insight1 25.6 26.6 26.1 8.7 10.3 9.5 BBN_fixed-ZJU_Insight2 28.2 24.7 26.3 9.9 9.4 9.7 BBN_fixed-ZJU_Insight3 16.2 26.8 20.2 3.8 10.2 7.0 BBN_fixed-ZJU_Insight4 31.7 20.9 25.2 10.2 7.1 8.7 BBN_fixed-ZJU_Insight5 17.2 24.9 20.3 4.3 9.4 6.8 ZJU_fixed-ZJU_Insight1 24.8 26.5 25.6 8.0 10.0 9.0 ZJU_fixed-ZJU_Insight2 27.1 25.4 26.2 9.3 9.4 9.3 ZJU_fixed-ZJU_Insight3 24.8 26.5 25.6 8.0 10.0 9.0 ZJU_fixed-ZJU_Insight4 29.6 23.2 26.0 10.0 8.2 9.1 ZJU_fixed-ZJU_Insight5 27.1 25.4 26.2 9.3 9.4 9.3 ##### Scoring Configuration: neutralizeRealis ##### submission P R F1 EAArg EALink Overall BBN_fixed-ZJU_Insight1 17.2 18.4 17.8 3.0 5.6 4.3 BBN_fixed-ZJU_Insight2 18.6 16.9 17.7 3.3 4.9 4.1 BBN_fixed-ZJU_Insight3 11.1 18.6 13.9 0.9 5.7 3.3 BBN_fixed-ZJU_Insight4 21.0 14.3 17.0 4.1 3.6 3.8 BBN_fixed-ZJU_Insight5 11.6 17.1 13.8 0.9 5.0 3.0 ZJU_fixed-ZJU_Insight1 17.1 18.8 17.9 3.0 6.1 4.6 ZJU_fixed-ZJU_Insight2 18.5 17.9 18.2 3.4 5.7 4.5 ZJU_fixed-ZJU_Insight3 17.1 18.8 17.9 3.0 6.1 4.6 ZJU_fixed-ZJU_Insight4 20.0 16.2 17.9 3.5 4.8 4.2 ZJU_fixed-ZJU_Insight5 18.5 17.9 18.2 3.4 5.7 4.5 #################################### ###### Restricted Event Types ###### ####### Genre: all_genre ####### ##### Scoring Configuration: withRealis ##### submission P R F1 EAArg EALink Overall BBN_fixed-ZJU_Insight1 11.9 14.1 12.9 4.0 4.5 4.2 BBN_fixed-ZJU_Insight2 12.8 12.6 12.7 4.0 3.8 3.9 BBN_fixed-ZJU_Insight3 7.5 14.4 9.9 2.4 4.6 3.5 BBN_fixed-ZJU_Insight4 14.1 9.9 11.6 3.8 2.5 3.1 BBN_fixed-ZJU_Insight5 7.7 12.8 9.6 2.3 3.9 3.1 ZJU_fixed-ZJU_Insight1 12.0 14.4 13.1 4.1 4.7 4.4 ZJU_fixed-ZJU_Insight2 13.1 13.8 13.4 4.4 4.4 4.4 ZJU_fixed-ZJU_Insight3 12.0 14.4 13.1 4.1 4.7 4.4 ZJU_fixed-ZJU_Insight4 13.8 11.8 12.7 4.2 3.3 3.8 ZJU_fixed-ZJU_Insight5 13.1 13.8 13.4 4.4 4.4 4.4 ##### Scoring Configuration: neutralizeRealisCoref ##### submission P R F1 EAArg EALink Overall BBN_fixed-ZJU_Insight1 21.5 24.9 23.1 11.7 10.1 10.9 BBN_fixed-ZJU_Insight2 23.0 22.2 22.6 11.3 8.7 10.0 BBN_fixed-ZJU_Insight3 13.2 25.2 17.3 7.4 10.1 8.8 BBN_fixed-ZJU_Insight4 25.3 17.3 20.5 9.9 6.0 7.9 BBN_fixed-ZJU_Insight5 13.5 22.5 16.9 7.3 8.8 8.1 ZJU_fixed-ZJU_Insight1 21.2 25.0 22.9 11.6 9.9 10.8 ZJU_fixed-ZJU_Insight2 22.8 23.5 23.1 11.7 9.2 10.4 ZJU_fixed-ZJU_Insight3 21.2 25.0 22.9 11.6 9.9 10.8 ZJU_fixed-ZJU_Insight4 24.0 20.2 21.9 10.7 7.0 8.9 ZJU_fixed-ZJU_Insight5 22.8 23.5 23.1 11.7 9.2 10.4 ##### Scoring Configuration: neutralizeRealis ##### submission P R F1 EAArg EALink Overall BBN_fixed-ZJU_Insight1 14.7 17.5 16.0 5.6 5.3 5.5 BBN_fixed-ZJU_Insight2 15.6 15.4 15.5 5.4 4.4 4.9 BBN_fixed-ZJU_Insight3 9.2 17.9 12.2 3.4 5.4 4.4 BBN_fixed-ZJU_Insight4 17.3 12.3 14.4 5.1 3.0 4.0 BBN_fixed-ZJU_Insight5 9.4 15.9 11.8 3.3 4.5 3.9 ZJU_fixed-ZJU_Insight1 14.9 18.1 16.3 5.9 5.6 5.7 ZJU_fixed-ZJU_Insight2 16.1 17.0 16.5 6.2 5.3 5.7 ZJU_fixed-ZJU_Insight3 14.9 18.1 16.3 5.9 5.6 5.7 ZJU_fixed-ZJU_Insight4 16.9 14.5 15.6 5.7 3.9 4.8 ZJU_fixed-ZJU_Insight5 16.1 17.0 16.5 6.2 5.3 5.7 ####### Genre: df ####### ##### Scoring Configuration: withRealis ##### submission P R F1 EAArg EALink Overall BBN_fixed-ZJU_Insight1 11.1 13.7 12.3 3.5 4.8 4.1 BBN_fixed-ZJU_Insight2 12.1 12.4 12.2 3.6 4.3 4.0 BBN_fixed-ZJU_Insight3 6.9 13.7 9.2 2.1 4.8 3.4 BBN_fixed-ZJU_Insight4 14.3 10.3 12.0 4.1 3.0 3.6 BBN_fixed-ZJU_Insight5 7.3 12.4 9.2 2.2 4.3 3.3 ZJU_fixed-ZJU_Insight1 11.0 13.7 12.2 3.4 4.7 4.0 ZJU_fixed-ZJU_Insight2 12.3 13.1 12.7 3.9 4.5 4.2 ZJU_fixed-ZJU_Insight3 11.0 13.7 12.2 3.4 4.7 4.0 ZJU_fixed-ZJU_Insight4 13.6 11.5 12.5 4.4 3.1 3.8 ZJU_fixed-ZJU_Insight5 12.3 13.1 12.7 3.9 4.5 4.2 ##### Scoring Configuration: neutralizeRealisCoref ##### submission P R F1 EAArg EALink Overall BBN_fixed-ZJU_Insight1 20.2 24.2 22.0 11.1 9.4 10.2 BBN_fixed-ZJU_Insight2 21.6 21.7 21.6 10.6 8.5 9.5 BBN_fixed-ZJU_Insight3 12.6 24.2 16.6 7.5 9.4 8.4 BBN_fixed-ZJU_Insight4 24.4 17.2 20.2 10.0 5.8 7.9 BBN_fixed-ZJU_Insight5 13.1 21.8 16.4 7.1 8.5 7.8 ZJU_fixed-ZJU_Insight1 20.5 24.8 22.4 11.5 9.8 10.6 ZJU_fixed-ZJU_Insight2 22.1 22.9 22.5 11.3 9.2 10.2 ZJU_fixed-ZJU_Insight3 20.5 24.8 22.4 11.5 9.8 10.6 ZJU_fixed-ZJU_Insight4 24.0 19.9 21.8 11.1 6.8 8.9 ZJU_fixed-ZJU_Insight5 22.1 22.9 22.5 11.3 9.2 10.2 ##### Scoring Configuration: neutralizeRealis ##### submission P R F1 EAArg EALink Overall BBN_fixed-ZJU_Insight1 13.8 16.9 15.2 5.0 5.3 5.1 BBN_fixed-ZJU_Insight2 14.6 15.0 14.8 4.7 4.7 4.7 BBN_fixed-ZJU_Insight3 8.8 17.2 11.6 3.0 5.3 4.2 BBN_fixed-ZJU_Insight4 16.9 12.2 14.2 5.0 3.0 4.0 BBN_fixed-ZJU_Insight5 9.1 15.4 11.4 2.9 4.7 3.8 ZJU_fixed-ZJU_Insight1 13.9 17.2 15.4 5.0 5.2 5.1 ZJU_fixed-ZJU_Insight2 15.1 16.1 15.6 5.3 5.0 5.2 ZJU_fixed-ZJU_Insight3 13.9 17.2 15.4 5.0 5.2 5.1 ZJU_fixed-ZJU_Insight4 16.4 13.8 15.0 5.6 3.5 4.5 ZJU_fixed-ZJU_Insight5 15.1 16.1 15.6 5.3 5.0 5.2 ####### Genre: nw ####### ##### Scoring Configuration: withRealis ##### submission P R F1 EAArg EALink Overall BBN_fixed-ZJU_Insight1 12.7 14.5 13.5 4.4 4.3 4.3 BBN_fixed-ZJU_Insight2 13.5 12.8 13.1 4.2 3.4 3.8 BBN_fixed-ZJU_Insight3 7.9 15.0 10.3 2.6 4.4 3.5 BBN_fixed-ZJU_Insight4 13.9 9.5 11.3 3.5 2.1 2.8 BBN_fixed-ZJU_Insight5 7.9 13.1 9.9 2.4 3.5 2.9 ZJU_fixed-ZJU_Insight1 12.9 15.1 13.9 4.7 4.8 4.7 ZJU_fixed-ZJU_Insight2 13.7 14.3 14.0 4.9 4.4 4.6 ZJU_fixed-ZJU_Insight3 12.9 15.1 13.9 4.7 4.8 4.7 ZJU_fixed-ZJU_Insight4 13.9 12.0 12.9 4.1 3.4 3.8 ZJU_fixed-ZJU_Insight5 13.7 14.3 14.0 4.9 4.4 4.6 ##### Scoring Configuration: neutralizeRealisCoref ##### submission P R F1 EAArg EALink Overall BBN_fixed-ZJU_Insight1 22.6 25.5 24.0 12.2 10.6 11.4 BBN_fixed-ZJU_Insight2 24.2 22.6 23.4 11.9 8.9 10.4 BBN_fixed-ZJU_Insight3 13.8 26.1 18.1 7.4 10.7 9.1 BBN_fixed-ZJU_Insight4 26.0 17.4 20.8 9.8 6.1 8.0 BBN_fixed-ZJU_Insight5 13.9 23.1 17.4 7.6 9.0 8.3 ZJU_fixed-ZJU_Insight1 21.8 25.2 23.4 11.7 10.0 10.8 ZJU_fixed-ZJU_Insight2 23.4 24.0 23.7 12.0 9.2 10.6 ZJU_fixed-ZJU_Insight3 21.8 25.2 23.4 11.7 10.0 10.8 ZJU_fixed-ZJU_Insight4 23.9 20.4 22.0 10.5 7.2 8.8 ZJU_fixed-ZJU_Insight5 23.4 24.0 23.7 12.0 9.2 10.6 ##### Scoring Configuration: neutralizeRealis ##### submission P R F1 EAArg EALink Overall BBN_fixed-ZJU_Insight1 15.6 18.0 16.7 6.1 5.3 5.7 BBN_fixed-ZJU_Insight2 16.5 15.8 16.1 6.1 4.3 5.2 BBN_fixed-ZJU_Insight3 9.7 18.6 12.8 3.7 5.4 4.6 BBN_fixed-ZJU_Insight4 17.7 12.3 14.5 5.1 2.9 4.0 BBN_fixed-ZJU_Insight5 9.6 16.3 12.1 3.6 4.4 4.0 ZJU_fixed-ZJU_Insight1 15.9 18.8 17.2 6.7 5.9 6.3 ZJU_fixed-ZJU_Insight2 16.9 17.9 17.4 6.9 5.5 6.2 ZJU_fixed-ZJU_Insight3 15.9 18.8 17.2 6.7 5.9 6.3 ZJU_fixed-ZJU_Insight4 17.2 15.1 16.1 5.9 4.3 5.1 ZJU_fixed-ZJU_Insight5 16.9 17.9 17.4 6.9 5.5 6.2