===================================================================== TAC KBP 2015 EVENT ARGUMENT EXTRACTION AND LINKING EVALUATION RESULTS ===================================================================== Team ID: BUPT_PRIS Organization: Beijing University of Posts and Telecommunications Run ID: BUPT_PRIS1 Did the run access the live Web during the evaluation window: No Did the run perform any cross-sentence reasoning: No Did the run use any distributed representations (e.g., of words): No Did the run return meaningful confidence values: No Run ID: BUPT_PRIS2 Did the run access the live Web during the evaluation window: No Did the run perform any cross-sentence reasoning: No Did the run use any distributed representations (e.g., of words): No Did the run return meaningful confidence values: No Run ID: BUPT_PRIS3 Did the run access the live Web during the evaluation window: No Did the run perform any cross-sentence reasoning: No Did the run use any distributed representations (e.g., of words): No Did the run return meaningful confidence values: No Run ID: BUPT_PRIS4 Did the run access the live Web during the evaluation window: No Did the run perform any cross-sentence reasoning: No Did the run use any distributed representations (e.g., of words): No Did the run return meaningful confidence values: No ************************************************************* ### The following are scores for from the TAC 2015 Event Argument and Linking Evaluation. ### For all scoring breakdowns, the summaries report: Precision, Recall, F1, EAArg Score, and Overall score. ### Details of the scoring and the scoring software can be found on the TAC 2015 EAL webpage. ### ### Scores are reported on the full data set (all_genre) and broken down by genre-- discussion forum only(df) newswire only(nw). ### ### The official score (withRealis) incorporates the correctness of the (ACTUAL, GENERIC, and OTHER) distinction ### and the correctness of canonical argument string resolution. As a diagnostic, we also report (a) a score ### that ignores the realis distinction (neutralizeRealis) and (b) a score that ignores both the realis distinction ### and canonical argument string resolution(neutraliseRealisCoref). ### ### Scores are reported over two data sets. Dataset1 (all_event_types), consists of 81 documents assessed for the ### full TAC EAL event taxonomy as specified in the 2015 evaluation plan. Dataset 2(restricted_event_types), ### consists of 201 documents assessed for only 6 event types (assertions outside of the 6 were ignored). Dataset2 ### includes the documents in Dataset1. Dataset 2 was assessed to allow a more in depth evaluation of event-specific ### performance (and variance across performance by event type). The 6 event types included in Dataset2 are: ### - Transaction.Transfer-Money ### - Movement.Transport-Artifact ### - Life.Marry ### - Contact.Meet ### - Conflict.Demonstrate ### - Conflict.Attack ### ### One participant (ZJU) submitted an submission an offset error. This system output was automatically fixed by BBN (the organizer) and ### the system by ZJU (the participant). Because the modifications were different, both numbers are reported. ### ### One participant (ver-CMU) participated in "verification" version of the task. This system took as its input all ### other system submissions. This submission included the ZJU submission which had broken offsets and ### did not include either BBN's fix or ZJU's fix. Thus it is not comparable to the other systems in task performed. ### ### The LDC submission was produced with an LDC annotator spending 45-60 minutes on the task of extracting arguments ### and grouping them. The low recall of the LDC submission is due at least in part to the time limitation. ### ### While all scores provide interesting diagnostic information, the "official" evaluation metric is Dataset1(all_event_types) on the ### both genres (all_genre) using the official(withRealis) metric. #################################### ###### All Event Types ###### ####### Genre: all_genre ####### ##### Scoring Configuration: withRealis ##### submission P R F1 EAArg EALink Overall BUPT_PRIS1 30.2 14.0 19.1 7.3 5.2 6.2 BUPT_PRIS2 33.6 7.7 12.5 4.6 1.6 3.1 BUPT_PRIS3 26.2 14.3 18.5 6.1 4.8 5.5 BUPT_PRIS4 29.9 16.2 21.0 8.2 6.4 7.3 ##### Scoring Configuration: neutralizeRealisCoref ##### submission P R F1 EAArg EALink Overall BUPT_PRIS1 47.4 21.8 29.9 15.8 8.4 12.1 BUPT_PRIS2 62.9 14.2 23.2 12.2 3.7 8.0 BUPT_PRIS3 47.7 25.5 33.2 18.6 10.6 14.6 BUPT_PRIS4 47.8 25.4 33.2 18.6 10.4 14.5 ##### Scoring Configuration: neutralizeRealis ##### submission P R F1 EAArg EALink Overall BUPT_PRIS1 36.8 17.6 23.8 10.5 6.1 8.3 BUPT_PRIS2 41.4 9.8 15.8 6.8 1.9 4.3 BUPT_PRIS3 32.1 18.0 23.1 9.1 5.7 7.4 BUPT_PRIS4 36.3 20.2 26.0 11.8 7.4 9.6 ####### Genre: df ####### ##### Scoring Configuration: withRealis ##### submission P R F1 EAArg EALink Overall BUPT_PRIS1 33.5 7.3 12.0 4.8 1.8 3.3 BUPT_PRIS2 33.5 7.3 12.0 4.8 1.8 3.3 BUPT_PRIS3 33.5 7.3 12.0 4.8 1.8 3.3 BUPT_PRIS4 33.5 7.3 12.0 4.8 1.8 3.3 ##### Scoring Configuration: neutralizeRealisCoref ##### submission P R F1 EAArg EALink Overall BUPT_PRIS1 62.8 13.3 22.0 11.6 3.8 7.7 BUPT_PRIS2 62.8 13.3 22.0 11.6 3.8 7.7 BUPT_PRIS3 62.8 13.3 22.0 11.6 3.8 7.7 BUPT_PRIS4 62.8 13.3 22.0 11.6 3.8 7.7 ##### Scoring Configuration: neutralizeRealis ##### submission P R F1 EAArg EALink Overall BUPT_PRIS1 40.7 9.1 14.9 6.7 2.0 4.3 BUPT_PRIS2 40.7 9.1 14.9 6.7 2.0 4.3 BUPT_PRIS3 40.7 9.1 14.9 6.7 2.0 4.3 BUPT_PRIS4 40.7 9.1 14.9 6.7 2.0 4.3 ####### Genre: nw ####### ##### Scoring Configuration: withRealis ##### submission P R F1 EAArg EALink Overall BUPT_PRIS1 29.5 18.0 22.4 8.8 7.2 8.0 BUPT_PRIS2 33.6 7.9 12.8 4.5 1.5 3.0 BUPT_PRIS3 24.9 18.5 21.2 6.9 6.6 6.8 BUPT_PRIS4 29.3 21.6 24.9 10.3 9.1 9.7 ##### Scoring Configuration: neutralizeRealisCoref ##### submission P R F1 EAArg EALink Overall BUPT_PRIS1 44.0 27.0 33.5 18.4 11.2 14.8 BUPT_PRIS2 63.0 14.8 24.0 12.6 3.7 8.2 BUPT_PRIS3 45.0 33.0 38.1 22.9 14.8 18.9 BUPT_PRIS4 45.1 32.9 38.0 22.9 14.4 18.7 ##### Scoring Configuration: neutralizeRealis ##### submission P R F1 EAArg EALink Overall BUPT_PRIS1 35.9 22.8 27.9 13.0 8.6 10.8 BUPT_PRIS2 41.8 10.3 16.5 6.8 1.9 4.3 BUPT_PRIS3 30.6 23.6 26.6 10.7 8.0 9.3 BUPT_PRIS4 35.5 27.1 30.7 15.1 10.7 12.9 #################################### ###### Restricted Event Types ###### ####### Genre: all_genre ####### ##### Scoring Configuration: withRealis ##### submission P R F1 EAArg EALink Overall BUPT_PRIS1 28.2 10.8 15.6 7.1 4.2 5.7 BUPT_PRIS2 36.7 7.8 12.9 6.2 2.6 4.4 BUPT_PRIS3 26.0 12.3 16.7 7.2 4.7 5.9 BUPT_PRIS4 29.4 13.7 18.7 8.5 6.0 7.3 ##### Scoring Configuration: neutralizeRealisCoref ##### submission P R F1 EAArg EALink Overall BUPT_PRIS1 42.4 16.3 23.5 12.3 6.4 9.4 BUPT_PRIS2 63.3 13.6 22.4 12.3 4.4 8.3 BUPT_PRIS3 45.0 21.3 28.9 16.2 9.5 12.9 BUPT_PRIS4 45.1 21.0 28.7 16.0 9.2 12.6 ##### Scoring Configuration: neutralizeRealis ##### submission P R F1 EAArg EALink Overall BUPT_PRIS1 33.7 13.3 19.1 8.8 4.9 6.9 BUPT_PRIS2 44.4 9.7 15.9 7.8 2.8 5.3 BUPT_PRIS3 31.4 15.3 20.6 9.3 5.5 7.4 BUPT_PRIS4 35.0 16.8 22.7 10.9 6.9 8.9 ####### Genre: df ####### ##### Scoring Configuration: withRealis ##### submission P R F1 EAArg EALink Overall BUPT_PRIS1 33.6 5.5 9.5 4.3 1.4 2.8 BUPT_PRIS2 33.6 5.5 9.5 4.3 1.4 2.8 BUPT_PRIS3 33.6 5.5 9.5 4.3 1.4 2.8 BUPT_PRIS4 33.6 5.5 9.5 4.3 1.4 2.8 ##### Scoring Configuration: neutralizeRealisCoref ##### submission P R F1 EAArg EALink Overall BUPT_PRIS1 58.6 9.5 16.3 8.6 2.9 5.8 BUPT_PRIS2 58.6 9.5 16.3 8.6 2.9 5.8 BUPT_PRIS3 58.6 9.5 16.3 8.6 2.9 5.8 BUPT_PRIS4 58.6 9.5 16.3 8.6 2.9 5.8 ##### Scoring Configuration: neutralizeRealis ##### submission P R F1 EAArg EALink Overall BUPT_PRIS1 37.8 6.3 10.8 5.0 1.4 3.2 BUPT_PRIS2 37.8 6.3 10.8 5.0 1.4 3.2 BUPT_PRIS3 37.8 6.3 10.8 5.0 1.4 3.2 BUPT_PRIS4 37.8 6.3 10.8 5.0 1.4 3.2 ####### Genre: nw ####### ##### Scoring Configuration: withRealis ##### submission P R F1 EAArg EALink Overall BUPT_PRIS1 26.9 15.3 19.5 9.5 6.4 8.0 BUPT_PRIS2 38.4 9.8 15.6 7.8 3.5 5.6 BUPT_PRIS3 24.5 18.0 20.8 9.7 7.3 8.5 BUPT_PRIS4 28.6 20.7 24.0 12.2 9.8 11.0 ##### Scoring Configuration: neutralizeRealisCoref ##### submission P R F1 EAArg EALink Overall BUPT_PRIS1 38.5 22.2 28.2 15.6 9.3 12.4 BUPT_PRIS2 65.9 17.0 27.0 15.4 5.6 10.5 BUPT_PRIS3 42.5 31.4 36.1 22.8 14.9 18.9 BUPT_PRIS4 42.5 30.9 35.8 22.4 14.3 18.3 ##### Scoring Configuration: neutralizeRealis ##### submission P R F1 EAArg EALink Overall BUPT_PRIS1 32.6 19.3 24.2 12.0 7.8 9.9 BUPT_PRIS2 48.0 12.7 20.1 10.3 4.0 7.1 BUPT_PRIS3 30.2 23.1 26.2 13.1 8.9 11.0 BUPT_PRIS4 34.4 25.9 29.6 15.9 11.3 13.6