Streaming Multimedia Knowledge Base Population (SM-KBP) 2020

Background

In scenarios such as natural disasters or international conflicts, analysts and the public are often confronted with a variety of information coming through multiple media sources. There is a need for technologies to analyze and extract knowledge from multimedia to develop and maintain an understanding of events, situations, and trends as they unfold around the world.

The goal of DARPA's Active Interpretation of Disparate Alternatives (AIDA) Program is to develop a multi-hypothesis semantic engine that generates explicit alternative interpretations of events, situations, and trends from a variety of unstructured sources, for use in noisy, conflicting, and potentially deceptive information environments. This engine must be capable of mapping knowledge elements (KE) automatically derived from multiple media sources into a common semantic representation, aggregating information derived from those sources, and generating and exploring multiple hypotheses about the events, situations, and trends of interest.

The streaming multimedia KBP track evaluates the performance of systems that have been developed in support of AIDA program goals. Following a pilot at TAC/TRECVID 2018, the first SM-KBP evaluation was run at TAC/TRECVID 2019. It is expected that the SM-KBP track will be run for a total of three phases of evaluation:

• Phase 1 Evaluation: June-August 2019
• Phase 2 Evaluation: August 2020 - January 2021
• Phase 3 Evaluation early 2022

Task Overview

The SM-KBP track has three evaluation tasks:

• Task 1: Extract mentions of Knowledge Elements from a stream of multimedia documents (including text, image, and video) and cluster together mentions of the same KE in each document to produce a document-level knowledge graph for each document.
• Task 2: Aggregate and link the document-level knowledge graphs from Task 1 to construct a KB of the entire document stream without access to the raw documents themselves
• Task 3: Generate hypotheses from a knowledge graph from Task 2, such that each hypothesis represents a semantically coherent interpretation of the document stream.

While tasks 2 and 3 and limited to teams that are part of DARPA's AIDA program, Tasks 1 is also open to non-AIDA researchers who are interested in multilingual multimedia information extraction.

Ontology: Teams will receive an "annotation" ontology that defines the entities, relations, events, and event and relation roles and arguments that must be extracted. The ontology contains approximately 180 entity types, 150 event types, and 50 relation types, including couarse-grained types (e.g., PER, Conflict.Attack, Physical.LocatedNear) and more fine-grained types (e.g., PER.Combatant.Sniper, Conflict.Attack.FireArmAttack, Physical.LocatedNear.Surround).

Documents: Task 1 systems will process a set of approximately 2000 documents in English, Spanish, and Russian, and output a document-level knowledge graph for each document. A document may contain multiple document elements in multiple modalities (text, image, video); therefore, cross-lingual and cross-modal entity, relation, and event coreference are required. For each document, systems must extract all mentions of entities, relations, and events and identify all arguments and temporal information for each event and relation.

Leaderboard: System output will be scored by comparing against gold standard annotations for a subset of the documents, and scores reported on a leaderboard. Two leaderboards will be set up for Task 1: A dry run leaderboard to submit results on practice documents, and an evaluation leaderboard to submit results on evaluation documents. Teams may submit to the dry run leaderboard as many times as desired, but they may submit to the evaluation leaderboard only a limited number of times. All Task 1 participants will submit to the same leaderboards, and scores will be viewable by all registered SM-KBP teams. However, while non-AIDA teams will be given the evaluation documents and will submit system output for evaluation, AIDA teams will not be given the evaluation documents, but must instead submit system dockers that will be run by NIST for the evaluation.

Schedule

TAC SM-KBP 2020 Schedule
August 28-October 18	Task 1 Dry Run Leaderboard active (AIDA dockers only)
September 21-October 18, 2020	Task 1 Evaluation Leaderboard active (AIDA dockers only)
December 8, 2020	Task 1 Evaluation Source Corpus available (all participants)
February 3, 2021	Deadline for short system descriptions
February 3, 2021	Deadline for workshop presentation proposals
February 8, 2021	Notification of acceptance of oral presentation proposals
February 14, 2021	Deadline for system reports (workshop notebook version)
February 22-23, 2021	Thirteenth TAC workshop (online)
April 1, 2021	Deadline for system reports (final proceedings version)
April 2021	Task 1 Evaluation Leaderboard re-opens (all participants)

Mailing List

• sm-kbp group (click to join)

Organizing Committee