Go to the NIST home page Go to the TAC Home Page TAC Banner

Return to TAC Homepage

TAC 2020 Tracks
  EPIC-QA
  RUFES
  SM-KBP
      Guidelines
      Data
      Tools
Call for Participation
Track Registration
Reporting Guidelines
TAC 2020 Workshop




Streaming Multimedia Knowledge Base Population (SM-KBP) 2020

Evaluation: August 2020 - January 2021
Workshop: February 22-23, 2021

Conducted by:
U.S. National Institute of Standards and Technology (NIST)

With support from:
U.S. Department of Defense

Background

In scenarios such as natural disasters or international conflicts, analysts and the public are often confronted with a variety of information coming through multiple media sources. There is a need for technologies to analyze and extract knowledge from multimedia to develop and maintain an understanding of events, situations, and trends as they unfold around the world.

The goal of DARPA's Active Interpretation of Disparate Alternatives (AIDA) Program is to develop a multi-hypothesis semantic engine that generates explicit alternative interpretations of events, situations, and trends from a variety of unstructured sources, for use in noisy, conflicting, and potentially deceptive information environments. This engine must be capable of mapping knowledge elements (KE) automatically derived from multiple media sources into a common semantic representation, aggregating information derived from those sources, and generating and exploring multiple hypotheses about the events, situations, and trends of interest.

The streaming multimedia KBP track evaluates the performance of systems that have been developed in support of AIDA program goals. Following a pilot at TAC/TRECVID 2018, the first SM-KBP evaluation was run at TAC/TRECVID 2019. It is expected that the SM-KBP track will be run for a total of three phases of evaluation:

  • Phase 1 Evaluation: June-August 2019
  • Phase 2 Evaluation: August 2020 - January 2021
  • Phase 3 Evaluation early 2022

Task Overview

The SM-KBP track has three evaluation tasks:

  • Task 1: Extract mentions of Knowledge Elements from a stream of multimedia documents (including text, image, and video) and cluster together mentions of the same KE in each document to produce a document-level knowledge graph for each document.
  • Task 2: Aggregate and link the document-level knowledge graphs from Task 1 to construct a KB of the entire document stream without access to the raw documents themselves
  • Task 3: Generate hypotheses from a knowledge graph from Task 2, such that each hypothesis represents a semantically coherent interpretation of the document stream.

While tasks 2 and 3 and limited to teams that are part of DARPA's AIDA program, Tasks 1 is also open to non-AIDA researchers who are interested in multilingual multimedia information extraction.

Ontology: Teams will receive an "annotation" ontology that defines the entities, relations, events, and event and relation roles and arguments that must be extracted. The ontology contains approximately 180 entity types, 150 event types, and 50 relation types, including couarse-grained types (e.g., PER, Conflict.Attack, Physical.LocatedNear) and more fine-grained types (e.g., PER.Combatant.Sniper, Conflict.Attack.FireArmAttack, Physical.LocatedNear.Surround).

Documents: Task 1 systems will process a set of approximately 2000 documents in English, Spanish, and Russian, and output a document-level knowledge graph for each document. A document may contain multiple document elements in multiple modalities (text, image, video); therefore, cross-lingual and cross-modal entity, relation, and event coreference are required. For each document, systems must extract all mentions of entities, relations, and events and identify all arguments and temporal information for each event and relation.

Leaderboard: System output will be scored by comparing against gold standard annotations for a subset of the documents, and scores reported on a leaderboard. Two leaderboards will be set up for Task 1: A dry run leaderboard to submit results on practice documents, and an evaluation leaderboard to submit results on evaluation documents. Teams may submit to the dry run leaderboard as many times as desired, but they may submit to the evaluation leaderboard only a limited number of times. All Task 1 participants will submit to the same leaderboards, and scores will be viewable by all registered SM-KBP teams. However, while non-AIDA teams will be given the evaluation documents and will submit system output for evaluation, AIDA teams will not be given the evaluation documents, but must instead submit system dockers that will be run by NIST for the evaluation.

Schedule

    TAC SM-KBP 2020 Schedule
    August 28-October 18Task 1 Dry Run Leaderboard active (AIDA dockers only)
    September 21-October 18, 2020Task 1 Evaluation Leaderboard active (AIDA dockers only)
    December 8, 2020Task 1 Evaluation Source Corpus available (all participants)
    February 3, 2021Deadline for short system descriptions
    February 3, 2021Deadline for workshop presentation proposals
    February 8, 2021Notification of acceptance of oral presentation proposals
    February 14, 2021Deadline for system reports (workshop notebook version)
    February 22-23, 2021Thirteenth TAC workshop (online)
    April 1, 2021Deadline for system reports (final proceedings version)
    April 2021Task 1 Evaluation Leaderboard re-opens (all participants)

Mailing List

Join the sm-kbp group to subscribe yourself to the sm-kbp@list.nist.gov mailing list (if not already subscribed): Registering to participate in a track does not automatically add you to the mailing list. If you were previously subscribed to the mailing list, you do not have to re-subscribe (the mailing list is for anyone interested in SM-KBP, rather than specifically for SM-KBP participants, and thus carries over from year to year).

Organizing Committee

Hoa Trang Dang (U.S. National Institute of Standards and Techonology)
George Awad (U.S. National Institute of Standards and Techonology)
Asad Butt (U.S. National Institute of Standards and Techonology)
Shahzad Rajput (U.S. National Institute of Standards and Techonology)
Jason Duncan (MITRE)
Boyan Onyshkevych (U.S. Department of Defense)
Stephanie Strassel (Linguistic Data Consortium)
Jennifer Tracey (Linguistic Data Consortium)


NIST is an agency of the
U.S. Department of Commerce

privacy policy / security notice / accessibility statement
disclaimer
FOIA

Last updated: Wednesday, 10-Mar-2021 10:14:40 EST
Comments to: tac-web@nist.gov