Tencent-Hunyuan/OmniCapBench

Tencent-Hunyuan/OmniCapBench

Description: [NeurIPS 2026] OmniCapBench: A Deep-Structured Evaluation Framework for Fine-Grained Audio-Visual Captioning

Language: Python

License: Apache-2.0

Stars: 4

Forks: 0

Open issues: 0

Created: 2026-10-12T07:14:58Z

Pushed: 2026-10-12T09:59:10Z

Default branch: main

Fork: no

Archived: no

README:

中文 ｜ English

OmniCapBench: A Deep-Structured Evaluation Framework for Fine-Grained Audio-Visual Captioning

📊 Full leaderboard on the project website · 🤗 Data on Hugging Face · 📖 Metrics reference

News

• [2026/10] 📄 The paper is on arXiv: arXiv:2610.12458. The project website is live with the full leaderboard across nine omnimodal MLLMs.
• [2026/10] 🤗 The benchmark data is hosted on Hugging Face — 786 densely annotated videos with 5,818 references, 11,419 shots, 39,160 subshots and 6,537 audio events.
• [2026/09] 🎉 OmniCapBench was accepted to NeurIPS 2026 (Evaluations & Datasets Track).
• [2026/09] 🧩 OmniCapBench is now open-source: the annotation pipeline for building a deep-structured evaluation set, plus the full scoring toolkit.

Contents

• [News](#news)
• [Introduction](#introduction)
• [Resources](#resources)
• [Evaluation Units](#evaluation-units)
• [Metrics](#metrics)
• [Installation](#installation)
• [Quick Start](#quick-start)
• [Step 1 — Build an Evaluation Set](#step-1--build-an-evaluation-set)
• [Step 2 — Score a Model](#step-2--score-a-model)
• [Supported Models](#supported-models)
• [Citation](#citation)
• [License](#license)

Introduction

OmniCapBench reframes audio–visual caption evaluation as a *deep-structured diagnostic* task. Instead of scoring a free-form paragraph with a global LLM judge, it shifts the prediction target to sets of atomic, verifiable evaluation units across three native tracks: entity references, visual shots, and audio events.

This resolves a trade-off existing caption benchmarks cannot escape. Whole-caption protocols cover the whole video but compress everything into an opaque scalar that hides local errors. Probe-based protocols localize errors but cover only sparse points. Both then hand dense, monolithic text to an unconstrained LLM judge, which makes scores swing with judge strength.

OmniCapBench keeps full coverage *and* exact localization by splitting the scoring operator in two:

1. Deterministic verification. Rules check schema validity, identifier scoping, time bounds, and cross-track links, then align predicted units to the ground truth by temporal IoU (shots, events) or a bounded LLM matcher (references, subshots). 2. Localized semantic checking. An LLM judge sees only *one aligned field pair at a time*. Structure is already resolved, so it never has to parse text, resolve identifiers, or track time.

Because every score traces back to a specific unit, failures become diagnosable: an undefined identifier, a weak temporal overlap, a sound attached to the wrong shot, or a hallucinated attribute are each counted separately rather than averaged away.

The paper applies this framework to 786 densely annotated videos (12.8 hours, 20 level-1 and 125 level-2 categories, yielding 5,818 references, 11,419 shots, 39,160 subshots, and 6,537 events). This repository contains the pipeline that produced those annotations and the metrics that scored them, so the same protocol can be reproduced on any video collection.

Resources

| Resource | Link | | ------------------------------ | --------------------------------------------------------------------------------- | | Project website & leaderboard | 01yzzyu.github.io/OmniCapBench/ | | Paper (arXiv) | arXiv:2610.12458 | | Paper (OpenReview) | NeurIPS 2026, Evaluations & Datasets Track | | Benchmark data | 🤗 OmniCapBench/OmniCapBench | | Code | Tencent-Hunyuan/OmniCapBench |

from huggingface_hub import snapshot_download

snapshot_download(
repo_id="OmniCapBench/OmniCapBench",
repo_type="dataset",
local_dir="./data/omnicapbench",
)

Evaluation Units

A video V is formalized as a structured state S*(V) = (R, E, H) across three interdependent tracks. This schema is what the pipeline emits and what the metrics consume.

| Track | What it encodes | Verifiable properties | | ---------------------------- | ------------------------------------------------------------------------------------------------------------------- | -------------------------------------------------------- | | References (R) | Persistent identities for people, objects, and scenes, each with a unique identifier and an appearance anchor | Identity permanence across cuts, appearance fidelity | | Events (E) | Auditory events and spoken dialogue with continuous time ranges | Sound recovery, temporal boundaries, transcription | | Shots (H) | The visual timeline split into camera shots and finersubshots, each linking back to references and events | Shot recovery, boundary precision, audio–visual binding |

For open-vocabulary fields (references and subshots) scoring is bidirectional: the recall direction takes the ground truth as the subject and penalizes omissions, while the precision direction takes the prediction as the subject and penalizes hallucinations. This stops a model from gaming the score with either terse answers or speculative detail dumps.

Field-level schema and the exact JSONL record shapes: [docs/METRICS.md](docs/METRICS.md).

Metrics

Rule-based structural metrics and localized semantic metrics form two independent tables; no aggregate total score is reported, so every number stays attributable to one component of the video state.

| Family | Metrics | What it measures | | ------------ | ------------------------------------------------------------------------------- | --------------------------------------------------------------------------------------- | | Gate | SGC | Schema gate compliance — is the output complete enough to be scored at all | | Visual | RefUse, CCC | Correct reference use inside shots/events/subshots, and identity permanence across cuts | | Visual | `Ref...