QwenLM/Omnilingua-Bench

QwenLM/Omnilingua-Bench

Language: Python

Stars: 0

Forks: 0

Open issues: 0

Created: 2026-09-16T10:31:13Z

Pushed: 2026-09-18T15:07:50Z

Default branch: main

Fork: no

Archived: no

README:

Omnilingua-Bench

English | [中文](#chinese)

Multilingual, omni-modal evaluation of long-audio recognition, multi-speaker understanding, and instruction following.

Subsets

| Subset | Modality / Track | Task | Languages | Size | Key metrics | Released dataset (split · version) | | --- | --- | --- | --- | --- | --- | --- | | Omnilingua-LongASR | Audio → text; text task instruction included | Long-form same-language transcription | 10: zh, en, ar, es, fr, id, ja, ko, ru, th | 135 items, ~129.94 h | WER ↓; empty / endless anomaly rate ↓ | [OmniLingua-LongAudioASR_v202609_public.jsonl](Omnilingua-LongASR/OmniLingua-LongAudioASR_v202609_public.jsonl) · test · v202609 | | Omnilingua-LongQA (MuLA-Bench) | Audio + text question → text | Long-audio QA, factual completeness, long-range understanding, reasoning and temporal grounding | 16: zh, en, ar, de, es, fr, hi, id, it, ja, ko, pt, ru, th, tr, vi | 5,029 QA pairs, 1,768 distinct audios, ~1,370.95 h | Answer / rubric score ↑ | [MuLA-Bench_v202609_public.jsonl](MuLA-Bench/MuLA-Bench_v202609_public.jsonl) · test · v202609 | | Omnilingua-MSpeaker | audio: audio → text with speakers and timestamps | Multi-speaker transcription, speaker attribution and time alignment | 6: es, fr, ja, ko, ru, th | 297 items, ~26.27 h | tcpWER ↓, cpWER ↓, DER ↓ | [Omnilingua-MSpeaker_v202609_public.jsonl](Omnilingua-MSpeaker/Omnilingua-MSpeaker_v202609_public.jsonl) · test · v202609 | | Omnilingua-MSpeaker | audiovisual: audio-visual → text with speakers and timestamps | Vision-augmented multi-speaker transcription and attribution | 6: es, fr, ja, ko, ru, th | 297 items, ~26.27 h | tcpWER ↓, cpWER ↓, DER ↓ | Shares the same source_url video (with visuals) as the audio track; data as above | | Omnilingua-MaXIFE | Speech instruction → text; some samples include text context | Multilingual complex instruction following | 7 mid-/high-resource languages: fr, id, ja, ko, it, pt, tr | 5,387 synthesized-speech items | Mean constraint score ↑, all-constraint satisfaction rate ↑ | [Omnilingua-MaXIFE_v202609_public.jsonl](Omnilingua-MaXIFE/Omnilingua-MaXIFE_v202609_public.jsonl) (with WAV) · test · v202609 |

Highlights

• Long-audio capability: single clips in the released Omnilingua-LongASR range about 30.56–200.04 minutes, focusing on transcription completeness and stability in long context.
• Long-audio understanding: Omnilingua-LongQA (MuLA-Bench) probes factual completeness, long-range information integration, reasoning and temporal grounding. The released dataset has 5,029 QA pairs across 16 languages, over 1,768 distinct audios, each 20.10–180.00 minutes, totaling about 1,370.95 hours.
• Multi-speaker, real-world scenes: the Omnilingua-MSpeaker audio track is drawn from public social-media videos; the released dataset has 297 items from 105 videos, about 26.27 hours in total (Omnilingua-MSpeaker/). It provides video URLs and start/end ranges via source_url, ships no media files, and users must download them themselves. Reference annotations keep text, timestamps and anonymized speaker IDs.
• Complex instruction composition: the full Omnilingua-MaXIFE source list contains 11 categories, 47 constraint types covering format, length, keywords, language switching, style, tone, citation and more; the current release supports 7 mid-/high-resource languages, and the in-scope constraint-type distribution is being re-counted. It evaluates not only the answer content but also whether the instruction is followed.

Quick start

Run from the repository root; requires Python 3.10+:

python -m pip install -r requirements.txt
python tools/validate.py Omnilingua-LongASR/OmniLingua-LongAudioASR_v202609_public.jsonl --require-reference

The released datasets use relative-path placeholders in input.media[].path; the repository does not include the actual media. The command above only validates structure and semantics. Once you have the media, you can check file existence:

python tools/validate.py your_manifest.jsonl --check-media --media-root /path/to/dataset

Unified data format

One sample per line, with the common structure below; see the [format spec](docs/data_format.md) and [JSON Schema](schemas/sample.schema.json) for the full definition.

{
"schema_version": "0.1.0",
"id": "longaudioasr-demo-zh-0001",
"subset": "Omnilingua-LongASR",
"subset_version": "draft",
"split": "example",
"task": "long_audio_asr",
"track": "audio",
"languages": {"source": ["zh"], "target": ["zh"], "instruction": ["zh"]},
"input": {
"messages": [{"role": "user", "content": [
{"type": "audio", "media_id": "audio-0"},
{"type": "text", "text": "请按原语言完整转写音频。"}
]}],
"media": [{"id": "audio-0", "type": "audio", "path": "media/demo.wav", "duration_sec": 3.0}]
},
"reference": {"text": "这是一条格式示例。"},
"metadata": {"synthetic": true, "license": "TBD", "domain": "example"}
}

Project structure

.
├── README.md
├── Omnilingua-LongASR/
│ └── OmniLingua-LongAudioASR_v202609_public.jsonl
├── MuLA-Bench/
│ └── MuLA-Bench_v202609_public.jsonl
├── Omnilingua-MSpeaker/
│ └── Omnilingua-MSpeaker_v202609_public.jsonl
├── docs/
│ ├── subsets.md # Subset tasks, language coverage and evaluation criteria
│ └── data_format.md # Unified format, prediction format and migration rules
├── metadata/subsets.json # Subset registry and verified coverage
├── schemas/sample.schema.json # Sample JSON Schema
├── tools/
│ └── validate.py # Format, reference and time-range validation
├── requirements.txt
├── .gitlab-ci.yml
└── .gitignore

Data usage statement and license

For third-party original audio/video (Omnilingua-MSpeaker, as well as media referenced by source_url in Omnilingua-LongASR and Omnilingua-LongQA (MuLA-Bench)), this dataset provides only index references and text annotation results. It does not include, copy or redistribute any such original audio/video content. It is for academic research only, not for commercial use, and the copyright of such original audio/video content belongs to its respective creators and rights holders. When accessing and using the referenced material, users must comply with the terms...