MTG-AD: A New Benchmark for Evaluating Inline Audio Description in MLLMs

Giving blind and low-vision (BLV) users a way to "watch" video relies on Audio Description (AD): narrating what is happening on screen through a speech track layered outside the visuals. Multimodal large language models (MLLMs) make it possible to generate audio descriptions on demand, but one key scenario has long lacked rigorous evaluation—inline AD: narration must be squeezed into the video's existing silent gaps without drowning out dialogue and sound effects.

Google's team has released the MTG-AD dataset and the MTG-AD-Eval evaluation framework to fill this gap. According to the paper, this work is the first to systematically examine MLLM performance on the end-to-end inline audio description task, and it provides fine-grained, reference-free metrics that can be used directly for model improvement.

Why "inline" is the harder problem

Traditional audio description is mostly post-production work, where the script can be flexibly adjusted in length during recording. Inline description is constrained by the video's own temporal structure: a passage must land exactly within a silent stretch, and one extra word can collide with the next line of dialogue.

The paper notes that existing MLLM audio description research rarely covers this constraint, and exploration of user-generated content (UGC) scenarios is especially lacking. UGC videos have irregular silence distributions and vary widely in speech rate and editing rhythm—exactly where inline description is most likely to fail. MTG-AD targets precisely this task of "working within the gaps."

MTG-AD: A benchmark co-annotated by blind and sighted experts

MTG-AD does not pursue scale for its own sake; it emphasizes annotation quality and task authenticity:

• 36 hours of video, averaging about 5 minutes each;
• 8k+ descriptions, a substantial total, each aligned to the timeline;
• Annotations completed by blind and sighted audio description experts working in paired collaboration.

The last point deserves elaboration. The end audience for audio description is BLV users, yet a considerable share of industry annotation is done by sighted people. Involving blind experts in co-creation means annotations do not merely describe "what is on screen" but also account for "what the listener needs to know right now"—so the criteria for information selection and narrative pacing are closer to real usage scenarios. The dataset is open-sourced on Hugging Face (google/mind-the-gap-inline-ad).

MTG-AD-Eval: Render the script into an audio track, then score it

MTG-AD-Eval is designed to "evaluate the final product, not the intermediate text." According to the paper, the framework is built on audio description expert guidelines and provides fine-grained, reference-free, actionable metrics.

The core step is end-to-end rendering: synthesizing the model-generated description script into an audio track, placing it back onto the original video's timeline, and checking whether it truly embeds into natural silent segments and whether it causes disruptive overlap. This step separates "looks good as text" from "holds up to the ear," and it is the step most text-level evaluations miss.

The framework and code are released on GitHub (google/mtg-ad), making it easy for researchers to reproduce and integrate with their own models.

Reference-free metrics refer to evaluating generation quality without relying on human-written "gold answers." In inline audio description this is especially critical: the same scene can have many reasonable descriptions, and forcing comparison against reference text would penalize creative expression. MTG-AD-Eval's reference-free metrics are designed based on audio description expert guidelines, directly measuring actionable dimensions such as whether the description embeds into silent segments, whether it overlaps with dialogue, and whether the pacing is coherent—thereby getting closer to the final listening experience.

Report card for frontier models: says the right thing, but not smoothly

The paper's analysis of multiple frontier MLLMs yields a clear conclusion: models perform well on accuracy of visual event description and substantially reduce speech overlap—the latter showing that models have indeed learned to fit words into the gaps.

But the weakness appears in narrative pacing and fluency. In other words, models know what to say and know when they can say it, but are not very good at organizing fragmented descriptions into a coherent narrative flow. For inline description this is no small matter: listeners receive a continuous speech stream, and broken pacing directly raises the cost of comprehension.

This finding also explains the nuanced "helpful but imperfect" result in the subsequent human evaluation.

Speech overlap here specifically means that the generated audio description overlaps in time with the video's original dialogue and sound effects, making it hard for listeners to distinguish the two speech streams. For BLV users, overlap directly masks key dialogue information and is the failure mode inline description should most avoid. The paper's "substantially reduce speech overlap" means the model has learned to consider timeline constraints while generating text, rather than focusing only on content correctness.

Human evaluation: automatic metrics are trustworthy, but human authors still edge ahead

The research team validated the effectiveness of the automated reference-free metrics using quality ratings from sighted raters and 7 BLV participants—the two were significantly correlated. This provides empirical support for "using automatic metrics to replace expensive human evaluation."

On user preference: participants slightly preferred human-written audio descriptions, but at the same time found MLLM-generated descriptions very helpful for understanding the video. This combined conclusion is quite pragmatic: machine descriptions have not yet reached expert level, but they have crossed the "usable" threshold, enough to support practical deployment of on-demand generation.

Based on this, the paper offers actionable system design recommendations, with a clear direction: the priority is not "seeing more accurately" but "speaking more smoothly."

What it means for the industry

The value of MTG-AD goes beyond a single dataset. It elevates inline audio description from an "engineering detail" to an evaluable research task, and offers three reusable methodological principles: define the task by real temporal constraints, use the rendered audio track as the evaluation object, and involve BLV users to validate metric validity.

For teams building accessibility products, this means a different yardstick is needed when evaluating MLLM audio description capabilities—text similarity cannot reflect whether narration steps on dialogue's toes, nor can it measure pacing breaks in the listening experience. Both the dataset and the evaluation framework are open-sourced, with low integration cost.

Reference: the paper Mind The Gap (MTG-AD): Dataset and Fine-grained Evaluation for End-to-End Inline Audio Descriptions.

Background note

User-generated content (UGC) refers to video produced and uploaded by ordinary users rather than professional organizations—typically vlogs, tutorials, and livestream clips. Unlike film and television, UGC audio tracks often contain irregular silent gaps, impromptu speech, ambient noise, and abrupt cut points, and lack a unified loudness standard. These characteristics mean inline audio description must rely on fine-grained voice activity detection to locate the "gaps" where narration can be inserted, placing higher demands on a model's temporal reasoning ability.