Caption-once, Frames-on-Demand: Visual-Need Routing for Budget-Aware Agentic Long Video Understanding

Overview

CFD captions a long video once on the edge, then reasons in the cloud and retrieves raw frames only when the Visual-Need Router determines pixel-level evidence is necessary. On Video-MME, it achieves 67.5 overall with only 5.8 frames per question on average.

(a) Visual compression reduces visual tokens but risks losing coverage on long videos. (b) Textual translation improves scalability but loses fine-grained visual evidence. (c) CFD separates offline narrative indexing on the edge from on-demand visual verification on the cloud.

Abstract

Long-video understanding on edge devices must reason over hours of content under tight compute and bandwidth budgets. Subsampling visual tokens loses temporal structure, while text-only video memories lose fine-grained visual attributes. We observe a visual-textual duality: language memories carry long-range temporal structure better than dense frames, while pixels remain decisive for attribute-level perception. Building on this insight, we propose Caption-once, Frames-on-Demand (CFD), a budget-aware edge-cloud agentic framework. The edge runs a single offline captioning pass that builds a dual-track narrative index, an event-level story skeleton plus a clip-level micro-log, cached and reused across queries without re-captioning. At query time, a cloud-side MLLM reasons over the index in a story-first loop centered on a lightweight Visual-Need Router: a per-query gating module that triggers bounded keyframe retrieval only for perceptual questions (appearance, on-screen text, attribute disambiguation) and keeps temporal-structural questions in language space. The router turns visual access into a first-class, query-conditioned cost, capping per-query frame consumption regardless of video length. Experiments on long-video benchmarks demonstrate strong accuracy-efficiency trade-offs while substantially reducing online visual processing.

Key Observation: Visual-Textual Duality

Language memories carry long-range temporal structure better than dense frames, while pixels remain decisive for attribute-level perception.

Method

CFD separates offline narrative indexing on the edge from on-demand visual verification on the cloud.

Overview of CFD. A single offline captioning pass on the edge builds a dual-track narrative index; a cloud-side MLLM reasons over it in a story-first loop and retrieves raw frames on demand only when the Visual-Need Router determines pixel-level evidence is necessary.

Three-Tier Edge-Cloud Memory

• Event memory (global story skeleton). Semantically coherent events from a shot-boundary detector, each captioned into a structured narrative profile; always presented in full to the reasoning agent.
• Clip memory (local temporal details). Fixed-length clips captioned one action per sentence; activated selectively for the localized event rather than fed in full.
• Visual working memory (pixel-level evidence). A FIFO queue with a deliberately small fixed capacity that stores raw frames when pixel-level inspection is necessary.

Story-First Reasoning Loop

Up to T = 5 iterations; if none yields a confident answer, a must-answer fallback commits to the best answer using all accumulated evidence.

Results

Video-MME

CFD achieves 67.5 overall with only 5.8 frames per question on average, matching or exceeding all reported agent-based baselines; with Qwen3.5 (9B + 27B) as captioner and reasoner it reaches 69.4 with only 2.5 frames. We do not aim to match dense-inference upper bounds such as Qwen3-VL-32B at 768 frames; we target the accuracy-efficiency frontier under bounded visual budgets.

InfiniBench

CFD achieves the best Chronological Understanding score (55.10) at only 14.8 frames per question on average, exceeding the 768-frame dense baselines, and remains competitive on Scene Transitions; it stays below the dense Qwen3-VL on Global Appearance and Character Actions.

LVBench

CFD achieves 52.9 overall, outperforming every agent-based system except VideoLucy (58.8), whose agent base model is the far larger DeepSeek-R1, and remaining competitive with the leading open-source MLLM AdaReTaKe-72B (53.3).

Analysis

Component Analysis

Text alone cannot substitute for visual evidence (Global Appearance 9.30 with event memory alone, 38.00 with all clip captions). Always-on frames bring sharp gains on appearance-centric tasks at the cost of Chronological Understanding (51.70) and long-video accuracy (62.6); the Visual-Need Router recovers them to 55.10 and 63.6, a more balanced accuracy-efficiency trade-off.

Clip-caption activation

CFD uses 66.3% fewer clip captions on average while outperforming the all-clip baseline overall.

Frame reduction and routing

Even with the Router disabled, online frame usage drops from 768 to 6.63 frames/query (99.1% fewer, about 116×); the Router further reduces it to 5.76 (−13.1%) while improving accuracy from 67.2 to 67.5. Frame savings do not imply lower wall-clock latency in every configuration, because the prompt-based Router introduces an additional LLM call. Increasing the budget from 2/4 to 8/16 improves accuracy by 0.3 points, while 32/64 yields only another 0.1 point.

Routing policy

The Router improves average accuracy by 2.2 points over no-frame reasoning and slightly exceeds always-frame inference (67.5 vs. 67.2) with 13.1% fewer online frames. In a paired diagnostic rerun, it retrieves frames for 46.2% of questions and captures 127 of 133 (95.5%) cases in which frames empirically improve the answer; routing accuracy is 68.5% on the 1,876 decidable questions.

Iteration budget

Long-video accuracy keeps improving from 60.4 at T = 1 to 64.0 at T = 20; we adopt T = 5, which achieves the best overall score of 67.5.

Video-MME accuracy versus the maximum iteration budget T.

Offline captioner model size

Scaling the captioner 2B→8B brings a substantial gain (62.6→67.5 overall), while 32B degrades long-video accuracy (61.0 vs. 63.6), which we attribute to verbosity accumulation; we adopt Qwen3-VL-8B.

Qualitative Examples

Attribute perception: “What's the main color of Nuna?” (A. Gray · B. Blue · C. Green · D. Brown)

Story-first answer: abstains → Temporal localization [0, 505] s → Clip-enriched re-answer: still uncertain → Visual-Need Router: YES → Multimodal re-answer → D. Brown ✓

Inference trajectory. CFD reasons over coarse event memory, narrows the relevant temporal span with clip memory, and invokes sparse visual backtracking only when textual evidence remains insufficient.

Direct answer: “What does this video document?”

Story-first answer → B. Behind the scenes at the National Theatre ✓

Direct-answer mode. The coarse event memory already provides sufficient global semantic evidence, so CFD answers at the initial story-first stage without temporal backtracking or visual evidence retrieval.

BibTeX

@misc{cai2026cfd,
      title={Caption-once, Frames-on-Demand: Visual-Need Routing for Budget-Aware Agentic Long Video Understanding},
      author={Weitong Cai and Hang Zhang and Yukai Huang and Yiqiao Xie and Shan Gao and Jiankang Deng and Songcen Xu and Jifei Song and Zhensong Zhang},
      year={2026},
      eprint={2609.11899},
      archivePrefix={arXiv},
      primaryClass={cs.CV},
      url={https://arxiv.org/abs/2609.11899},
}