New AI, videothinker, Achieves Breakthrough in Long-Form Video Understanding
A novel agentic VideoLLM is overcoming key limitations in artificial intelligence’s ability to comprehend extended video content, paving the way for more refined AI applications.
The persistent challenge of long-form video understanding – a notable hurdle for current Video large Language Models (VideoLLMs) – might potentially be nearing a solution. Researchers Chenglin Li, Qianglong chen (Zhejiang University), and Feng han (Fudan University), alongside Yin Xingxi, yan Gong, and others, have unveiled VideoThinker, a groundbreaking agentic VideoLLM. This new model distinguishes itself by overcoming the limitations of conventional static frame analysis through adaptive exploration of crucial video moments.
This work is particularly significant becuase it breaks a common “circular dependency” in the field. Previously, creating agentic training data required pre-existing video comprehension capabilities. VideoThinker,however,learns through synthetic tool interaction trajectories generated in caption space and then grounded in video. According to a senior official involved in the project, “This approach allows the model to develop reasoning skills without needing to already ‘understand’ the video content.” By training on this uniquely constructed dataset, the model demonstrates substantially improved dynamic reasoning, temporal awareness, and multi-step tool use, ultimately surpassing existing methods on established long-video benchmarks.
The research team achieved this breakthrough by training VideoThinker entirely on synthetic tool-interaction trajectories – a unique approach that eliminates the need for pre-existing strong long-form video comprehension. This innovative method converts videos into detailed captions and employs a powerful agentic language model to generate multi-step tool-use sequences within that caption space, effectively creating a large-scale dataset for interleaved video and tool reasoning.
Grounding Reasoning in Visual Data
The core innovation lies in grounding thes trajectories back to video by replacing captions with corresponding frames. This process yields a dataset that doesn’t require the underlying model to initially possess strong long-form video understanding. This synthetic data equips VideoThinker with dynamic reasoning capabilities, allowing it to adaptively explore key moments in videos and utilize multi-step tools effectively. Experiments demonstrate that VideoThinker significantly outperforms both caption-only language model agents and strong video model baselines across established long-video benchmarks. Specifically, the model achieves a +6.8% improvement on MLVU and a +10.6% improvement on LVBench compared to standard VideoLLMs.
VideoThinker leverages two key agentic tools: Temporal Retrieval, which identifies potentially relevant temporal intervals using audio transcripts, scene descriptions, and summaries, and Temporal Zoom, which allows for detailed inspection of intervals through subtitles or frames. The team also measured the performance of SubtitleRetrieval, employing Whisper to transcribe video audio and retrieve relevant subtitle segments with timestamps, and subtitlesummary, built upon Qwen3-30B, which generates concise, query-focused summaries of complete subtitle transcripts. FrameZoom extracts raw frames within specified temporal intervals, resampling to increase visual density – for example, retrieving 8 frames from a 10-second interval within a 32-frame video.
VideoThinker is trained entirely on synthetic data, generated by converting videos into rich captions and utilizing a powerful language model to simulate multi-step tool use sequences within caption space.This innovative approach bypasses the need for pre-existing long-form video comprehension capabilities, resolving a common circular dependency in agentic video understanding research.
The resulting large-scale dataset interweaves video and tool reasoning, enabling VideoThinker to demonstrate dynamic reasoning, adaptive temporal exploration, and proficient multi-step tool use, significantly outperforming both caption-only language models and established video model baselines on long-video benchmarks. The authors acknowledge that the model’s performance is dependent on the quality of the synthetic data generation process and the capabilities of the underlying language model. Future research coudl explore more sophisticated synthetic data generation techniques and investigate the transferability of these agentic capabilities to other video understanding tasks.
