A one-hour video can fit inside a model's advertised context window and still be a terrible input. The problem is not only memory. It is that the model has to find one useful frame while hundreds of irrelevant frames compete for attention.

A new paper from researchers at the University of Illinois, Microsoft Research, and Google DeepMind takes a surprisingly small swing at that problem. ReToken adds one learned embedding and one projection matrix to an existing vision-language model. The model uses that token to rank visual frames, keeps the best few, then answers the question from the reduced cache.

ReToken retrieves relevant visual frames from a long visual context

The result is much larger than the patch. On Visual Haystacks, a benchmark that hides one relevant image among distractors, ReToken raises Qwen3VL-8B accuracy from 58.6% to 72.0% when the context contains 50 images. InternVL3.5-8B rises from 57.3% to 69.7% in the same setting. The gains are 13.4 and 12.4 percentage points, respectively. On LVBench, whose videos average 68 minutes, Qwen3VL-8B improves from 40.6% to 48.6% while processing 100 sampled frames.

Those are useful numbers, but the paper is more interesting for the diagnosis underneath them. The obvious way to retrieve frames is to use the model's query and key attention scores. That is what ReKV does. ReToken's authors tested the assumption and found that it is a poor visual search signal. In a long-video multiple-choice test, the attention-based retriever found the right frame only 5.1% of the time on average across layers for Qwen3VL-8B.

The better signal sits in the value vectors. In a transformer, keys help decide where attention goes. Values carry the content that gets passed along after that decision. ReToken treats those values as a searchable representation. Its learned token reads the question, gets contextualized by the visual cache, and is projected into the same space as the mean value vector for each frame. A cosine similarity score produces the ranking.

That sounds like a small implementation detail. It is not. In a controlled two-image test, a precise target phrase scored against value vectors reaches 78.0% recall at one, compared with 65.7% for the matching query-key method on Qwen3VL. InternVL3.5 shows the same pattern, 83.8% versus 78.8%. The paper's claim is not that attention is useless. It is that attention weights answer a different question. They describe aggregation for next-token prediction, not necessarily which visual item contains the evidence a user asked for.

The two-pass trick

ReToken does not throw away the original visual processing. It encodes the video once and stores the per-layer key-value cache. For every new question, the first pass appends the retrieval token and scores the cached frames. The second pass loads only the selected frames and generates the answer from that smaller visual context.

That split matters for repeated queries. The expensive video encoding takes about 14.7 seconds in the paper's Qwen3VL setup, but it happens once per video. With 16 retrieved frames, the retrieval phase takes 0.519 seconds per question and the answer phase takes 0.167 seconds. Uniform sampling takes 0.081 seconds to load its frames and 0.169 seconds to answer, so ReToken spends roughly 0.4 extra seconds per question for much better recall. If a system asks dozens of questions about the same recording, the cache begins to look like the actual product.

The method also uses an early-layer budget of 256 frames during retrieval, then narrows the answer stage to the selected K frames. That is an important caveat. ReToken does not magically make every long-video pass cheap. It makes ranking more selective after the model has built a cache, and it keeps the final answer pass small.

The paper trains the VLM mostly as-is. In the default setup, only the retrieval token and projection are updated. Qwen3VL-8B trains for three epochs on one H100 with an effective batch size of 64. The filtered training set contains 70,686 examples derived from multi-image question answering. Training the token is cheap because the backbone stays frozen. The authors also test tuning the first three layers, which improves image benchmark accuracy, but the frozen version transfers better to video. That is a useful warning against casually fine-tuning the whole model on image data and expecting temporal understanding to survive.

Where it works, and where it breaks

ReToken is strongest when the answer is localized and nameable. On LVBench, it adds 15.4 points for key information retrieval and 10.5 points for entity recognition. It adds only 3.6 points for temporal grounding and 1.9 points for event understanding. Summarization gets worse, dropping from 36.2% to 31.0%.

That failure makes sense. A single frame can contain a person's name or a specific object. It cannot summarize a forty-minute argument whose evidence is spread across time. ReToken is a search tool, not a replacement for temporal memory. If the question asks what happened before an event, the system may retrieve the event itself instead of the preceding frames. The authors call out set-based and temporally offset retrieval as open problems.

There is another catch in the cache itself. When many images are encoded into one persistent cache, later visual tokens may have already attended to earlier distractors. Even an oracle that loads the ground-truth image's cached representation loses accuracy as the original context grows. With 50 images, the clean re-encoded ground-truth image scores 86.8%, while the corresponding stored cache scores 82.3% after partial tuning. Retrieval can select the right evidence and still inherit contamination from the way the cache was built.

Compared with external systems such as SigLIP2, ReKV, and MIRAGE, ReToken has a neat systems advantage: the retriever and the answerer share the same VLM representation. There is no separate image encoder pass followed by a second multimodal encoding pass. That reduces moving parts, though it also ties retrieval quality to the quirks of the backbone. A dedicated retriever may still win when the corpus is huge, the visual domain is specialized, or the cache cannot stay warm.

The paper is new, and its public repository has only a handful of stars so far. That is not evidence against the method. It means deployment claims should stay modest until other teams reproduce the results on more backbones, real video streams, and questions that require temporal composition. The current evidence is strongest for a narrower proposition: when a frozen VLM already knows how to answer questions but loses the relevant frame in a crowded visual context, a learned retrieval token can recover a meaningful part of that lost signal.

I like the direction because it attacks the expensive part without demanding a new frontier model. The uncomfortable part is that the system still needs to process the video before it can decide what matters. The next useful test is not another image haystack. It is a live archive where frames arrive once, queries arrive later, and the system must preserve relationships between events instead of picking isolated winners.

Sources