Your LLM still can't watch video. This tool fixes that.

Your LLM can't watch a video. It reads transcripts, samples frames at random, and misses every fast cut. A new open-source tool fixes the part nobody talks about.

Paste a YouTube link into ChatGPT and you get a transcript summary. Ask Claude to analyze a video file and it errors out. Even Gemini, which can technically process video, samples at one frame per second by default. A thirty-second TikTok with twenty cuts gets maybe thirty frames, most of them duplicates.

The problem isn't that LLMs are bad at video. It's that nobody bothered to preprocess the video before feeding it to them.

claude-real-video solves this with a simple insight: don't send every frame, send the right ones. The tool uses scene-change detection instead of fixed-interval sampling, deduplicates frames with a sliding window algorithm, and pairs the result with a Whisper transcript. Everything runs locally. Nothing uploads to the cloud.

The difference shows up immediately. A ten-minute screencast with static slides collapses from six hundred near-identical frames down to maybe fifteen. A fast-cut product demo that would miss half its transitions with fixed sampling captures every visual change. The tool generates a folder with the selected frames, a MANIFEST.txt describing what's in each one, and optionally the full audio track. Drop that folder into any LLM and ask questions.

Why frame selection matters more than model size

The research backs this up. A Stanford team published T* at CVPR 2025, framing long-video understanding as a temporal search problem: given a question, find the five frames that actually answer it. Their LV-HAYSTACK benchmark showed that most video QA failures come from feeding models the wrong frames, not from the model being too small.

Apple's VBenchComp study found the same thing from a different angle. Many video QA benchmarks have a dirty secret: models can answer most questions without watching the video at all. Strong language priors let them guess correctly from the transcript alone. When you shuffle the frames, performance barely drops. The questions that actually require temporal understanding, the ones where frame order matters, are where models consistently fail.

This is exactly what claude-real-video addresses. By selecting frames based on scene changes rather than timestamps, it ensures the model sees the visual information that transcripts miss: a UI transition, a chart update, a physical demonstration. The sliding-window deduplication prevents the model from wasting context on repeated shots.

The token economics of watching video

Here's where it gets practical. Sending video frames to Claude or GPT-4V costs tokens. Lots of them. A forty-five minute lecture sampled at one frame per second produces two thousand seven hundred frames. At roughly fifteen tokens per image, that's over forty thousand tokens just for the visual data, before any text prompt.

claude-real-video cuts that by eighty to ninety percent. Scene detection plus deduplication might select three hundred frames for the same lecture. Same visual coverage, fraction of the cost. The tool's author ran the math: about a dollar per forty-five minute video, with transcription almost always free on Groq's tier.

The HN thread lit up with alternatives. Several commenters pointed out that Gemini handles video natively and is far more token-efficient. One noted that Gemini 3.1 Flash Lite processes an hour of video for twenty-four cents at API rates. That's a valid point. But Gemini requires sending your video to Google's servers. claude-real-video keeps everything local.

What it can't do

The tool has real limitations. Frames are not video. Motion, object permanence, and temporal continuity are things Claude cannot infer from a static contact sheet. If you need to understand how something moves, or track an object across a scene, keyframes won't help.

There's also the question of whether this is a temporary workaround. Multiple HN commenters predicted that native video understanding will ship in next-generation models. Gemini already does it. The feature request for Claude has over three thousand votes on GitHub. When Anthropic adds native video support, tools like this become unnecessary.

The naming is also a problem. Several commenters suggested renaming to something like "llm-real-video" since the tool works with any model, not just Claude. The author acknowledged this.

The bigger pattern

What makes this interesting isn't the tool itself. It's the pattern it represents: preprocessing data for LLMs instead of scaling the model to handle raw input. We saw this with RAG for documents, with frame selection for images, and now with scene detection for video. The most effective AI applications often aren't the ones with the biggest models. They're the ones that feed the right data to whatever model they're using.

The tool is MIT licensed, pip-installable, and works on macOS, Linux, and Windows. If you work with video content, whether lecture notes, competitive analysis, or content creation, it's worth five minutes to try.