August 3, 2026
Lights, camera, comment war
Frame selection is the whole game: notes on making LLMs watch video
AI can only watch about 150 video snapshots, and commenters are roasting the whole idea
TLDR: The project says teaching AI to understand video comes down to choosing about 150 key images carefully, because showing it everything is too expensive. Commenters quickly turned that into a bigger fight over whether the article sounded machine-written and whether still images are even the right way to teach a bot to “watch” anything.
A developer’s big lesson from trying to make an artificial intelligence system “watch” video is surprisingly simple: it can only afford to look at a tiny number of images, so picking the right ones is everything. In plain English, the project argues that if you feed the bot the wrong snapshots, it’s not really watching a video at all — it’s just guessing from a glorified slideshow. That part landed. But the comments? That’s where the popcorn starts flying.
The loudest reaction wasn’t even about the code — it was about the writing. One commenter delivered the kind of drive-by roast that powers the internet, basically asking whether anything about language models can be written without sounding like it was generated by one. Ouch. Another took the debate in a completely different direction, saying maybe this whole “pick frames” strategy is flawed and that movement itself, not still images, is what really matters when humans watch video. In other words: some readers think the project is clever, others think it may be solving the wrong problem.
Then there was the cooler-headed camp pointing people toward the GitHub README, saying it actually feels more human and is more fun to read than the article itself. So yes, the project is about helping AI notice what really happened in a video — the blink-and-you-miss-it error message, the suspicious jump cuts, the stuff a human summary leaves out. But the community turned it into a mini-drama about whether AI writing already has a recognizable “voice,” and whether we’re teaching bots to watch videos the wrong way in the first place.
Key Points
- •The article argues that giving a vision LLM direct video input preserves visual and temporal details that may be lost in a human-written summary.
- •A practical vision LLM budget is described as roughly 100 to 150 images per video, making frame selection the main optimization problem.
- •Uniform time-based frame sampling is presented as ineffective because it both over-samples repetitive footage and misses brief important events.
- •The pipeline uses ffmpeg scene scores plus a low-density floor, then adapts scene thresholds using a rolling average rather than a fixed cutoff.
- •Deduplication evolved from a single comparator to multiple channels, including a global channel based on 16×16 RGB signatures and a sliding comparison window, plus an action-focused channel to avoid missing small moving subjects.