Ferret-UI Lite: Lessons from Building Small On-Device GUI Agents

Ferret‑UI Lite is a bite‑size on‑device assistant that watches your screen like a human and tries to click its way through apps. The team says this 3B model (small by AI standards) learns by thinking step‑by‑step and trial‑and‑reward, and posts some eye‑catching scores—over 90% on spotting things on test screens and roughly 20–28% success on full navigation tasks. The paper is here if you like receipts: direct to paper.

But the real action? The comments. One camp is Team Tiny, cheering fast, local models that follow instructions inside your app. As one dev put it, they were “impressed at the speed and accuracy” and loved that it can translate a written instruction into tool use. That’s the Apple‑pilled crowd: keep it small, keep it on device, keep it useful.

Then the Accessibility Purists stormed in. “Why are we screenshotting the screen at all?” asks one commenter, blasting the “long way around” of visual recognition. Apple apps already have accessibility labels—basically a hidden map of the interface—and a throwback shout to Apple’s 1990s “Virtual User” had everyone dropping “back in my day” memes.

So the fight is set: see like a human vs. read the app’s map. Meanwhile, link‑droppers are tossing the paper like confetti, and everyone agrees on one thing: if tiny agents can actually navigate reliably on device, your phone might soon have a real tap‑happy co‑pilot.