July 7, 2026
Now measuring pure vibes
Weighing smoke: why AI visibility dashboards are mostly useless
People are calling these pricey AI scoreboards glorified horoscopes for marketers
TLDR: A new critique says many AI brand-tracking dashboards sell polished numbers without proving those numbers mean anything, especially since chatbots often give different answers to the same question. Commenters split between mocking the tools as expensive nonsense and admitting anxious companies will still buy them for boardroom comfort.
The article basically lobs a grenade at the booming business of AI visibility dashboards: those expensive tools that promise to tell brands how often they show up in chatbot answers. The author’s message is brutal in plain English: these neat little scorecards may look scientific, but they’re often measuring smoke, vibes, and wishful thinking. Community reactions were instantly spicy. One camp went full scorched earth, calling the tools "SEO snake oil in a fresh hoodie" and comparing them to paying for a weather app that just guesses. The biggest applause line? Research suggesting that if you ask chatbots the same question again and again, you almost never get the exact same list of brands twice. For critics, that was game over.
But the comments weren’t all one-note dunking. Some marketers pushed back, arguing that bosses still want something to show in meetings, even if the number is messy. That sparked the real drama: is a shaky metric better than no metric at all, or is that just fake certainty with extra charts? Skeptics kept winning the meme war. People joked that these dashboards are the modern version of reading tea leaves, with decimal points added to make the magic look official. Others said the entire market feels like a panic-buy by brands terrified of missing the next big shift in search. The mood across the crowd was a mix of laughter, dread, and the sinking realization that a lot of companies may be spending big money to measure something that can’t be pinned down in the first place.
Key Points
- •The article says AI visibility tracking tools have become a large market, with estimated annual spending above $100 million.
- •It argues that many dashboards rely on vendor-selected prompts, modeled query volumes, and sampled query surfaces that may not reflect real user behavior.
- •A study by Rand Fishkin and Patrick O’Donnell tested 12 prompts across ChatGPT, Claude, and Google’s AI 2,961 times and found highly inconsistent repeated results.
- •The article cites research showing only 45% to 59% overlap in returned brands across identical repeated prompts and describes generative-engine citation metrics as random variables.
- •Ahrefs is cited as finding that Google’s AI Mode and AI Overviews use different cited sources 87% of the time for the same query.