July 25, 2026
Bot fight at the click corral
Cloudflare's new AI traffic options for customers
Cloudflare says site owners get more AI control, but commenters are yelling “sure, Jan”
TLDR: Cloudflare rolled out new ways for website owners to control how AI-related bots use their content, including stricter default blocking in some cases. Commenters immediately turned it into a trust-and-hypocrisy brawl, questioning whether bots will obey and whether Google search could get caught in the crossfire.
Cloudflare just announced new options that let website owners decide how different kinds of automated visitors can use their sites. In plain English: site owners can now be more picky about whether bots that power search, AI training, or AI assistants get access. Cloudflare says this is about giving people more control in a world where AI tools can slurp up content without sending readers back. It’s also tightening rules so some AI-related traffic will be blocked by default on ad-supported pages for new domains.
But in the comments, the mood was less “finally, freedom” and more “is this actually real, or just vibes?” One of the loudest reactions mocked the whole thing as an “honors system,” basically accusing bad actors of simply ignoring the labels and doing whatever they want anyway. Another big flashpoint came from Simon Willison, who flagged a potentially huge consequence: if Google uses the same crawler for both search and AI training, then blocking training could also block Google search crawling. That’s the kind of twist that made readers do a double take.
Then there’s the simmering pay-per-crawl drama. People are openly annoyed that Cloudflare keeps name-dropping the program while many still seem stuck on a “request access” waiting list. And one commenter accused Cloudflare of “playing both sides” by helping people block AI while also selling AI-building tools. The jokes weren’t exactly meme-heavy, but the sarcasm was flowing freely: lots of exhausted, eye-rolling energy, with a side of “cool announcement, now where’s the part that actually works?”
Key Points
- •Cloudflare says it is expanding AI traffic controls beyond its earlier one-click Block AI Bots approach.
- •The company says it launched Block AI Bots and a Pay-Per-Crawl marketplace a year earlier under its Content Independence Day initiative.
- •The article argues that website owners want more nuanced control than simply blocking all automation, especially because smaller sites depend on discoverability.
- •Cloudflare says the distinction between AI and non-AI bots is becoming less useful as AI features become embedded in products such as search.
- •The updated approach described in the article focuses on classifying bots by behavior, including what they do on a site, what they store, and how they may reshare content.