July 7, 2026
Botched by the bots
Sites that block AI training crawlers mostly ignore the answer time bots
Websites blocked AI ‘study bots’ but forgot the ones showing up for live answers — and commenters are roasting the irony
TLDR: The study says many websites blocked AI bots used for training years ago but mostly ignored the newer bots that grab pages live to answer questions, which matters because those visits may soon cost money. Commenters, though, were fixated on the article sounding like AI writing itself, turning the whole thing into an irony pile-on.
This story had all the ingredients for a serious internet policy debate: a data study, a legal backdrop, and a new way for websites to charge artificial intelligence companies on the spot when their bots grab a page to answer someone’s question in real time. The big finding was simple: lots of sites rushed to block the bots used to train artificial intelligence back in late 2023, especially after OpenAI launched GPTBot and The New York Times sued. But almost nobody updated those rules to deal with the newer bots that appear later, when a live answer is being generated for a user.
But the real fireworks were in the comments, where readers basically ignored the policy angle and went straight for the author’s throat. The loudest reaction? The article itself was accused of sounding like AI-generated mush. One commenter sneered that it read like the writer “couldn’t be bothered” to write it themselves. Another delivered the bluntest possible review: “AI slop.” A third twisted the knife by saying an article complaining about AI’s effect on the web probably shouldn’t look like “very obvious AI copy.” Ouch.
That’s the delicious irony everyone latched onto: a piece warning that the web is sleepwalking into the “answer era” got hit with the classic internet comeback — buddy, your post sounds like the very thing you’re warning us about. So yes, there’s a real issue here about websites preparing for the wrong fight. But in the court of public opinion, the headline battle quickly became: is the internet being overrun by AI, or are readers now so primed for it that any stiff writing gets instantly convicted?
Key Points
- •The article studied robots.txt files from the top 10,000 websites and found 5,577 readable files.
- •Among 861 sites with dateable GPTBot rules, 38% added those rules in the last quarter of 2023, and half did so within roughly six months of GPTBot's launch.
- •The article links the late-2023 surge in GPTBot rules to OpenAI's launch of GPTBot and The New York Times lawsuit against OpenAI.
- •Of the 430 sites that added GPTBot rules during the early surge, 87% later updated their robots.txt files, usually to add more training-crawler rules.
- •The article argues that websites largely wrote rules for training crawlers but paid much less attention to answer-time bots that fetch pages in real time for user queries.