May 14, 2026
Bot behavior or PR cleanup?
Amazonbot is finally respecting robots.txt
After years of scraping chaos, Amazon says it’ll finally take the hint
TLDR: Amazon says its web crawler will finally obey the standard “keep out” rules that website owners can post, a change many say is long overdue. The reaction was a mix of relief, mockery, and distrust, with commenters joking about blocking Amazon using Amazon’s own tools and asking why the bot was snooping around in the first place.
Amazon has apparently decided to stop acting like the internet’s most awkward party guest and finally listen to website owners who say “please don’t come in here.” The big update: starting June 15, 2026, Amazon says its crawler, Amazonbot, will follow the standard website rules file that tells bots where they can and can’t go. For the person who posted the news, this landed with a very specific sting: they say Amazon’s aggressive scraping was the reason they built defenses in the first place. So yes, the mood was less “thank you, Amazon” and more “you caused this mess!”
The comments, naturally, were where the real fireworks happened. One site owner said Amazonbot was hammering their weather site so badly they had to block it with Amazon’s own security tools, which drew instant applause for the sheer irony. Another commenter declared the whole rules system basically toothless, arguing that bots only behave if they feel like it, while others shared horror stories of Amazonbot getting trapped in endless loops and chewing through pointless pages like a confused Roomba.
And then there was the comedy. The tiny email footer line, “Get Outlook for Mac,” somehow became a star of its own, with people joking about whether the message was drafted in a rush and blasted out to everyone. Meanwhile, one blunt question summed up the vibe perfectly: why does Amazonbot even exist? That’s the real drama here — not just that Amazon is changing course, but that the community seems deeply unconvinced this redemption arc is sincere.
Key Points
- •The author says they received an email from Amazon announcing that Amazonbot will use robots.txt directives as the sole method for managing crawl preferences starting June 15, 2026.
- •According to the quoted email, Amazonbot crawl control will no longer rely on manual requests.
- •The email states that if a site does not implement robots.txt directives by the effective date, Amazonbot will follow standard web crawling practices.
- •Amazon says robots.txt can be used to control Amazonbot access at the page, directory, or site level and can be updated at any time.
- •The author says Amazon’s scraper prompted the creation of Anubis and that they plan to incorporate the robots.txt-related changes into it if needed.