April 20, 2026
Logs, lies, and user agents
I prompted ChatGPT, Claude, Perplexity, and Gemini and watched my Nginx logs
Bots are knocking, humans are clicking, and the comments are on fire
TLDR: Server logs show major chatbots like ChatGPT and Claude actively fetch pages with their own bot IDs, proving a clear split between bot reads and real human clicks. Commenters cheered the experiment, debated prompt-injecting bots and header details, joked about costs, and argued over whether the article felt AI-written—because the logs don’t lie, but the vibes do.
A site owner ran a no-nonsense test: ask popular chatbots about his site and watch the server logs. The receipts? ChatGPT and Claude both hit the pages in real time with their own bot names, while Claude politely checked robots.txt first. Perplexity also fetched. That confirms two kinds of “AI traffic”: the bot reading your page vs a person clicking a link after seeing the answer. Simple, but spicy.
The comments lit up. One crowd cheered the DIY vibe—“looks fun, I’ll try it too!”—while the mischief-makers pounced: If the user-agent says ChatGPT-User, can I feed it a trap? Ethics alarms went off as others warned user-agents can be spoofed and that’s a slippery slope. The tinkerers zeroed in on headers, wondering what formats these bots accept—is anyone asking for markdown?—turning the thread into a mini lab. And the skeptics brought shade: one user called the writing “AI slop,” sparking a meta-fight about tone vs substance.
Money jokes flew—who’s paying for all this crawling and querying?—while normies appreciated the plain result: bots do knock, humans do click. In a world of hype, a little log-watching and some NGINX real talk felt weirdly refreshing—and wildly chaotic in the comments.
Key Points
- •The author used a custom nginx log format to capture detailed access data while prompting five AI products.
- •Two distinct signals were defined: provider-side origin fetches versus human clickthrough visits with referrers.
- •ChatGPT executed provider-side retrievals using the ChatGPT-User/1.0 user-agent, with bursty multi-IP requests and no referrer.
- •Claude executed provider-side retrievals using Claude-User/1.0, requested /robots.txt first, and followed redirects correctly, with no referrer.
- •Perplexity executed a direct fetch using Perplexity-User/1.0 with no referrer, indicating provider-side retrieval.