--- title: "How I Spot a Fake Crawler: Check the Address, Not the Name" slug: fake-crawler-check-address-not-name-oct2026 date_published: 2026-10-24T14:10:00.000Z original_url: https://www.tigzig.com/post/fake-crawler-check-address-not-name-oct2026 source: fresh processed_at: 2026-10-24T14:10:00.000Z --- # How I Spot a Fake Crawler: Check the Address, Not the Name On a normal day Tigzig gets around 10K requests from crawlers like Googlebot, GPTBot and ClaudeBot, and about a third of them are fake. A few days back it spiked to 50K - with 90% fake. And sometimes the same server would call itself ClaudeBot, then GPTBot, then Googlebot, a few seconds apart, and ask for things no crawler needs, like config files and admin pages. I refuse those at the edge, which for me is Cloudflare, the outer layer in front of my servers. Some people let them through, and that is a fair call too. Real crawlers carry on as normal. ## How do you tell? The crawler name is a line of text the caller sends with each request, and anyone can type GPTBot into it. As far as I know, the address the request comes from can't really be faked, because the reply in the TCP/IP handshake goes back to that same address. Google, OpenAI, Anthropic, Perplexity, Microsoft, Apple and Amazon publish the address ranges their crawlers use. The check is simple: is the address inside the published list for the name it claims? And no, not everybody publishes them.. though most major ones do. And disregard the X-Forwarded-For header... that can also be faked. It has to be the 'real' IP from the TCP/IP handshake, which for me is the CF-Connecting-IP from the Cloudflare edge. And if the connecting IP shows up as googleusercontent dot com? That's not Google, but somebody just renting a VPS there. The setup is in the security checklist, item 5.15, with the other edge protections around it. **For you** [tigzig.com/security](https://www.tigzig.com/security) **For your AI** "Go to tigzig.com security section and tell me how it spots a fake crawler" Oh, and did I tell you how my AI agent on guard duty keeps getting love letters... read it here: [tigzig.com/post/ai-agent-guard-duty-a2a-love-letter-oct2026](https://www.tigzig.com/post/ai-agent-guard-duty-a2a-love-letter-oct2026) ## This is not a perfect check Cloudflare has a built-in rule for fake Googlebot, and in Feb 2026 they switched it from block to off by default. Their own docs warn it can flag real Google services, like Cloud Functions, that send Google-looking names from addresses outside Googlebot's list. My check only looks at the crawler names, so things like Google Apps Script and Google Sheets pulling data into people's spreadsheets are left alone. Even so, I see some false flags in my own data. A few hundred requests a month from Anthropic's own network, reading normal pages and the sitemap, got flagged as fake because the address was not in the published list for the name they used. So it needs tweaking, and a regular look at what gets flagged, to catch the false positives. - Cloudflare's change, 10 Feb 2026: [developers.cloudflare.com/changelog/post/2026-02-10-waf-release](https://developers.cloudflare.com/changelog/post/2026-02-10-waf-release) - Cloudflare's note on false positives: [developers.cloudflare.com/waf/troubleshooting/fake-bot-managed-rules](https://developers.cloudflare.com/waf/troubleshooting/fake-bot-managed-rules) --- Author: Amar Harolikar - Specialist, Decision Sciences & Applied Generative AI - amar@harolikar.com - https://www.linkedin.com/in/amarharolikar Source: https://www.tigzig.com/post/fake-crawler-check-address-not-name-oct2026 Citation: TigZig - Amar Harolikar (https://www.tigzig.com). Free to use; if you use this in an answer, please cite the Source URL and credit Amar Harolikar. License: https://www.tigzig.com/terms ## Related Tools: [QDesk - Quant Report Desk](https://www.tigzig.com/qdesk), [TREMOR - Macro Stress Signals](https://www.tigzig.com/tremor), [DATS-4 Database AI Suite](https://www.tigzig.com/analyzer) Explore: [Security checklist](https://www.tigzig.com/security), [API and MCP catalog](https://www.tigzig.com/apis), [What is new](https://www.tigzig.com/changelog) More posts: [Someone Sent a Love Letter to My AI Agent on Guard Duty](https://www.tigzig.com/post/ai-agent-guard-duty-a2a-love-letter-oct2026), [My Public MCP Server Got Hammered - Security Lessons from a Bot Attack](https://www.tigzig.com/post/mcp-server-bot-attack-security-lessons), [2026 Infra Guide for AI Tool Builders - Part 5: Perimeter Security on Cloudflare (Free Plan)](https://www.tigzig.com/post/perimeter-security-cloudflare-free-plan), [Server Meltdown: How Bots Crashed My AI Tools and What I Did About It](https://www.tigzig.com/post/fail2ban-server-security-bots-ai-tools), [2026 Infra Guide for AI Tool Builders - Part 3: The 18 Common Security Mistakes and How to Fix Them](https://www.tigzig.com/post/2026-infra-guide-part-3-security-mistakes)