How I Spot a Fake Crawler: Check the Address, Not the Name
Top•QDesk - Quant Report Desk•TREMOR - Macro Stress Signals•DATS-4 Database AI Suite

How I Spot a Fake Crawler: Check the Address, Not the Name

Published: October 24, 2026

Related toolsQDesk - Quant Report Desk•TREMOR - Macro Stress Signals•DATS-4 Database AI Suite

On a normal day Tigzig gets around 10K requests from crawlers like Googlebot, GPTBot and ClaudeBot, and about a third of them are fake. A few days back it spiked to 50K - with 90% fake.

And sometimes the same server would call itself ClaudeBot, then GPTBot, then Googlebot, a few seconds apart, and ask for things no crawler needs, like config files and admin pages.

I refuse those at the edge, which for me is Cloudflare, the outer layer in front of my servers. Some people let them through, and that is a fair call too. Real crawlers carry on as normal.

How do you tell?

The crawler name is a line of text the caller sends with each request, and anyone can type GPTBot into it. As far as I know, the address the request comes from can't really be faked, because the reply in the TCP/IP handshake goes back to that same address.

Google, OpenAI, Anthropic, Perplexity, Microsoft, Apple and Amazon publish the address ranges their crawlers use. The check is simple: is the address inside the published list for the name it claims?

And no, not everybody publishes them.. though most major ones do.

And disregard the X-Forwarded-For header... that can also be faked. It has to be the 'real' IP from the TCP/IP handshake, which for me is the CF-Connecting-IP from the Cloudflare edge.

And if the connecting IP shows up as googleusercontent dot com? That's not Google, but somebody just renting a VPS there.

The setup is in the security checklist, item 5.15, with the other edge protections around it.

For you

tigzig.com/security

For your AI

"Go to tigzig.com security section and tell me how it spots a fake crawler"

Oh, and did I tell you how my AI agent on guard duty keeps getting love letters... read it here: tigzig.com/post/ai-agent-guard-duty-a2a-love-letter-oct2026

This is not a perfect check

Cloudflare has a built-in rule for fake Googlebot, and in Feb 2026 they switched it from block to off by default. Their own docs warn it can flag real Google services, like Cloud Functions, that send Google-looking names from addresses outside Googlebot's list.

My check only looks at the crawler names, so things like Google Apps Script and Google Sheets pulling data into people's spreadsheets are left alone.

Even so, I see some false flags in my own data. A few hundred requests a month from Anthropic's own network, reading normal pages and the sitemap, got flagged as fake because the address was not in the published list for the name they used.

So it needs tweaking, and a regular look at what gets flagged, to catch the false positives.

Working on something similar? How I work covers the rates, the availability and what I take on.