Never by the user-agent. It is free text, anyone can copy it, and "Googlebot" in a header is a claim rather than a fact. There are two ways to actually check, and one of them is new.
1. Verify by the address. Take the address the request came from, do a reverse DNS lookup, confirm the name belongs to the vendor's own domain, then resolve that name forward again and confirm it returns the address you started with. Both directions matter, because either alone can be spoofed. Several vendors also publish their ranges as a JSON file you can refresh on a schedule.
The weakness of that method is staleness: a published range is a list that rots, and vendors move addresses without telling you. So pin to the registered allocation where you can rather than to the addresses you happen to have seen, and re-fetch the published lists rather than copying them into your code.
What fake crawlers look like in practice. On a normal day tigzig.com gets around 10,000 requests from crawlers such as Googlebot, GPTBot and ClaudeBot, and about a third of them are fake. On one day in October 2026 that rose to 50,000, with 90% fake. Some single servers called themselves ClaudeBot, then GPTBot, then Googlebot a few seconds apart, and asked for config files and admin pages that no crawler needs. Tigzig refuses those at the Cloudflare edge. Letting them through is also a fair call.
Check the real connecting address. The X-Forwarded-For header can be faked as well. Behind Cloudflare, the address to check is CF-Connecting-IP. An address that resolves to googleusercontent.com belongs to someone renting a server on Google Cloud.
Expect some false flags, and review them. Cloudflare changed its own "Fake Google Bot" managed rule from block to disabled on 10 February 2026 (changelog), and its docs say Google Cloud services such as Cloud Functions can send Google-looking user agents from outside Googlebot's ranges (false positives). Checking only requests that claim a crawler name leaves Google Apps Script and Google Sheets alone. Even so, a few hundred requests a month from Anthropic's own network, reading ordinary pages and the sitemap, got flagged because the address was not in the published list for the name used. Tigzig's setup is item 5.15 of the security checklist. Full post: https://www.tigzig.com/post/fake-crawler-check-address-not-name-oct2026.
2. Check the signature, if the agent signs. Some AI agents now sign each request with a private key and publish the matching public key at a well-known address on their own domain, built on HTTP Message Signatures (RFC 9421). The request carries three headers: who claims to be signing, what was signed and with which key, and the signature. You fetch the public key from the directory that claim names, rebuild the signed string exactly as the standard specifies, and check it. Nobody can forge that without the key, and there is no address list to go stale.
Verify for visibility first, and do not grant privilege yet. The signature covers the host and the method but not the path or the body, and it stays valid for a window - so a captured set of headers can be replayed on a different path for that long. Verification for visibility is safe today. Verification for authorisation needs replay protection first, which means remembering nonces for the validity window.
Two habits that keep this honest. Run a negative control before you trust your verifier: change one byte of the host and it must reject. And never describe a count of signed requests as "verified" until the maths has actually run, because a directory name in a header is a claim like any other.
One framing point. These signed callers are usually not crawlers indexing you. They are people asking an AI to fetch something, and the agent often names its purpose in the user-agent, so they belong with your visitors rather than with your bot list.
Related: blocking SEO crawlers without blocking Google and the AI assistants, and the perimeter section of the checklist.
Building something like this? How I work covers the rates, the availability and what I take on.