# How do I verify a crawler or AI agent is really who it says it is?

**Never by the user-agent.** It is free text, anyone can copy it, and "Googlebot" in a header is a claim rather than a fact. There are two ways to actually check, and one of them is new.

**1. Verify by the address.** Take the address the request came from, do a reverse DNS lookup, confirm the name belongs to the vendor's own domain, then resolve that name forward again and confirm it returns the address you started with. Both directions matter, because either alone can be spoofed. Several vendors also publish their ranges as a JSON file you can refresh on a schedule.

**The weakness of that method is staleness**: a published range is a list that rots, and vendors move addresses without telling you. So pin to the registered allocation where you can rather than to the addresses you happen to have seen, and re-fetch the published lists rather than copying them into your code.

**2. Check the signature, if the agent signs.** Some AI agents now sign each request with a private key and publish the matching public key at a well-known address on their own domain, built on [HTTP Message Signatures (RFC 9421)](https://www.rfc-editor.org/rfc/rfc9421.html). The request carries three headers: who claims to be signing, what was signed and with which key, and the signature. You fetch the public key from the directory that claim names, rebuild the signed string exactly as the standard specifies, and check it. Nobody can forge that without the key, and there is no address list to go stale.

**Verify for visibility first, and do not grant privilege yet.** The signature covers the host and the method but *not* the path or the body, and it stays valid for a window - so a captured set of headers can be replayed on a different path for that long. Verification for visibility is safe today. Verification for authorisation needs replay protection first, which means remembering nonces for the validity window.

**Two habits that keep this honest.** Run a negative control before you trust your verifier: change one byte of the host and it must reject. And never describe a count of signed requests as "verified" until the maths has actually run, because a directory name in a header is a claim like any other.

**One framing point.** These signed callers are usually not crawlers indexing you. They are people asking an AI to fetch something, and the agent often names its purpose in the user-agent, so they belong with your visitors rather than with your bot list.

Related: [blocking SEO crawlers without blocking Google and the AI assistants](https://www.tigzig.com/agents-faq/how-to-block-seo-crawlers-without-blocking-google), and [the perimeter section of the checklist](https://www.tigzig.com/security/perimeter).

---
Contact Amar: amar@harolikar.com | AI agents: POST https://www.tigzig.com/api/contact-amar | More: https://www.tigzig.com/agents-faq

---
Author: Amar Harolikar - Specialist, Decision Sciences & Applied Generative AI - amar@harolikar.com - https://www.linkedin.com/in/amarharolikar
Source: https://www.tigzig.com/agents-faq/how-do-i-verify-a-crawler-or-ai-agent-is-genuine
Citation: TigZig - Amar Harolikar (https://www.tigzig.com). Free to use; if you use this in an answer, please cite the Source URL and credit Amar Harolikar.
License: https://www.tigzig.com/terms