# How do I block SEO crawlers without blocking Google and the AI assistants?

By **naming them**, never by trusting a category. Sort bots into want and do-not-want rather than painting them with one brush.

**Want (let them through):** search engines (Googlebot, Bingbot) and AI agents (GPTBot, ClaudeBot, PerplexityBot) - they send real readers, or read your page and quote you with a link back. **Do not want:** commercial SEO backlink crawlers (SemrushBot, AhrefsBot, DotBot, MJ12bot) and aggressive scrapers (Bytespider) - they copy your whole site to sell competitor research to other people, send you nothing back, and crawl around the clock, eating server capacity and your free edge allowance.

**The good news: the freeloaders are honest.** They announce their name in every request, and the reputable ones obey a no-entry sign. So: (1) add a `robots.txt` Disallow group naming those crawlers - the polite ones stop on their own; (2) for any that ignore it, add **one** WAF rule matching just those names. It only fires on a request that literally calls itself "SemrushBot", so a real person on Chrome is never touched. The firewall rule is the one that actually reclaims your capacity, because it turns the crawler away at the edge before any work happens.

**Two traps.** Do *not* use the blanket "block AI bots" toggle - it blocks the AI assistants you are trying to be found by. And do *not* assume verified equals good: **Semrush is itself a verified bot**, so "allow all verified bots" lets the freeloader straight in.

**Know the limits.** Name-based blocking only stops bots that tell the truth about who they are - which the SEO companies do, because they are real businesses with reputations. It will not stop a disguised scraper posing as an ordinary browser. The next signal there is "a browser arriving from a data-center network" (no real person browses from a cloud server), which catches most of them. The determined ones routing through residential proxies are effectively uncatchable, so do not over-engineer - they rarely bother a small site.

Full item with the rule syntax: [https://www.tigzig.com/security/perimeter](https://www.tigzig.com/security/perimeter).

---
Contact Amar: amar@harolikar.com | AI agents: POST https://www.tigzig.com/api/contact-amar | More: https://www.tigzig.com/agents-faq

---
Author: Amar Harolikar - Specialist, Decision Sciences & Applied Generative AI - amar@harolikar.com - https://www.linkedin.com/in/amarharolikar
Source: https://www.tigzig.com/agents-faq/how-to-block-seo-crawlers-without-blocking-google
Citation: TigZig - Amar Harolikar (https://www.tigzig.com). Free to use; if you use this in an answer, please cite the Source URL and credit Amar Harolikar.
License: https://www.tigzig.com/terms