Perimeter and Edge Protection (Cloudflare)

The Risk

DNS records set to "grey cloud" (DNS-only) expose your server's real IP address. Once an attacker knows the IP, they can bypass Cloudflare entirely - all DDoS protection, WAF rules, and rate limiting become useless. They hit your server directly. This is the #1 real-world IP spoofing risk: without Cloudflare in the path, CF-Connecting-IP can be trivially spoofed (it's just an HTTP header), and your get_client_ip function that trusts it first will accept any value the attacker sends.

The Solution

In your Cloudflare DNS settings, make sure every record has the orange cloud icon (proxied mode) turned on - not the grey cloud (DNS-only). When proxied, all traffic goes through Cloudflare first, which hides your server's real IP address and applies all your security rules. With the grey cloud, your server's IP is visible to anyone. For defense in depth: configure your server firewall to ONLY accept connections from Cloudflare's published IP ranges, and enable Cloudflare Authenticated Origin Pulls. This way, even if the origin IP leaks, direct connections are rejected.

The Fix

All DNS records must be orange cloud (proxied).
In Cloudflare DNS settings, toggle the proxy
icon to orange for every A/AAAA/CNAME record.

The Risk

Some subdomains must be grey cloud (DNS-only) because third-party services like Auth0, Clerk, or AWS RDS require direct CNAME resolution. These subdomains bypass all Cloudflare protection - no rate limiting, no WAF, no IP blocking. If you don't track which domains are grey and why, you lose visibility into your actual attack surface.

The Solution

Maintain a list of every grey-clouded subdomain and the reason it must be grey (e.g., "auth.example.com - Auth0 requires direct CNAME"). Periodically audit this list: if a service is decommissioned, either delete the DNS record or switch it to orange cloud. Ensure no origin servers (your VPS, your hosting) are ever exposed through grey-cloud records - only third-party CNAMEs should be grey.

The Fix

# Audit grey-cloud records via Cloudflare API
# List all DNS records and filter for proxied=false
curl -s "https://api.cloudflare.com/client/v4/zones/{zone_id}/dns_records" \\
  -H "Authorization: Bearer {token}" | \\
  jq '.result[] | select(.proxied==false) | {name, type, content}'

# Expected grey cloud: third-party CNAMEs only
# auth.example.com    -> Auth0 tenant (required)
# clerk.example.com   -> Clerk frontend (required)
# db.example.com      -> managed DB (required)

# NOT acceptable: your VPS IP exposed via grey cloud
# app.example.com -> 1.2.3.4  (must be orange!)

The Risk

Bots and automated tools use fake or missing User-Agent headers. Without Browser Integrity Check, these requests reach your application unchallenged. The Security Level setting controls how aggressively Cloudflare challenges suspicious IPs - the default is too permissive for sites under active attack.

The Solution

Turn on three settings in Cloudflare: Browser Integrity Check (blocks requests with suspicious or missing browser headers), Bot Fight Mode (challenges known bot fingerprints), and set the Security Level to Medium or High (challenges visitors from IP addresses with a bad reputation). These are toggle switches in your Cloudflare dashboard - no code needed.

The Fix

Cloudflare dashboard > Security > Settings:

- Browser Integrity Check: ON
  (blocks suspicious User-Agent headers)
- Security Level: "Medium" or "High"
  ("I'm Under Attack" mode for active DDoS only)
- Bot Fight Mode: ON
  (challenges known bot fingerprints)

The Risk

Browser Integrity Check and Bot Fight Mode catch known bad signatures, but many automated tools (scrapers, vulnerability scanners, credential stuffers) use real browser headers. They pass basic checks and hit your frontends unchallenged. Without a JS challenge, any script that can send HTTP requests can access your application pages.

The Solution

Create a Cloudflare WAF custom rule that applies a JS Challenge to all your frontend domains. Real browsers solve the challenge transparently (invisible to users). Automated scripts and bots cannot execute JavaScript, so they get blocked with a 403. List all your frontend hostnames in the rule expression. If you have API paths that legitimate bots need to access (AI agents, webhooks, health checks), add path exclusions to the same rule so those paths return 200 without a challenge.

The Fix

Cloudflare dashboard > Security > WAF > Custom rules

Action: JS Challenge
Expression example (multiple frontends):
  (http.host eq "app.example.com") or
  (http.host eq "dashboard.example.com") or
  (http.host eq "www.example.com" and
    not starts_with(http.request.uri.path, "/api/webhook") and
    not starts_with(http.request.uri.path, "/robots.txt"))

# Browsers: pass transparently (no visible CAPTCHA)
# Bots/scrapers: blocked (cannot execute JS)
# Excluded paths: return 200 to all clients

The Risk

Cloudflare's free tier provides DDoS protection at the network level but NOT HTTP-level (L7) rate limiting by default. Without a WAF rate limit rule, an attacker can flood your application endpoints with legitimate-looking HTTP requests that pass through Cloudflare unblocked.

The Solution

Create a rate limiting rule in Cloudflare's WAF that blocks any IP address exceeding a threshold (e.g., 50 requests per 10 seconds). This stops HTTP-level floods before they reach your server. The free tier allows one rate limit rule with a minimum 10-second window. Make it zone-wide so it covers all subdomains.

The Fix

Cloudflare dashboard > Security > WAF > Rate limiting rules

Rule: 50 requests per 10 seconds per IP
Action: Block for 10 seconds
Characteristics: MUST include cf.colo.id

Expression: (true)  - applies to all subdomains

Note: Free tier = 1 rule only, 10s minimum period.

The Risk

The free WAF rate limit (one rule, one threshold) treats all domains equally. But frontends need higher limits than backends - a React app sends 30-40 requests on page load (JS, CSS, images, API calls), while a backend API might only need 10 requests per minute per user. A single threshold either blocks legitimate frontend users or leaves backends wide open.

The Solution

Deploy a Cloudflare Worker that intercepts traffic at the edge and applies different rate limit thresholds per domain. Use Cloudflare's native rate limit bindings (defined in wrangler.toml) so counters are managed at the edge with zero additional latency. Map each domain to a tier - for example, frontends at 150 requests/60 seconds, backends at 10 requests/60 seconds, sensitive endpoints at 20 requests/60 seconds. Add a wildcard catch-all so new domains automatically get a moderate default.

The Fix

# wrangler.toml - define rate limit tiers as bindings
[[unsafe.bindings]]
name = "BACKEND_STRICT"
type = "ratelimit"
namespace_id = "1"
simple = { limit = 10, period = 60 }

[[unsafe.bindings]]
name = "FRONTEND_STANDARD"
type = "ratelimit"
namespace_id = "2"
simple = { limit = 150, period = 60 }

# src/index.js - map domains to tiers
const DOMAIN_CONFIG = {
  "api.example.com":       { limiter: "BACKEND_STRICT" },
  "dashboard.example.com": { limiter: "FRONTEND_STANDARD" },
  // ... add each domain
};

// Worker checks rate limit, returns 429 if exceeded
// Domains not in config get a moderate catch-all tier
// Deploy: npx wrangler deploy
// Add Worker Route per domain in Cloudflare dashboard

The Risk

When you identify a malicious IP or subnet (from attack logs, vulnerability scanners, or brute-force attempts), blocking it at the application level still lets the traffic reach your server - consuming bandwidth and CPU for every rejected request. The attacker can also try other subdomains or endpoints.

The Solution

Block malicious IPs at the Cloudflare zone level using IP Access Rules. Traffic from blocked IPs is dropped at the edge before it reaches your server - zero bandwidth, zero CPU cost. Block entire /24 ranges when an attacker operates from a known bulletproof hosting provider. Document each block with the reason and evidence so you can audit and clean up later. Deciding WHICH networks to block, and for how long, matters as much as the mechanics. Before blocking a whole network, sort it by what it actually is. A network on a vetted criminal list (see 5.10) is safe to block wholesale. A mainstream cloud (AWS, Azure, Google, big VPS hosts) is the opposite: real users and the AI/search bots you want live there too, so block only the individual offending IP, never the whole network. A consumer ISP is never a whole-network block, or you wall off real people on infected home machines. That leaves grey, abuse-tolerant cheap hosts, where you use judgment and verify first: check your own logs for any legitimate (2xx) traffic ever from that network. Zero legit traffic means blocking it costs you nothing; any real traffic means drop down to per-IP. Match the block durability to where the badness lives, because a public IP is not permanently tied to one person. Datacenter and VPS IPs are pooled and reassigned: an attacker holds an IP today, gives it up, and weeks later an innocent customer inherits it. So a single IP should only ever be blocked temporarily and auto-expire, or you eventually wall out a stranger who did nothing. A whole network can stay blocked far longer, because there the badness belongs to the provider, not one renter, and innocent people do not host on abuse-run networks. The ladder: a repeat single-IP offender gets escalating but still auto-expiring blocks; a grey subnet under review gets a medium, auto-expiring block so a wrong call self-corrects; only a confirmed bad provider earns a standing, indefinite block with periodic review.

The Fix

# Block an IP range at Cloudflare edge
curl -X POST "https://api.cloudflare.com/client/v4/zones/{zone_id}/firewall/access_rules/rules" \\
  -H "Authorization: Bearer {token}" \\
  -H "Content-Type: application/json" \\
  -d '{
    "mode": "block",
    "configuration": {
      "target": "ip_range",
      "value": "185.177.72.0/24"
    },
    "notes": "Bulletproof hosting. Path traversal attack on 2026-03-01."
  }'

# List all blocked IPs/ranges
curl -s "https://api.cloudflare.com/client/v4/zones/{zone_id}/firewall/access_rules/rules" \\
  -H "Authorization: Bearer {token}" | jq '.result[] | {value: .configuration.value, notes}'

The Risk

Automated scanners hammer every site looking for leaked secrets and misconfigurations - requesting paths like /.env, /wp-config.php, /.git/config, /.aws/credentials. A normal 404 tells them nothing and they keep probing. The day one of those files is genuinely exposed, it is game over. You want to catch the scanner the instant it reveals itself, not after it finds something.

The Solution

Serve deception endpoints at well-known scanner paths. No human or legitimate app ever requests /.env - so a single hit is a 100%-confidence attack signal. On a hit, block the source IP at the edge immediately and return convincing-but-fake bait (so the attacker wastes time analysing junk instead of realising they were detected). Run the check before your rate limiter so a scanner is caught on request number one, not after it has burned through a rate-limit window.

The Fix

// At the edge (Cloudflare Worker) or in middleware, before rate-limiting:
const BAIT_PATHS = ["/.env", "/wp-config.php", "/.git/config",
                    "/.aws/credentials", "/phpinfo.php"]; // keep your real list private

if (BAIT_PATHS.some(p => url.pathname === p || url.pathname.includes(".env"))) {
  // 1) block the IP at the edge (fire-and-forget)
  blockIpAtEdge(clientIp, "honeypot: " + url.pathname);
  // 2) return fake bait with 200 (not 403 - don't confirm it's special)
  return new Response("DB_PASSWORD=REDACTED\\nSECRET_KEY=REDACTED",
                      { status: 200, headers: { "Content-Type": "text/plain" } });
}

The Risk

A scanner that avoids your honeypot paths still floods you, enumerating hundreds of URLs and racking up 4xx responses. Edge rate limits slow each burst, but the same IP keeps coming back, and every request still costs you an edge-Worker invocation or a server hit. You want repeat offenders promoted from "throttled every time" to "dropped before they reach anything". But there is a subtler trap in how you DETECT them: not every abusive request even reaches your application logs. Anything your CDN already blocked at the edge never touches your server, so it is never logged. Some backends do not write logs at all. And a flood of perfectly "successful" 200s leaves no 4xx trail to spot. A jail that reads only your own app logs is blind to all three.

The Solution

Run a small scheduled job that finds IPs which crossed a velocity threshold within a short rolling window and promotes them to an edge IP block, so the next requests from that IP are dropped at the perimeter, not merely rate-limited. The choice that matters most is WHERE you read the signal from. Prefer your CDN's analytics API over your own application logs. The CDN sees every single request, its real status code, and the real client IP, including the traffic it blocked at the edge and the traffic to backends that never log a line. Reading from there instead of your logs closes all three blind spots above in one move. Auto-expire each block after a set period so stale rules don't pile up, and lengthen the block for repeat offenders.

The Fix

# Cron job (every minute or two). Read your signal from the CDN's analytics
# API, NOT just your app logs. The CDN also sees edge-blocked traffic,
# non-logging backends, and floods of "successful" 200s:
#
#   query the CDN analytics API: group the last N minutes of requests by
#   client IP, keep the IPs over your threshold (4xx velocity, and/or raw
#   request volume to catch 200-floods), then push each to the edge blocklist.
#
# If you must fall back to app logs (blind to edge-blocked / non-logging traffic):
SELECT client_ip, count(*) AS hits
FROM request_logs
WHERE created_at > now() - interval '5 minutes'
  AND status_code >= 400 AND status_code < 500
  AND client_ip NOT IN (SELECT cidr FROM allowlist)   -- see note
GROUP BY client_ip
HAVING count(*) >= :threshold;

# For each offender: create an edge IP block (with an expiry you track),
# then a second pass deletes edge rules whose expiry has passed.

The Risk

You can only block IPs you have personally watched attack you - but the same hosts are hitting thousands of sites simultaneously. Reacting one IP at a time keeps you permanently a step behind the networks that exist purely to host abuse.

The Solution

Sync vetted, free threat-intelligence feeds to your edge as block rules on a daily schedule - Spamhaus DROP ("Do not Route or Peer") lists entire networks so malicious they should be dropped wholesale, and it is conservative enough to trust for blocking. Optionally close the loop: report the scanners your honeypot catches to a community abuse database so other operators benefit from your detections.

The Fix

# Daily cron: fetch a vetted feed and sync to your edge blocklist
curl -s https://www.spamhaus.org/drop/drop_v4.json \\
  | jq -r '.cidr' \\
  | while read net; do
      upsert_edge_block "$net" "Spamhaus DROP - daily sync"
    done

# (Optional) report a honeypot-caught scanner back to the community
curl -s https://api.abuseipdb.com/api/v2/report \\
  -H "Key: $ABUSEIPDB_KEY" \\
  --data-urlencode "ip=$SCANNER_IP" --data "categories=21"  # web app attack

The Risk

Every edge protection - WAF, rate limits, bot rules, IP blocks - only applies to traffic that actually goes THROUGH Cloudflare. But your origin server still has a public IP, and that IP is NOT a secret: services like Shodan and Censys scan the whole internet and catalogue every server from its TLS certificate, and old DNS records leak it. An attacker who connects straight to the origin IP - just putting a valid hostname in the TLS SNI - reaches your application with Cloudflare entirely out of the path. Every edge defense is skipped, including the rate limits that would otherwise stop a flood. A common false-comfort here: a bare-IP request with no/blank hostname often returns a connection error, which looks like "the origin refuses direct access" - but that is just the web server rejecting an unmatched hostname, NOT a firewall. With a valid hostname it answers anyone.

The Solution

Restrict your origin's inbound HTTPS port to Cloudflare's published IP ranges ONLY, so any direct-to-IP connection is dropped before it reaches your app - forcing all traffic back through Cloudflare where your layers live. Cloudflare publishes the official list of its ranges; load it into your firewall (a cloud-provider network firewall is cleanest, or an on-box rule) and refresh it on a schedule so new ranges are picked up automatically. Keep SSH and the HTTP port (80) open: port 80 carries the Let's Encrypt renewal challenge, which is only a public proof-of-control token, never a secret. Verify the fix from a non-Cloudflare host: a direct connection to the origin IP must now time out, while the normal hostname path still returns 200.

The Fix

# Cloud network firewall (preferred): allow 443 only from Cloudflare ranges.
# Cloudflare publishes them at https://www.cloudflare.com/ips (v4 + v6).
#   inbound  tcp/443  source = <cloudflare ranges>      # app traffic, CF only
#   inbound  tcp/80   source = 0.0.0.0/0, ::/0          # ACME renewal (no secrets)
#   inbound  tcp/22   source = <your admin IPs>         # SSH
#   (everything else: default deny)

# On-box alternative (Docker example) - drop non-Cloudflare hits on 443:
ipset create cf4 hash:net family inet
curl -s https://api.cloudflare.com/client/v4/ips \\
  | jq -r '.result.ipv4_cidrs[]' | while read c; do ipset add cf4 "$c"; done
iptables -I DOCKER-USER -i eth0 -p tcp --dport 443 -m set ! --match-set cf4 src -j DROP

# Verify from an OUTSIDE (non-Cloudflare) host - must fail now:
curl --connect-to app.example.com:443:<ORIGIN_IP>:443 https://app.example.com/  # -> timeout

The Risk

If people download big files (a whole-database export, a large spreadsheet, a video) straight from your own server, you're exposed two ways. First, your server can only push data out so fast - if a few people, or a script, keep pulling a big file over and over, they clog your pipe and everyone else's experience slows to a crawl. It's a cheap way for someone to grind your site down: each download looks like a normal, successful request, so your usual alarms (which watch for errors) never notice. Second, big files often just fail anyway: the systems in front of your server usually give up on a slow transfer after about a minute and a half, and many "serverless" setups refuse to send large responses at all. So it's both unreliable AND an easy target. (Technical: origin egress saturation + the ~100s edge timeout + serverless response caps.)

The Solution

Don't hand out big files from your own server at all. Put them on a service built for storing and serving files - like Cloudflare R2 or Amazon S3 - which sits close to users around the world and has a huge, cheap pipe (with R2, downloads are free). Build the files ahead of time on a timer, copy them onto that "shelf", and let people download straight from there, so your own server is never part of the download. Keep a simple list (a manifest) of what's available and when it was last refreshed, so people and AI agents can find the current files. Now if a download flood comes, it hits the storage service - which is built to shrug it off - instead of your app. (Technical: pre-generate artifacts on a cron, serve them as static objects from R2/S3 via a small edge worker, origin out of the path.)

The Fix

# 1. Pre-generate artifacts on a cron, mirror them to the bucket (mtime-gated).
# 2. A small edge worker serves them from the bucket, edge-cached, origin untouched:
async fetch(request, env) {
  const key = mapPathToObjectKey(new URL(request.url).pathname);  // ignore junk params
  const obj = await env.BUCKET.get(key);            // from object storage, at the edge
  if (!obj) return fetch(request);                  // rare miss -> origin fallback
  return new Response(obj.body, { headers: { "cache-control": "public, max-age=7200" }});
}
# Keep a manifest.json in the bucket so consumers see available files + freshness.

The Risk

Not every visitor to your site is a person. A big slice is automated "bots", and they are not all the same. Some are the good kind: a search engine like Google sending real readers your way, or an AI assistant like ChatGPT or Claude that reads your page and quotes you to its users with a link back. Some are freeloaders: commercial "SEO" crawlers (names like SemrushBot, AhrefsBot, DotBot) that copy your whole site to sell competitor-research to other people, send you nothing back, and crawl around the clock - quietly eating your server capacity and your monthly free allowance. Two traps people fall into: (1) "I will just block all bots" - which also blocks Google and the AI assistants, making you invisible to the exact audiences you want. (2) Flipping on Cloudflare's one-click "Block AI bots" switch - which slams the door on ChatGPT, Claude, and Perplexity precisely when you are trying to be found by them. (Technical: SEO backlink crawlers and scrapers consume origin CPU and Worker/edge quota; blanket bot-blocking and the managed AI-bots rule both catch the crawlers you actually want.)

The Solution

Sort bots into "want" and "do not want", and treat each group differently - do not paint them all with one brush. WANT (let them through): search engines (Googlebot, Bingbot) and AI agents (GPTBot, ClaudeBot, PerplexityBot). DO NOT WANT (turn away): SEO backlink crawlers (SemrushBot, AhrefsBot, DotBot, MJ12bot) and aggressive scrapers (Bytespider). The good news: the freeloaders are honest - they announce their name in every request - and the reputable ones obey a "no entry" sign. So: (1) Add a robots.txt "no entry" list naming those crawlers; the polite ones stop on their own. (2) For any that ignore the sign, add ONE firewall rule that blocks just those names - it only matches a request that literally calls itself "SemrushBot", so a real person on Chrome is never touched. Two cautions: do NOT use the blanket "Block AI bots" toggle (it blocks the AI agents you want), and do NOT assume "verified bots equals good" - Semrush is itself a verified bot, so "allow all verified bots" lets the freeloader straight in. You exclude them by NAMING them, never by trusting a category. (Technical: robots.txt Disallow groups for the named user-agents plus a single WAF custom rule matching http.user_agent; Cloudflare Verified Bots includes SEO vendors so it is not an allowlist; the managed AI-scrapers rule blocks GPTBot/ClaudeBot/etc.)

The Fix

# 1. robots.txt - name the freeloader crawlers (the honest ones obey this)
User-agent: AhrefsBot
User-agent: SemrushBot
User-agent: DotBot
User-agent: MJ12bot
User-agent: Bytespider
Disallow: /
# Do NOT list Googlebot / Bingbot / GPTBot / ClaudeBot / PerplexityBot here -
# they stay under "User-agent: *" and remain fully welcome.

# 2. One Cloudflare WAF rule for any that ignore robots.txt. It blocks by NAME,
#    so a real browser (whose user-agent says "Chrome", not "SemrushBot") is safe:
(http.user_agent contains "SemrushBot") or
(http.user_agent contains "AhrefsBot")  or
(http.user_agent contains "Bytespider")     ->  Action: Block
# Runs at the edge, BEFORE your app, so a blocked crawler costs you nothing.

The Risk

A file-serving endpoint (a download or report URL) is like a clerk who fetches a file by name from a back room. An attacker asks for a file with a "climb out of the room" name - like ../../../etc/passwd (the Linux password file) or the Windows equivalent - hoping the clerk walks out of the allowed folder and hands over a system file. They try dozens of disguised spellings (URL-encoded slashes like ..%2f, doubled dots like ....//, backslashes) to slip past a naive filter. If even one works, they read the secrets on your server; even when it fails, every file-serving backend is a constant target for these scans. (Technical: path traversal / Local File Inclusion - ../ sequences plus encodings that try to escape the served directory and read /etc/passwd, /proc/self/environ, etc.)

The Solution

The safest place to stop this is at your edge (Cloudflare), before it reaches any backend - one rule then protects all your apps at once. The trick is knowing WHICH patterns to block. Some strings are 100% certain to be an attack and never appear in normal traffic - block those with total confidence: the actual target-file names (/etc/passwd, /etc/shadow, win.ini, boot.ini, /proc/self/) and the "disguise" encodings whose only purpose is to dodge a filter (..%2f, ....//, the backslash form of dot-dot, %2e%2e/). A real browser or app never sends those. But do NOT block loose tokens: a bare ".." (version strings and relative paths use it), a bare "%2f" (some APIs legitimately accept an encoded slash), or plain words like "shadow"/"etc"/"windows" (your own content may contain them - a blog post about "shadow lending" would get blocked). Block loose and you take your own site down. One more safety lever: match the URL PATH only, never the query string, so user input in ?params is never affected. Start with the certain list; if new traversal shapes show up in your logs, append them. (Technical: at the edge - a Cloudflare Worker or WAF rule - substring-match the request path against a tight set of evasion-and-target signatures; on hit, block the IP and log. Backends should still resolve and confirm the final file path stays inside the intended folder as a last resort.)

The Fix

// Edge worker / WAF - block ONLY 100%-certain traversal signatures.
// Match the URL PATH only (never the query string, so ?params stay safe).
const TRAVERSAL = [
  "..%2f", "..%5c", "....//", "%2e%2e/", "%2e%2e%2f",       // evasion-only encodings
  "/etc/passwd", "/etc/shadow", "win.ini", "boot.ini", "/proc/self/",  // target files
];                                    // (also include the backslash form of dot-dot)
const p = url.pathname.toLowerCase();
if (TRAVERSAL.some(sig => p.includes(sig))) {
  return new Response("", { status: 200 });   // dead-end + block the IP + log it
}
// NEVER add bare "..", bare "%2f", or words like "shadow"/"etc" - they break real traffic.

The Risk

A user-agent is a free-text header. Anyone can send "Googlebot" or "GPTBot", and most sites treat that name as a pass: crawlers get gentler rate limits, skip challenges, and are trusted on paths nobody else is. Scanners know this. A large share of the traffic that calls itself a well-known crawler is not that crawler; it is a scanner wearing the name, usually from a rented cloud address, hoping to inherit the goodwill. If your allowances key on the name, the name is your weakest gate.

The Solution

Treat the name as a claim and prove it with something the caller cannot fake: the source address. Most major crawlers publish their address ranges as a JSON file. Fetch those on a schedule and check the claimed crawler against its own list. A claimed crawler outside its published ranges is lying, and you can refuse it on the first request with no threshold, because there is no innocent explanation. Where a vendor publishes no ranges, do a reverse-DNS lookup and forward-confirm that the hostname resolves back to the same address. A third check needs no list at all: a crawler arriving from a cloud provider that vendor never uses is the same lie. Keep the published lists fresh, because vendors move the file and a stale list quietly turns real crawlers into "forged" ones. And remember the limit of the proof: a verified address proves who CARRIED the request, never who wrote the payload. A real crawler can be someone else's courier, so identity earns a crawler ordinary access, not a way past your payload checks.

The Fix

# On a schedule: fetch each vendor's published ranges into one table (cidr, provider).
# At the edge, when the user-agent CLAIMS a crawler:
claimed = crawler_named_in(user_agent)              # "googlebot", "gptbot", ...
if claimed and not address_in_ranges(ip, claimed):
    refuse_and_block(ip, reason="forged-crawler")   # first request, no threshold

# Vendor with no published ranges: reverse DNS, then forward-confirm
host = reverse_dns(ip)                              # e.g. crawl-66-249-66-1.googlebot.com
if host.endswith(vendor_domain) and forward_dns(host) == ip:
    verified = True

# Test it the way an attacker would: send a claimed-crawler UA from an ordinary
# address and confirm it is refused; send a real one and confirm it is served.

The Risk

The rate limit you configured at the edge is probably not the rate limit you are enforcing. Edge counters are kept per data centre and are built for speed, not accuracy: measured against one of ours, most requests that should have been refused passed. That is fine for shaping a burst and useless for anything you PUBLISH as a limit, bill against, or rely on to protect a heavy endpoint. And per-address limits say nothing about total load: a hundred addresses each under their own limit can still take an app down together.

The Solution

Run two kinds and know which is which. The loose edge counter stays for burst shaping because it is nearly free. For anything that must be exact, count in a single-writer object (on Cloudflare, a Durable Object; elsewhere, a Redis counter) keyed on address and host. Add a daily budget on top, per APP GROUP rather than per hostname, because two hostnames on the same backend would otherwise double the allowance. When a budget trips, do not keep consulting the counter under load: set a cached kill flag that every later request reads cheaply, so refusing costs less work than serving did. A breach must make the system do less, never more.

The Fix

# Layer 1 - loose edge bucket (per PoP): shapes bursts; do not trust the number.
# Layer 2 - exact counter (single writer, e.g. a Durable Object):
count = await counter.increment(key=f"{ip}:{host}", window=60)
if count > limit: return 429

# Layer 3 - daily budget per app GROUP (all hostnames of one app share it):
group = group_for(host)                          # "analytics-api", not "api.example.com"
if await kv.get(f"tripped:{group}"):             # cheap cached read, no counter touched
    return 429
if await budget.increment(group) > daily_cap:
    await kv.put(f"tripped:{group}", "1", expires_at=next_utc_midnight)

# Test from an address you do NOT allowlist: your own infrastructure short-circuits
# every limiter and returns 200 whether or not the limit works.

The Risk

A honeypot PATH (item 5.8) catches a scanner that asks for a secret file. A scanner that only walks ordinary pages never touches one. Meanwhile every site accumulates retired hostnames: an old staging name, a decommissioned tool, a subdomain that was created and never announced. Left as they are they serve nothing useful and tell you nothing. Left pointing at a live origin, they quietly serve a full mirror of your site under a name you are not watching.

The Solution

Turn hostnames that were never shared into deception hosts. A subdomain nobody was ever told about has no legitimate visitors, so every request to it is hostile by definition: zero false positives and no allowlist needed, which is the highest-quality signal in security. Serve a realistic, slightly misconfigured-looking server there: plausible fake configuration files and credentials that are format-valid but synthetic and connected to nothing. Block the caller at the edge on first contact. Never use a real credential as bait and never let the decoy execute anything; it returns strings and that is all. Do not call it a honeypot anywhere a scanner can see (hostnames, headers, bodies). For a retired name that WAS shared, keep a redirect for people with old links and let only the probe-shaped paths fall through to the trap.

The Fix

# Edge worker: trap membership decides the response before anything else.
DEAD_HOSTS = {"old-staging.example.com", ...}   # never shared: block on ANY request
SHARED_OLD = {"legacy-app.example.com", ...}    # shared once: redirect, trap only probe paths

if host in DEAD_HOSTS:
    block(ip, reason="trap-host"); return fake_server_response(path)
if host in SHARED_OLD:
    if looks_like_probe(path):
        block(ip, reason="trap-host"); return fake_server_response(path)
    return redirect_to_canonical(path)           # a real person with an old link

# fake_server_response: a believable listing, a synthetic .env, a fake wp-config.
# Seed the fake values per caller so two scanners never compare identical notes.

The Risk

Blocking is easy; unblocking is the part nobody designs. Rules pile up. A datacenter address that was a scanner in June is an innocent customer in August, and your permanent block is now a trap for a stranger. A block that exists at the edge but has no record in your database can never be released, because nothing knows it is there. A block that expired in your database but still stands at the edge looks released while still refusing. And a rule keyed on lifetime history ("this address was ever bad") can never release anyone, because history does not reset.

The Solution

Treat a block as two records that must travel together: the rule at the edge and the row in your ledger, each carrying the identifier of the other. Give every single-address block an expiry, rolling from the LAST offence, so a one-time visitor self-releases and a persistent offender sets their own term. Release on present behaviour, never on history: a network that has been quiet for a window is released whatever it did before. Escalate to a whole subnet only when several distinct addresses inside it have each earned their own block, and make the release test for the subnet look at the same evidence that put it there. Self-heal your mistakes on an unspoofable signal: a blocked address that reverse-DNS forward-confirms to a real search crawler is released under ANY block reason, because a cure keyed on identity works for causes you have not met yet. Run a reconciler that compares the edge and the ledger and reports every mismatch in both directions.

The Fix

# Every block writes BOTH halves and links them:
rule_id = edge.block(ip, note=f"{reason} until {expires}")
db.insert(ip, reason, expires_at, rule_id)       # a row with no rule_id cannot be released

# Expiry pass (scheduled): release BOTH halves, and prove the pass ran even when idle
for row in db.expired(): edge.delete(row.rule_id); db.mark_released(row)
log(f"expiry pass ran: released={n}")            # a silent no-op looks like a dead pass

# Quiet release: PRESENT behaviour, not lifetime counters
for net in db.blocked_subnets():
    if db.last_bad_contact(net) < now - QUIET_WINDOW: release(net)

# Reconcile (scheduled): compare as addresses, not strings (IPv6 formats differ)
edge_set = {inet(r.value) for r in edge.list()}; db_set = {inet(r.ip) for r in db.active()}
report(edge_set - db_set, db_set - edge_set)     # orphans in both directions

The Risk

An attacker no longer needs one fast machine. They rent a pool of thousands of home-internet addresses and send one request from each, so no address ever comes close to a per-address limit. Set that limit to two a minute and nothing changes. A global limit low enough to stop the pool also stops your real users, because the pool is built to look like ordinary traffic one request at a time. And because the addresses are real households, a block lands on a person who did nothing. The obvious lever is the useless one.

The Solution

Stop reasoning about addresses and start reasoning about the request and the load. What actually holds against a pool is a cap on how much work runs at once, server-wide (the pool gets the same few slots as one caller), and the checks on the payload itself, neither of which reads the address. Read the signal as DISTINCT CALLERS failing together rather than volume from any one of them: many addresses failing at once means either you are broken or a pool is probing. Notice that "one request per address, hundreds of addresses, the same client fingerprint" is itself a signature; the addresses rotate, the client usually does not. Treat the block as a last resort, keep it short, and expect the pool to move on the next day whatever you do.

The Fix

# What holds, and none of it reads the address:
MAX_INFLIGHT_GLOBAL = 10          # a pool of 300 addresses shares these ten slots with everyone
if inflight >= MAX_INFLIGHT_GLOBAL: return 429
classify(payload)                 # the payload gates refuse a hostile query from any address

# What to WATCH (a detector, not a block):
#   distinct addresses in the last 10 min with the same client fingerprint and ~1 request each
#   distinct addresses failing on the same endpoint at once  (spread), vs one address (stuck client)
# What NOT to do: lower the per-address limit (catches zero), or a global limit tight enough to
# refuse the pool (refuses your users first).

The Risk

Every crawler identity you can check today is either a claim (the user-agent) or a list that rots (published address ranges that the vendor moves without telling you). Some AI agents now sign each request with a private key and publish the matching public key at a well-known address on their own domain. That is a real proof: nobody can forge a signature without the key, and there is no address list to go stale. Ignoring it means treating a proven caller the same as an anonymous one; trusting it too early has its own trap, below.

The Solution

Verify the signature and record the result, for visibility first. The request carries three headers: who claims to be signing, what was signed and with which key, and the signature itself. Fetch the public key from the directory the claim names, rebuild the signed string exactly as the standard specifies, and check it. Show verified and unverified separately on your own panel, not on the crawler page, because these are not crawlers indexing you: they are people asking an AI to fetch your data, and the caller often names its purpose in the user-agent. Do not give a verified signature any privilege yet. The signature covers the host and the method but not the path or the body, and it stays valid for a window, so a captured set of headers can be replayed on any path for that long. Verification for visibility is safe today; verification for authorisation needs replay protection first.

The Fix

# The three headers (Web Bot Auth, built on HTTP Message Signatures):
#   signature-agent   "https://agent.example"        who claims to sign
#   signature-input   sig1=("@authority" "@method" "signature-agent");keyid=...;alg="ed25519";created=...;expires=...;nonce=...
#   signature         sig1=:<base64>:
key   = fetch(claimed_origin + "/.well-known/http-message-signatures-directory").key_for(keyid)
base  = rebuild_signature_base(request, covered_components, params)   # exact bytes, per the RFC
valid = ed25519_verify(key, base, signature)
record(agent=claimed_origin, verified=valid, purpose=user_agent)      # visibility, no privilege

# Negative control before you trust the verifier: change one byte of the host and it must reject.
# Before any privilege: remember nonces for the validity window, or a replay walks in.