Blocking AI Scraper Bots (GPTBot, LLM Crawlers) Without Breaking Your SEO: OWASP CRS WAF Rules

One morning, the hosting dashboard shows an unexplained bandwidth spike. No attack, no marketing campaign underway, just traffic that has been climbing steadily for weeks. Digging into the logs reveals the pattern: tens of thousands of daily requests from User-Agents like GPTBot, CCBot, or Bytespider, methodically crawling every page, every product sheet, every blog post. This is not a security incident in the classic sense, but a very real problem of cost, server load, and content ownership.

The instinctive reaction, blocking anything that looks like a bot, is the worst possible fix: it also catches Googlebot in the net, and organic rankings collapse within days. This article covers how to target AI training crawlers precisely without touching legitimate search engines.

The concrete problem: crawlers vacuuming everything, for free

Since 2024, several large language model providers have run dedicated crawlers to collect web content for model training:

• GPTBot (OpenAI)
• ClaudeBot (Anthropic)
• CCBot (Common Crawl, frequently reused upstream by model trainers)
• Bytespider (ByteDance)
• Google-Extended (distinct from Googlebot, dedicated to Gemini training)

For an e-commerce site or a small business, these crawlers bring no direct benefit: no clicks, no conversions, no measurable visibility. What they do bring is outbound bandwidth consumption, load on the origin server (often bypassing the CDN cache, since they sometimes ignore standard caching headers), and full extraction of site content, including pages a business would rather not see republished verbatim elsewhere.

On a product catalog with several thousand SKUs, the cumulative cost in bandwidth and server CPU becomes significant, without a single line of revenue improving as a result.

The trap of blocking too broadly

Faced with this, the most common reflex is to block on broad User-Agent keywords like "bot", "crawl", or "spider". That is exactly the mistake to avoid.

Googlebot, Bingbot, and social media preview crawlers (useful for link previews) also contain these terms. An overly generic block removes the site from search results within days, with a recovery time measured in weeks once the mistake is fixed.

The correct method separates two very different families:

The third case is the sneakiest one: nothing stops a malicious scraper from declaring itself as Googlebot to bypass a naive block based solely on the claimed User-Agent string.

A targeted WAF rule by User-Agent, with reverse DNS verification

Good practice combines two layers of control. The first identifies the claimed User-Agent; the second confirms that the source IP actually belongs to the infrastructure of the provider it claims to represent.

An example ModSecurity rule (OWASP CRS compatible) that specifically targets known AI training crawlers:

SecRule REQUEST_HEADERS:User-Agent "@rx (?i)(GPTBot|ClaudeBot|CCBot|Bytespider|Google-Extended|Amazonbot|Applebot-Extended)" \
    "id:9001001,\
    phase:1,\
    deny,\
    status:403,\
    log,\
    msg:'Blocked known AI training crawler by User-Agent',\
    tag:'ai-scraper-block',\
    chain"
    SecRule REQUEST_HEADERS:User-Agent "!@rx (?i)googlebot|bingbot" "t:none"

For spoofed User-Agents claiming to be a legitimate search engine, verification runs through a reverse DNS lookup followed by a forward lookup (double verification, the method Google itself recommends for validating Googlebot):

#!/bin/bash
# verify_crawler.sh: confirms an IP claiming to be Googlebot
# actually belongs to Google infrastructure
IP="$1"
HOSTNAME=$(dig +short -x "$IP" | sed 's/\.$//')
 
if [[ "$HOSTNAME" != *".googlebot.com" && "$HOSTNAME" != *".google.com" ]]; then
    echo "SUSPECT: $IP does not resolve to a Google domain"
    exit 1
fi
 
FORWARD_IP=$(dig +short "$HOSTNAME" | tail -n1)
if [[ "$FORWARD_IP" != "$IP" ]]; then
    echo "SUSPECT: forward lookup does not match ($FORWARD_IP != $IP)"
    exit 1
fi
 
echo "OK: $IP is a genuine Google crawler ($HOSTNAME)"

Run periodically, or wired in ahead of the WAF, this check lets you tell apart a real Googlebot from a scraper spoofing its identity.

Why robots.txt alone is not enough

The first instinct, adding a robots.txt directive, is necessary but far from sufficient on its own:

User-agent: GPTBot
Disallow: /

User-agent: CCBot
Disallow: /

User-agent: Bytespider
Disallow: /

The problem: robots.txt is a declarative convention, not an enforcement mechanism. Nothing requires a crawler to honor it, and several AI training crawlers have already been documented ignoring it outright, or switching infrastructure to keep collecting despite the disallow rule. robots.txt remains good first-level hygiene (some providers do respect it), but the only genuinely effective control is enforced at the WAF layer, where the request can be blocked before it ever reaches the application.

Rate limiting and anomaly scoring instead of binary blocking

Blocking outright is not always the right call, particularly when identification of the crawler stays uncertain or when a site wants a more permissive policy toward certain agents. A graduated approach, consistent with the OWASP CRS anomaly scoring model, raises a behavior score instead of deciding all-or-nothing:

SecRule REQUEST_HEADERS:User-Agent "@rx (?i)(GPTBot|CCBot|Bytespider)" \
    "id:9001002,\
    phase:1,\
    pass,\
    setvar:'tx.anomaly_score_pl1=+15',\
    msg:'AI crawler detected, scoring applied'"
 
SecRule IP:REQCOUNT "@gt 60" \
    "id:9001003,\
    phase:1,\
    deny,\
    status:429,\
    msg:'AI crawler rate limit exceeded (60 req/min)',\
    expirevar:IP.REQCOUNT=60"

This logic throttles excessive traffic (capping at N requests per minute, returning 429) without categorically banning an IP on its first suspicious request. It leaves room for ambiguous cases (a real search engine misidentified, a legitimate partner) without letting a content scraper run at full speed either.

Before and after: what to monitor

Two sets of metrics let you judge how effective the setup really is, and above all avoid the dreaded side effect, a drop in organic traffic:

Before deployment:

• Monthly bot bandwidth (usually identifiable from access logs filtered by known User-Agent)
• Share of server traffic attributable to AI crawlers (often between 10 and 30 percent of total non-human traffic on a content-heavy site)
• Unchanged Google indexation rate (Search Console coverage)

After deployment:

• A sharp drop in bandwidth consumed by blocked crawlers
• False block rate on legitimate traffic: monitor closely for two to three weeks via 403/429 logs, specifically looking for hits from verified Google, Bing, or known partner IPs
• Stable organic traffic and indexation in webmaster tools, week over week

A successful rollout shows bot bandwidth sharply down and a false block rate near zero on verified Googlebot traffic. If that last figure moves, it is a signal that a rule is too broad and needs tightening before it affects rankings.

The anti-scraping AI rule pack

Writing and maintaining these rules by hand, tracking the constantly shifting User-Agents and IP ranges of new training crawlers, requires ongoing monitoring that few small businesses can sustain over time. A pre-tuned rule set, validated against real traffic and regularly updated as new crawlers appear, avoids the overly broad block while genuinely shutting off wasted bandwidth.

ThreatClaw's AI anti-scraping WAF rule pack covers known crawlers, includes the reverse DNS verification logic and graduated rate limiting, and is ready to deploy without breaking legitimate traffic or search rankings.