AI crawler robots.txt guide: balance search and scraping

AI crawler robots.txt guide: balance search and scraping

AI bots changed SEO permanently. Teams now manage two types of automated visitors. First: search indexers that send traffic back to websites. Second: training scrapers that pull text for AI models without sending visitors.

Blocking all bots removes your site from ChatGPT Search and Perplexity citations. Letting all bots in costs server bandwidth with zero clicks. This guide shows a balanced robots configuration.

The two classes of AI crawlers: search retrieval vs model training

Manage bot traffic by separating agents by commercial function. Search retrieval bots send clickable citations. Training scrapers download text for AI models without sending traffic.

Conflating these groups causes self-inflicted ranking penalties. The table below maps primary user-agents, their operators, their function, and their citation behavior:

Treating every user-agent as an identical threat cuts off pathways to early-adopter searchers. Allow verified search indexers while filtering bulk scrapers to maintain visibility.

How Google separates search indexing from Gemini training

Site operators often assume blocking Googlebot is required to prevent Google from training AI models. This removes your website from organic search and AI Overviews at the same time.

Google created Google-Extended to give webmasters independent control over model training.

# Block Gemini training while preserving organic rankings
User-agent: Google-Extended
Disallow: /

User-agent: Googlebot
Allow: /

Under Google’s official documentation, Google-Extended controls whether site content feeds training pipelines for Gemini and Vertex AI. It does not affect:

1. Organic search indexing or ranking.
2. Google Discover feeds.
3. AI Overviews. Because AI Overviews pull from organic search index caches, Googlebot controls AI Overview inclusion.

If your organization restricts AI training on copyright grounds, you can safely disallow Google-Extended without losing organic search impressions.

The OpenAI crawler ecosystem: OAI-SearchBot versus GPTBot

OpenAI runs two separate bots with different goals. Confusing these two is a common mistake on modern websites.

GPTBot collects text to train OpenAI language models. Pages fetched by GPTBot do not generate search links. Their content goes into AI model weights.

OAI-SearchBot runs ChatGPT Search. When a user asks a question, ChatGPT queries its retrieval engine, fetches documents, and shows cards with outbound links.

# Allow ChatGPT search citations while blocking model weight scraping
User-agent: OAI-SearchBot
Allow: /
Disallow: /api/
Disallow: /private/

User-agent: GPTBot
Disallow: /

If you block both, ChatGPT cites third-party aggregators instead of your brand. Allow OAI-SearchBot and disallow GPTBot. This protects your content from extraction while keeping you in conversational search results.

Managing PerplexityBot, ClaudeBot, and aggressive scrapers

Not all bots behave the same. Search engines like Google and Bing use adaptive crawl rates based on server response times. AI scrapers often deploy high-concurrency requests that can saturate origin databases.

The operational cost of Bytespider

ByteDance runs Bytespider for content feeds and AI training. Log audits often show Bytespider sending tens of thousands of requests per hour across staging paths and filter URLs.

Bytespider sends negligible search traffic to Indian business websites. Allowing it wastes CPU cycles. Block it in robots.txt and reinforce with edge firewall rules:

User-agent: Bytespider
Disallow: /

How to handle Anthropic and ClaudeBot

Anthropic uses ClaudeBot to collect training data. Unlike search indexers, ClaudeBot does not route traffic back to publishers. Unless you have a data partnership, disallowing ClaudeBot saves bandwidth with zero organic search downside:

User-agent: ClaudeBot
Disallow: /

For live web browsing features within Claude chat interfaces, Anthropic historically deployed the anthropic-ai user-agent. If your marketing objective includes being cited when Claude users ask for real-time recommendations, allow anthropic-ai on your primary content directories while keeping training bots blocked.

Recommended robots.txt configuration for balanced AI governance

Writing a balanced robots configuration requires strict adherence to RFC 9309 (the Robots Exclusion Protocol specification).

Directive precedence under RFC 9309

When authoring multi-bot rules, keep these three parsing behaviors in mind:

1. Specific matches override wildcards: A block addressed to User-agent: OAI-SearchBot takes precedence over a generic block addressed to User-agent: *. Crawlers look for their specific token first; if found, they ignore the wildcard block entirely.
2. Longest matching path wins: If a user-agent matches both an Allow and a Disallow rule for a given URL, the rule with the longer character path takes precedence.
3. No directive concatenation: Do not combine multiple user-agents on a single line. Each user-agent declaration requires its own dedicated line.

Production robots.txt template

Here is a tested configuration that protects internal application routes, blocks uncredited training scrapers, and invites legitimate search indexers:

# Global rule for standard search crawlers
User-agent: *
Allow: /
Disallow: /api/
Disallow: /search/
Disallow: /admin/
Disallow: /*?*sort=
Disallow: /*?*filter=

# Google search indexing: open
User-agent: Googlebot
Allow: /
Disallow: /api/
Disallow: /search/

# Google Gemini model training: blocked
User-agent: Google-Extended
Disallow: /

# OpenAI conversational search indexing: open
User-agent: OAI-SearchBot
Allow: /
Disallow: /api/
Disallow: /search/

# OpenAI training data scraper: blocked
User-agent: GPTBot
Disallow: /

# Perplexity citation indexing: open
User-agent: PerplexityBot
Allow: /
Disallow: /api/
Disallow: /search/

# Anthropic model training scraper: blocked
User-agent: ClaudeBot
Disallow: /

# Apple Intelligence generative training: blocked
User-agent: Applebot-Extended
Disallow: /

# ByteDance aggressive scraper: blocked
User-agent: Bytespider
Disallow: /

# Meta AI model training: blocked
User-agent: Meta-ExternalAgent
Disallow: /

# Canonical sitemap location
Sitemap: https://www.webgrow24.com/sitemap.xml

You can test and generate clean directives matching this structure using the free Robots.txt Generator.

How to verify crawler compliance in server logs

Placing rules in a text file is a declaration of policy, not a physical firewall. Well-behaved corporate crawlers respect RFC 9309; rogue scrapers frequently ignore directives or spoof common browser user-agents.

Reverse DNS verification for legitimate bot IPs

Never trust the User-Agent header alone. Bad actors often label their scraper Googlebot or OAI-SearchBot to bypass simple filters. To confirm an automated visitor is authentic, perform a reverse DNS lookup followed by a forward lookup.

On Linux or macOS terminal environments, verify incoming IP addresses using standard command-line network utilities:

# Step 1: Reverse lookup on the suspicious IP
host 66.249.66.1
# Output: 1.66.249.66.in-addr.arpa domain name pointer crawl-66-249-66-1.googlebot.com.

# Step 2: Forward lookup to verify domain authenticity
host crawl-66-249-66-1.googlebot.com
# Output: crawl-66-249-66-1.googlebot.com has address 66.249.66.1

If the domain does not resolve to *.googlebot.com, *.openai.com, or *.perplexity.ai, the client is spoofing its identity. You can feed your raw access logs into our Server Log Analyzer to spot anomalous crawl spikes, check HTTP status ratios, and isolate malicious IP ranges.

Identifying rogue crawlers and handling HTTP 429 responses

When an aggressive bot ignores your disallow rules, relying on robots directives is insufficient. Implement rate-limiting rules at your reverse proxy or web application firewall:

1. Enforce HTTP 429 Too Many Requests: When a single autonomous IP issues more than 60 requests per minute across non-cached routes, return a 429 status code with a standard Retry-After: 300 header.
2. Block verified spoofers: Create firewall blocklists for IPs claiming to be search bots that fail reverse DNS verification.
3. Protect dynamic database endpoints: Never allow crawlers to trigger internal search queries, multi-facet sorting, or expensive report generation.

To test how search and AI systems process your pages once crawled, run your key templates through the AIO Inspector.

Key decisions before publishing your bot governance rules

Before deploying your updated configuration, gather engineering, marketing, and legal leads to establish clear organizational standards:

1. Establish your intellectual property posture: Decide whether your company treats model training as acceptable fair use or unauthorized extraction. If you plan to license your data in commercial agreements, block GPTBot, ClaudeBot, and Google-Extended immediately.
2. Prioritize citation discovery: If conversational search referral traffic is part of your acquisition funnel, never place a blanket block on OAI-SearchBot or PerplexityBot.
3. Monitor server capacity: Track origin CPU and database connection counts during peak morning crawl hours. If unmanaged bots cause latency for real human visitors, restrict aggressive agents at the edge.
4. Schedule quarterly directive audits: The AI bot landscape shifts rapidly. New user-agents emerge every quarter, while existing providers update their token definitions. Revisit your rules every 90 days.

For teams managing complex technical architectures, review our Technical, On-Page, and Local SEO Services to structure a durable crawling foundation, or compare engagement options on our Pricing page.