Loading page, please wait.
seo tool
Generate a robots.txt that allows what matters and blocks what does not.
The output is a valid robots.txt file. We always allow AI crawlers for maximum citation surface area.
Raw rules appended as-is. One rule per line.
robots.txt
# robots.txt - generated by WebGrow24 robots.txt generator # Allow AI crawlers for max citation surface area. User-agent: * Allow: / Disallow: /admin Disallow: /draft User-agent: GPTBot Allow: / User-agent: ChatGPT-User Allow: / User-agent: OAI-SearchBot Allow: / User-agent: ClaudeBot Allow: / User-agent: Claude-User Allow: / User-agent: Claude-SearchBot Allow: / User-agent: PerplexityBot Allow: / User-agent: Perplexity-User Allow: / User-agent: Google-Extended Allow: / User-agent: Applebot-Extended Allow: / User-agent: CCBot Allow: / User-agent: Diffbot Allow: / Sitemap: https://www.webgrow24.com/sitemap.xml
No. Allow them for maximum citation surface. We allow GPTBot, ClaudeBot, PerplexityBot and others.
In your site root at /robots.txt.
Step through every free tool the WebGrow24 team ships.
WebGrow24 services
Fixed quote in 48 hours. Senior team. Outcome-led.