Technical · 3 min read
How to allow AI crawlers in robots.txt (GPTBot, ClaudeBot, PerplexityBot)
Robots.txt for AI crawlers: how to allow GPTBot, ClaudeBot and PerplexityBot so ChatGPT, Claude and Perplexity can read and cite your pages.
The crawlers that matter
Each AI company uses named crawlers. Check your robots.txt doesn't block these:
- GPTBot, OAI-SearchBot, ChatGPT-User — OpenAI / ChatGPT
- ClaudeBot, Claude-SearchBot — Anthropic / Claude
- PerplexityBot — Perplexity
- Google-Extended — Gemini and Google AI features
A robots.txt that allows them
If you want AI visibility, allow them explicitly and point to your sitemap:
User-agent: GPTBot
Allow: /
User-agent: OAI-SearchBot
Allow: /
User-agent: ClaudeBot
Allow: /
User-agent: PerplexityBot
Allow: /
User-agent: *
Allow: /
Sitemap: https://yourdomain.com/sitemap.xmlOther blockers to check
- A noindex meta tag on important pages.
- Firewall or bot-protection rules (e.g. Cloudflare "block AI bots") that return errors to crawlers.
- Key content rendered only by JavaScript — make sure the text is in the HTML.
See where your client stands
Run a free check to see what AI says about a brand today, or start a 14-day trial.