How to Configure robots.txt for AI Crawlers in 2026 (GPTBot, PerplexityBot, ClaudeBot)
A practical guide to configuring robots.txt for AI crawlers in 2026 - the difference between training bots and citation bots, and how to allow one without the other.
If you have a robots.txt file, there's a good chance it still treats every crawler the same way: one User-agent: * block, allow or disallow everything. That made sense when the only thing reading your site was Googlebot. It doesn't hold up anymore, because "an AI crawler" isn't one thing.
Training Bots vs. Search Bots: Not the Same Job
AI companies run separate crawlers for separate jobs, and conflating them is the most common mistake I see in robots.txt files right now:
- Training bots (
GPTBot,ClaudeBot,Google-Extended,CCBot) collect content to train future models. Allowing these gives your content away for free, with no guarantee it ever gets attributed back to you. - Search/citation bots (
OAI-SearchBot,PerplexityBot,Claude-SearchBot) crawl live, specifically to answer a real query someone just asked, and cite the source. This is the traffic-driving, attribution-carrying half of the equation.
A single wildcard rule can't tell these apart. If you disallow *, you block both. If you allow *, you allow both. Neither is obviously right, which is exactly why treating it as one decision is the mistake.
The 2026 Default: Block Training, Allow Search
The configuration I'd point most people toward, and the one that's become the most common single setup among sites that care about this, blocks the training crawlers and explicitly allows the citation crawlers:
User-agent: GPTBot
Disallow: /
User-agent: ClaudeBot
Disallow: /
User-agent: Google-Extended
Disallow: /
User-agent: CCBot
Disallow: /
User-agent: OAI-SearchBot
Allow: /
User-agent: PerplexityBot
Allow: /
User-agent: Claude-SearchBot
Allow: /
User-agent: *
Allow: /
Sitemap: https://yourdomain.com/sitemap.xml
This isn't the only reasonable answer. If you're fine with your content training future models, in exchange for whatever visibility that might eventually bring, a blanket allow is simpler and not wrong. It's a genuine trade-off, not a technical bug to be fixed, and it's worth actually deciding rather than defaulting into it by not thinking about it.
Why This Matters More Than llms.txt
A lot of 2026 GEO advice leads with adding an llms.txt file. I'd deprioritise it. Google's own guidance from May 2026 states plainly that it isn't used for AI Overviews or AI Mode, and in practice the major AI search crawlers skip it and read your HTML directly instead. Where it has found a real use is coding-agent tools like Cursor and Claude Code, which do fetch it routinely when pointed at documentation sites. Useful for a very different audience, not the one most sites writing about GEO actually have.
Getting the robots.txt split right, on the other hand, directly controls whether the bots that generate real citations can even reach your content in the first place. It's a five-minute change with a much clearer mechanism than most of what gets recommended.
Checking It Worked
Once it's deployed, load yourdomain.com/robots.txt directly and confirm the bot-specific blocks are there. Most parsers apply the most specific matching User-agent block rather than the first one in the file, so a bot-specific rule further down still takes precedence over an earlier wildcard rule, but keeping them clearly separated (as above) avoids relying on that behaviour being implemented consistently everywhere.
