Which AI Crawlers Should You Allow in robots.txt?

If you want to be citable in ChatGPT, Perplexity, or Google AI Overviews, you should generally allow OAI-SearchBot, Claude-SearchBot, and PerplexityBot in robots.txt, since those are the search and answer agents that actually retrieve content in response to a live query. Whether to allow the separate training crawlers (GPTBot, ClaudeBot, Google-Extended) is a different decision with a different tradeoff, and conflating the two is the most common mistake in this area.

That distinction, search agents versus training agents, is the single most important thing to understand here, and most generic "block AI bots" advice online doesn't make it. Blocking the wrong one can quietly remove you from AI-generated answers without affecting your Google ranking at all, since these crawlers are entirely separate from Googlebot.

What's the actual difference between a training crawler and a search crawler?

A training crawler collects content to include in a model's training data, which shapes the model's general knowledge over time but has no immediate, per-query effect. A search or answer crawler fetches content in real time (or near real time) specifically to answer a live user query, which is the direct mechanism behind getting cited in a specific AI-generated answer today. Blocking a training crawler doesn't stop you from being cited tomorrow. Blocking a search crawler can.

Which specific user-agents fall into each category?

Should I block the training crawlers or allow them?

It depends on what you're trying to protect and what kind of business you run, and there's a genuine tradeoff either way, not an obviously correct default.

The case for allowing training crawlers: long-term brand familiarity inside the model itself. If GPTBot and ClaudeBot can train on your content, future versions of these models are more likely to already "know" who you are when answering a general question, independent of any specific live retrieval. This is generally the better default for B2B, editorial, and service businesses whose content isn't proprietary in a way that matters if it's memorized.

The case for blocking training crawlers: protecting content you specifically don't want reproduced or summarized without attribution, such as premium research, proprietary datasets, or content behind a paywall you don't want effectively scraped into a training set for free.

For most sites built to be found and cited (which describes almost any GEO-focused content strategy), allowing training crawlers is the more consistent choice, since blocking them while trying to optimize for AI visibility works against your own stated goal.

Does blocking search crawlers ever make sense?

Rarely, if AI visibility is a goal at all. The only common scenario is a site actively trying to avoid AI-generated summaries eating into direct traffic, for content types where a click genuinely matters more than a citation (subscription content, ad-supported publishers with thin margins). Even then, it's worth weighing against the citation and brand-visibility loss, since a blocked search crawler generally means your content simply doesn't get mentioned as an option at all, not that traffic gets redirected to you some other way.

What does a sensible robots.txt actually look like for a business that wants AI visibility?

A reasonable baseline, adjusted for your specific priorities:

User-agent: *
Allow: /

User-agent: GPTBot
Allow: /

User-agent: OAI-SearchBot
Allow: /

User-agent: ChatGPT-User
Allow: /

User-agent: ClaudeBot
Allow: /

User-agent: Claude-SearchBot
Allow: /

User-agent: PerplexityBot
Allow: /

User-agent: Google-Extended
Allow: /

Sitemap: https://example.com/sitemap.xml

This allows both the training and search/answer agents from the three major providers, on the reasoning that a business actively pursuing AI citation benefits more from broad visibility than from selectively blocking any one of them.

Frequently asked questions

Does blocking Google-Extended hurt my regular Google ranking? No. Google-Extended is specifically for Gemini and Google's generative AI features, and is entirely separate from Googlebot, which handles classic search indexing and ranking. Blocking one doesn't affect the other.

Do all AI crawlers actually respect robots.txt? Most of the major, named crawlers from OpenAI, Anthropic, and Google document their user-agents and respect standard robots.txt directives. Some crawlers, including Bytespider and certain stealth-mode fetchers, have been documented ignoring robots.txt in practice, so treat robots.txt as a strong signal rather than an absolute guarantee for every crawler.

How do I check which AI crawlers are actually visiting my site? Check your server access logs for requests to your root-level pages and filter by known AI crawler user-agent strings (GPTBot, ClaudeBot, PerplexityBot, and the others listed above). This tells you what's actually happening, rather than relying on assumptions about what should be happening.

Is there a downside to allowing everything? Mainly server load from increased crawl traffic, which is rarely significant for a typical content or marketing site, and the general content-reproduction concerns already covered above. For most businesses trying to build AI visibility, the upside outweighs both.

This is the crawlability half of AI-readiness. Once your robots.txt is sorted, why schema markup matters more for AI search covers what happens once these crawlers are actually let in, and do you need an llms.txt file in 2026 covers the related, more limited-use standard that sometimes gets confused with this one.

Next step

Pull up your current robots.txt file and check whether it has any explicit rules for AI crawlers at all. If it's silent on GPTBot, PerplexityBot, and the others, that's not neutral. Depending on your hosting setup and any default configuration, an unlisted crawler could be implicitly allowed or blocked, and it's worth making the decision explicit rather than leaving it to a default you haven't checked.