AI Crawlers and robots.txt: Which Bots to Allow in 2026

AI crawlers and robots.txt are now a visibility decision, not just a security one. If you want ChatGPT, Perplexity, and Google''s AI answers to cite your brand, the AI crawlers that fetch pages to build those answers must be allowed through your robots.txt. Block the wrong bot and you vanish from AI search — block the right one and you simply opt out of a training dataset. The two are not the same, and confusing them is the most expensive robots.txt mistake of 2026.

This guide gives you the distinction that actually matters, a copy-paste robots.txt policy, and a reference table of every AI crawler user agent worth knowing — so you can protect what you want protected without accidentally deleting yourself from the AI answer layer.

The one distinction that changes everything: training vs. retrieval crawlers

Almost every "should I block AI bots?" argument goes wrong because it treats AI crawlers as a single category. They are two categories with opposite consequences.

• Training crawlers collect pages to train future foundation models. Blocking them keeps your content out of the next model''s weights. It has no effect on whether you get cited today. Examples: GPTBot, ClaudeBot, Google-Extended, Applebot-Extended, Meta-ExternalAgent, Bytespider, CCBot.
• Retrieval crawlers (also called search or user agents) fetch live pages at answer time to build and cite responses. Blocking them removes you from AI search results and citations. Examples: OAI-SearchBot, ChatGPT-User, PerplexityBot, Perplexity-User, Claude-SearchBot, Claude-User, and — critically — Googlebot, which powers AI Overviews and AI Mode.

Here is the rule in one sentence: block training crawlers if you want to protect your content as an asset; never block retrieval crawlers if you want to be visible in AI answers. As one plain summary of the trade-off puts it, AI systems cannot cite what they cannot access and understand.

The mistake that hurts brands is the tidy one-liner User-agent: * Disallow: / aimed at "AI," or a blanket block of everything with "GPT" or "bot" in the name. That conflates two separate decisions — protecting content from model training and being present in AI search results — and quietly deletes you from the answer engines your buyers now use.

Which AI crawlers to allow or block: the 2026 reference table

Use this as your source of truth. "Allow" here means let it read your pages; "block" means add a Disallow for that user agent.

Two things to note. First, Google-Extended is not a separate crawler — it is a robots.txt token that lets you opt out of Gemini training without touching your normal Google Search ranking. Second, this list moves. New agents appear every quarter, so treat the table as a living document and re-check it when you run your next AI visibility audit.

The copy-paste robots.txt for AI search visibility

For most companies whose goal is to be found and cited — SaaS, ecommerce, media, services — the sensible default is allow retrieval, optionally block training. Here is a policy you can adapt:

# --- Retrieval / AI search: ALLOW (these get you cited) ---
User-agent: Googlebot
Allow: /

User-agent: OAI-SearchBot
Allow: /

User-agent: ChatGPT-User
Allow: /

User-agent: PerplexityBot
Allow: /

User-agent: Perplexity-User
Allow: /

User-agent: Claude-SearchBot
Allow: /

User-agent: Claude-User
Allow: /

# --- Training crawlers: block only if you want to opt out of model training ---
User-agent: GPTBot
Disallow: /

User-agent: ClaudeBot
Disallow: /

User-agent: Google-Extended
Disallow: /

User-agent: CCBot
Disallow: /

User-agent: Bytespider
Disallow: /

# --- Everything else ---
User-agent: *
Allow: /

Sitemap: https://www.yourdomain.com/sitemap.xml

Three implementation notes that save real traffic:

1. Never Disallow: / a retrieval bot to "save crawl budget." AI crawl volume is real, but the fix is caching and a CDN, not blocking the agents that cite you.
2. Keep the training block optional. If your content marketing is your distribution, being in training data can help future models recommend you unprompted. Blocking GPTBot is a rights decision, not a visibility one — decide it deliberately.
3. robots.txt is a norm, not a wall. Compliant bots respect it, but a spoofed user agent can claim to be anything. For genuinely sensitive paths, use authentication or a WAF, not a Disallow line.

Who should block, and who absolutely should not

Block training crawlers if: you are a publisher or hold premium, paywalled, or licensable content and want to preserve leverage in AI-licensing deals. Protecting a proprietary corpus is a legitimate strategy.

Do not block retrieval crawlers if: you sell software, run an ecommerce store, offer professional services, or otherwise depend on being discovered. With roughly 69% of searches now ending without a click, the AI answer is the impression. If you are not inside it, you are not in the consideration set at all. This is the same shift we cover in the AI Overviews recovery playbook — the traffic didn''t disappear, it moved into answers you have to be cited in.

A useful gut check: an established site with real topical authority almost always gains more from being cited than it loses from being read. If you have spent years earning trust, the last thing you want is a robots.txt line quietly withholding that trust from the engines your buyers now ask.

Access is necessary but not sufficient

Allowing the right crawlers gets you eligible to be cited. It does not get you cited. Once the door is open, extractability does the rest: question-led headers, specific and citable data points, clean structure, and clear entity signals. A statistic like "AI campaigns deliver 20–30% higher ROI" is far more quotable than "AI improves results" — and quotability is what turns crawl access into citations. For the full method, see how to get cited by ChatGPT and our primer on what generative engine optimization actually is.

The order matters: fix access first, then optimize for extraction. No amount of GEO craft rescues a page an answer engine was told to ignore. If you would rather have this checked, structured, and monitored continuously across every answer engine, that is exactly what ivon''s autonomous marketing platform is built to do.

Frequently asked questions

Should I block AI crawlers in robots.txt?

Only block training crawlers (like GPTBot, ClaudeBot, Google-Extended) if you want to keep your content out of future model training. Never block retrieval crawlers (OAI-SearchBot, PerplexityBot, Claude-SearchBot, Googlebot) if you want to appear in AI search answers — those bots fetch live pages to build and cite responses, so blocking them removes you from AI visibility entirely.

What is the difference between GPTBot and OAI-SearchBot?

GPTBot is OpenAI''s training crawler; it collects pages to help train future models and blocking it has no effect on your live citations. OAI-SearchBot is OpenAI''s retrieval crawler; it builds the search index ChatGPT uses to answer questions and cite sources. If you want to be cited in ChatGPT search, allow OAI-SearchBot even if you block GPTBot.

Does blocking AI crawlers help SEO?

No. Blocking retrieval crawlers actively hurts you by removing your pages from AI Overviews, AI Mode, ChatGPT, and Perplexity answers. Blocking training crawlers is neutral for SEO — it protects content from model training but does not change traditional rankings or AI-search citations. For most sites, the visibility cost of over-blocking outweighs any benefit.

Do AI crawlers actually obey robots.txt?

Major crawlers from Google, OpenAI, and Anthropic document their user agents and respect standard robots.txt directives. However, compliance is voluntary and some bots have been reported to ignore it, while spoofed user agents can impersonate any crawler. Treat robots.txt as a norm for well-behaved bots — and protect truly sensitive content with authentication or a firewall instead.

What is Google-Extended?

Google-Extended is a robots.txt control token, not a distinct crawler. Adding Disallow: / for it opts your site out of training Google''s Gemini and Vertex AI models without affecting your ranking in Google Search or your eligibility for AI Overviews. It lets you separate "train on my content" from "rank my content" as two independent choices.

Sources

• AI Crawlers Explained: GPTBot, ClaudeBot, PerplexityBot and How to Let Them In (2026) — Anagram
• AI Crawler User Agents Explained: A 2026 Reference — HoneyB
• Which AI Crawlers to Allow in robots.txt: 2026 List — Cite.sh
• AI bots in SEO: To block, or not to block — Yoast
• Do LLMs Respect robots.txt? — Seer Interactive
• Block AI Crawlers or Let Them In: An Honest Look at Both Sides — Insightland
• Google''s Guide to Optimizing for Generative AI Features on Google Search