robots.txt for AI crawlers: GPTBot, ClaudeBot, PerplexityBot

To control AI crawlers with robots.txt, add a User-agent group for each bot token (GPTBot, OAI-SearchBot, ClaudeBot, PerplexityBot, and so on) with the Allow or Disallow rules you want. The catch: each AI company runs several bots with different jobs. Blocking GPTBot stops OpenAI from using your pages for training, but ChatGPT search uses OAI-SearchBot, so blocking the wrong token can remove you from AI answers without protecting anything.

• AI crawlers fall into three groups: training crawlers, search and citation crawlers, and user triggered fetchers. Decide on each group separately.
• Google-Extended does not affect Google Search or AI Overviews. Those use Googlebot.
• A bot that matches a specific User-agent group ignores the User-agent: * group entirely. Repeat shared rules in every group.
• Most “AI cannot see my site” problems come from blanket blocks, SPA hosts serving robots.txt as HTML, or a CDN firewall, not from a missing allow rule.
• Bot names and policies change. Verify against each vendor’s official crawler docs before you ship.

What are the three kinds of AI crawlers?

Before writing any rules, separate AI bots by what they do with your page. This is the single most useful mental model for robots.txt decisions.

1. Training crawlers collect public pages that may be used to train future models. Blocking them is an opt out of training. It does not remove you from AI answers today, and it does not delete data already collected.
2. Search and citation crawlers build the index an answer engine searches when it composes a response and links sources. Blocking these is how sites accidentally disappear from ChatGPT search or Perplexity citations.
3. User triggered fetchers visit a page because a person asked the assistant to, for example by pasting a URL into a chat. Several vendors say robots.txt may not apply to these the same way, because the request acts on behalf of a user rather than as automated crawling.

Which AI crawler does what?

The table below reflects vendor documentation at the time of writing. Vendors add, rename, and redefine bots regularly, so treat it as a starting point and confirm on the official pages, such as OpenAI’s crawler overview, Google’s list of crawlers, Perplexity’s bot docs, and Anthropic’s help center.

Other bots you may see in your logs include Meta-ExternalAgent, Amazonbot, and Bytespider. Each operator documents its own purpose and token, and not every crawler honors robots.txt equally. robots.txt is a voluntary standard, not access control.

Google-Extended is not an AI Overviews switch

This is the most common misunderstanding. Google-Extended is a robots.txt token only; there is no separate crawler with that name. Google uses it to decide whether content Googlebot already crawled may be used to train and ground Gemini models. Google states it does not affect inclusion or ranking in Google Search, and AI Overviews and AI Mode are part of Search, built from what Googlebot crawls. To keep content out of AI Overviews specifically, you would need snippet controls such as nosnippet or data-nosnippet, which also affect regular search snippets.

The same logic applies to Apple: Applebot-Extended is a usage token, while Applebot does the actual crawling. Microsoft has no separate Copilot crawler token; Copilot draws on the Bing index, so Bingbot access is what matters there.

How does robots.txt matching actually work?

The rules are standardized in RFC 9309 and Google’s robots.txt documentation. Five details cause most of the confusion:

• One group per bot. A crawler picks the group whose User-agent line matches it most specifically and follows only that group. If you write a User-agent: GPTBot group, GPTBot ignores everything under User-agent: *.
• Groups can list several bots. Stack multiple User-agent lines above one set of rules to apply the same rules to all of them.
• Longest path wins. Within a group, the most specific matching Allow or Disallow path applies. When an Allow and a Disallow are equally specific, Allow wins.
• Wildcards work in paths, not names. * and $ are supported in paths. A user agent line like User-agent: *bot is not a pattern; it simply does not match anything useful.
• The file is per host. https://example.com/robots.txt does not cover blog.example.com or docs.example.com. Each subdomain needs its own file.

Copy and paste robots.txt recipes

Replace the sitemap URL and any private paths with your own. Keep the file at the root of each host and serve it as plain text.

Recipe 1: allow everything (best for most SaaS marketing sites)

If you want maximum visibility in Google, Bing, and every answer engine, you do not need bot specific groups at all. Just keep private areas out.

User-agent: *
Disallow: /app/
Disallow: /admin/
Allow: /

Sitemap: https://example.com/sitemap.xml

Recipe 2: allow AI search and citations, block training

The middle path many publishers choose: stay quotable in ChatGPT search, Claude, and Perplexity, while opting out of training where vendors offer a separate token. Note that the private paths are repeated in each group, because named bots ignore the * group.

# AI search, citation, and user-triggered fetchers: allowed
User-agent: OAI-SearchBot
User-agent: ChatGPT-User
User-agent: Claude-SearchBot
User-agent: Claude-User
User-agent: PerplexityBot
User-agent: Perplexity-User
Disallow: /app/
Disallow: /admin/
Allow: /

# Model training: blocked
User-agent: GPTBot
User-agent: ClaudeBot
User-agent: Google-Extended
User-agent: Applebot-Extended
User-agent: CCBot
Disallow: /

# Everyone else, including Googlebot and Bingbot
User-agent: *
Disallow: /app/
Disallow: /admin/
Allow: /

Sitemap: https://example.com/sitemap.xml

One judgment call: the line between “training” and “search” is drawn by each vendor and can shift. If Claude visibility matters a lot to you, re-read Anthropic’s current crawler descriptions before blocking ClaudeBot.

Recipe 3: block all AI crawlers, keep classic search

For sites that do not want any AI use. Understand the tradeoff: you will stop being cited by ChatGPT search, Claude, and Perplexity. You will still appear in Google AI Overviews and Copilot, because blocking Googlebot or Bingbot would remove you from search entirely.

User-agent: GPTBot
User-agent: OAI-SearchBot
User-agent: ChatGPT-User
User-agent: ClaudeBot
User-agent: Claude-SearchBot
User-agent: Claude-User
User-agent: PerplexityBot
User-agent: Perplexity-User
User-agent: Google-Extended
User-agent: Applebot-Extended
User-agent: CCBot
User-agent: Meta-ExternalAgent
User-agent: Bytespider
Disallow: /

User-agent: *
Disallow: /admin/
Allow: /

Sitemap: https://example.com/sitemap.xml

User triggered fetchers may not honor this file, and some scrapers ignore robots.txt completely. If you need enforcement, that happens at your CDN or firewall, which is covered in Is Cloudflare blocking AI crawlers on your site?

Recipe 4: block training on some sections only

You do not have to make one decision for the whole site. A SaaS company might want product and pricing pages used everywhere, while keeping long form research or customer only docs out of training sets. Path rules let you do that per bot.

# Training crawlers: marketing pages yes, research and docs no
User-agent: GPTBot
User-agent: ClaudeBot
User-agent: CCBot
Disallow: /research/
Disallow: /docs/
Disallow: /app/
Allow: /

User-agent: *
Disallow: /app/
Allow: /

Sitemap: https://example.com/sitemap.xml

How long does a robots.txt change take to apply?

Crawlers cache robots.txt rather than fetching it before every request. Google documents caching it for up to about a day in normal conditions, and other vendors describe similar windows. So a change usually takes effect within a day or two for new crawling. Two things it does not do: it does not remove pages already in a search index right away, and it does not pull content back out of datasets or models trained earlier. If a page is already cited and you block the search crawler, expect the citation to fade as the vendor’s index refreshes, not instantly. In the other direction, unblocking does not trigger an immediate recrawl; well linked pages and an up to date sitemap help bots come back sooner.

Common robots.txt mistakes with AI crawlers

1. A blanket block left over from staging

User-agent: * followed by Disallow: / blocks every compliant crawler, AI and search alike. It often ships to production from a staging template or a “coming soon” phase. Check the live file after every major deploy.

2. Wildcards in user agent names

Rules like User-agent: *GPT* or User-agent: AI* look clever and do nothing. Use the exact tokens from vendor docs. Matching is case insensitive, so gptbot and GPTBot are equivalent.

3. Assuming a named group inherits the default rules

Adding User-agent: PerplexityBot with Allow: / “to be safe” means PerplexityBot now ignores your /admin/ and /app/ disallows. Copy shared rules into each group, as in Recipe 2.

4. Using meta robots to “block AI”

A <meta name="robots" content="noindex"> tag removes the page from Google and Bing, not just from AI tools. Tags like noai and noimageai are not part of any standard, and major AI vendors do not document support for them. Also remember the order of operations: a crawler blocked in robots.txt never fetches the page, so it never sees a noindex tag on it.

5. robots.txt served as HTML by your SPA

Single page apps on hosts like Vercel, Netlify, or S3 often rewrite every path to index.html. If no real robots.txt file exists in your build output, /robots.txt returns your app shell with a 200 status. Crawlers then parse HTML as rules, find nothing valid, and your intended blocks or sitemap line are silently ignored. The same SPA setup often hides your content from AI crawlers too; see why AI bots see an empty React page.

6. A robots.txt that errors

Under RFC 9309, a robots.txt that returns a 4xx is treated as “no restrictions”, but one that returns a 5xx or times out is treated as “disallow everything” until it recovers. A flaky origin or a firewall challenge on /robots.txt can therefore block all compliant bots at once.

7. Your CDN overrides your file

Some CDNs and hosts can manage or prepend robots.txt rules for you, or block AI bots at the firewall before robots.txt is ever consulted. If your file says “allow” but bots get 403s, the problem is upstream of robots.txt.

How do I test my robots.txt for AI crawlers?

1. Fetch the live file. Confirm a 200 status, a text/plain content type, and that the body is your rules rather than HTML.
2. Check every host. Repeat for www, the apex domain, and any docs or blog subdomains.
3. Run the AI crawler checker. The free SEOSync AI crawler checker evaluates your robots.txt for nine crawlers (including GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, PerplexityBot, Google-Extended, Bingbot, and CCBot), fetches the page as the main AI bots and Googlebot, and lists the fixes to make.
4. Use Google Search Console. The robots.txt report shows which version Google fetched and any parse errors.
5. Read your server or CDN logs. Look for real requests from the bots you allowed. No hits for weeks can mean a firewall block or simply low crawl priority.

# Status code and content type
curl -sI https://example.com/robots.txt

# The rules themselves (should not start with <!doctype html>)
curl -s https://example.com/robots.txt | head -40

Should you block AI crawlers at all?

It depends on how you make money from your content. A rough guide:

• SaaS, agencies, and most B2B sites: allow everything. Your marketing pages exist to be found and recommended, and buyers increasingly ask ChatGPT and Perplexity which tool to use.
• Publishers and paid content: Recipe 2 is a common compromise. You stay citable, which can send referral traffic, while opting out of training.
• Sensitive or licensed content: Recipe 3 plus firewall enforcement, and accept the loss of AI visibility.

If you choose to be visible, access is only the first step. Pages also need crawlable text and a structure answer engines can quote. Our guide on how to get cited by ChatGPT covers the on page side, and a free SEO and AEO score shows how citable a specific URL is today. robots.txt also pairs naturally with llms.txt, a newer, optional file that points AI tools at your most useful pages.

FAQ

How do I block GPTBot in robots.txt?

Add a group with User-agent: GPTBot followed by Disallow: / to the robots.txt at the root of each host. This opts your pages out of OpenAI model training. It does not remove you from ChatGPT search, which uses a separate crawler called OAI-SearchBot, so leave that one allowed if you want ChatGPT to keep citing you.

Does blocking GPTBot remove my site from ChatGPT answers?

Not by itself. OpenAI documents GPTBot as its training crawler, while OAI-SearchBot powers ChatGPT search results and citations, and ChatGPT-User handles pages users ask ChatGPT to open. Blocking only GPTBot is an opt out of training. Check OpenAI's current bot documentation, since names and roles can change.

Does Google-Extended block AI Overviews?

No. Google-Extended is a robots.txt token that controls whether your content is used for Gemini models. Google says it does not affect Google Search inclusion or ranking, and AI Overviews and AI Mode are part of Search, built from what Googlebot crawls. Snippet controls like nosnippet are the lever for AI Overviews, and they also affect normal snippets.

Can I allow AI search engines but block AI training?

Mostly, yes. Allow search and citation crawlers such as OAI-SearchBot, Claude-SearchBot, and PerplexityBot, and disallow training tokens such as GPTBot, ClaudeBot, Google-Extended, Applebot-Extended, and CCBot. Remember that a bot matching a named group ignores the User-agent: * group, so repeat any private path rules in each group.

Do AI crawlers actually respect robots.txt?

The major vendors say their training and search crawlers follow robots.txt. User triggered fetchers, like ChatGPT-User or Perplexity-User, may not apply it the same way because a person requested the page. Some scrapers ignore robots.txt entirely. If you need enforcement rather than a request, block at your CDN or firewall.

How can I test which AI crawlers can access my site?

Fetch your live robots.txt and confirm it returns plain text with a 200 status, then run the free SEOSync AI crawler checker. It evaluates robots.txt rules for nine crawlers, fetches the page as GPTBot, OAI-SearchBot, ClaudeBot, PerplexityBot, and Googlebot, and compares what bots read with what a browser shows.

Check which AI crawlers can read your site

Run the free AI crawler checker, then get dual SEO and AEO scores and the fixes to ship on your money pages.