Cloudflare blocking AI crawlers? How to check and fix it

Cloudflare can block AI crawlers on your site even if your robots.txt allows them. The usual causes are the one click “block AI bots” setting, AI Crawl Control rules, Bot Fight Mode, or a WAF custom rule, and since mid 2025 Cloudflare has pushed AI crawler blocking as a default for many new domains. To check, look for 403 responses or “Just a moment…” challenge pages served to bots like GPTBot, OAI-SearchBot, and PerplexityBot, then review the Bots and Security sections of your Cloudflare dashboard.

• robots.txt is a request; Cloudflare is enforcement. A firewall block wins no matter what robots.txt says.
• Several separate settings can block AI bots. Check all of them, not just the obvious toggle.
• A challenge page usually returns a 403 with a cf-mitigated: challenge header and a “Just a moment…” title. To a crawler, that page is your entire site.
• Testing with a spoofed user agent from your laptop is useful but not conclusive, because Cloudflare verifies real bots by network origin, not just by name.
• You can allow AI search and citation bots while still blocking training bots.

Why would Cloudflare block AI crawlers on my site?

Cloudflare sits in front of a large share of the web and has made AI crawler controls a headline feature. Over the past two years it has added a one click option to block AI bots, dashboards showing which AI services crawl you, managed robots.txt rules, and experiments with charging crawlers for access. Since mid 2025, Cloudflare has also offered or enabled AI crawler blocking by default for many newly added domains, often as a choice during onboarding.

That is great if you are a publisher who wants control. It is a problem if you are a SaaS company that wants ChatGPT, Claude, and Perplexity to recommend you, and someone on your team clicked “yes” to a default months ago. Product names and defaults shift often, so treat this guide as a map of where to look, and confirm the current options in your own dashboard and in Cloudflare’s bots documentation.

Which Cloudflare settings can block AI crawlers?

The one click “block AI bots” setting

Found in the Bots area of the dashboard, this toggle blocks bots that Cloudflare places in its AI categories. It is convenient, but blunt. Depending on how Cloudflare categorizes each bot at the time, it can block crawlers that power citations in ChatGPT search or Perplexity, not only training crawlers. Your dashboard may also offer narrower scopes for the block than “everywhere”; check exactly what yours is set to.

AI Crawl Control and managed robots.txt

AI Crawl Control (previously called AI Audit) lists which AI services request your pages, how often, and whether they were allowed. It lets you allow or block individual crawlers. Related features can manage your robots.txt for you by prepending AI crawler rules or content usage signals. If your live robots.txt contains rules you never wrote, this is the likely source.

Bot Fight Mode and Super Bot Fight Mode

Bot Fight Mode (free plans) and Super Bot Fight Mode (paid plans) challenge traffic that Cloudflare scores as automated. Cloudflare says verified bots are treated differently from unknown automation, and Super Bot Fight Mode has an explicit option for verified bots. But Bot Fight Mode is a common source of hard to debug blocks because, per Cloudflare’s docs, WAF skip rules cannot bypass it. If a specific crawler is being challenged and you cannot exempt it, turning Bot Fight Mode off may be the only fix.

WAF custom rules

Look for rules written years ago to stop scrapers: blocks on user agents containing “bot”, “crawler”, or “python”, country blocks that cover data center regions, or challenges on every non browser request. These predate AI crawlers and quietly catch them. See Cloudflare’s custom rules docs for the expression syntax.

How do I tell if Cloudflare is blocking AI bots?

1. Run the free AI crawler checker. Paste a money page into the SEOSync AI crawler checker. It fetches the page as GPTBot, OAI-SearchBot, ClaudeBot, PerplexityBot, and Googlebot, checks robots.txt for nine crawlers, and compares the words each bot reads with what a browser sees. A bot that gets a tiny page while the browser gets thousands of words is a red flag.
2. Inspect the response headers. Request the page with a bot user agent and read the status and headers.
3. Check Security Events. In the dashboard, filter security events by user agent (for example GPTBot or PerplexityBot) or by verified bot. Each blocked request shows which rule or feature acted on it.
4. Check AI Crawl Control. If it shows a crawler as blocked, or shows no requests at all from a bot you expected, you have your answer.
5. Check robots.txt as served. Fetch it through Cloudflare, not from your repo, to see any managed additions.

curl -sI -A "Mozilla/5.0 (compatible; PerplexityBot/1.0; +https://perplexity.ai/perplexitybot)" \
  https://example.com/pricing

# Signs of a Cloudflare block or challenge:
HTTP/2 403
server: cloudflare
cf-mitigated: challenge
cf-ray: 8a1b2c3d4e5f6789-FRA

What a challenge page looks like to a crawler

• A 403 status (sometimes 503) instead of 200.
• A cf-mitigated: challenge header on managed and JavaScript challenges.
• An HTML title of “Just a moment…” or “Attention Required! | Cloudflare”, and script references to /cdn-cgi/challenge-platform/.
• Error pages such as “Error 1020: Access denied”, which means a firewall rule matched.

Crawlers do not solve challenges. To an answer engine, that interstitial is your page: a few dozen words with no product information. If you only notice a low word count without a block, the issue is more likely client side rendering; see why AI bots see an empty React page.

Verified bots vs spoofed user agents

Anyone can send a request that says “GPTBot” in the user agent. Cloudflare maintains a list of verified bots and checks real crawlers against the networks and signatures their operators publish. That has two consequences for testing:

• If your curl with a GPTBot user agent is blocked, you have at least one rule that matches by name, or a rule that blocks unverified clients claiming to be bots.
• If your curl gets through, that does not prove the real GPTBot does. Real crawler traffic may be scored differently by Bot Fight Mode or bot category rules.

So combine a quick outside test with the dashboard’s view of actual verified bot traffic. Rules that key on Cloudflare’s verified bot signals are also more robust than rules that trust the user agent string, which scrapers routinely fake.

How do I allow AI search bots but block training bots?

Most SaaS and B2B sites want to be cited in AI answers. If you also want to opt out of model training, set it up deliberately:

1. Decide per bot. List the crawlers you want to allow (for example OAI-SearchBot, ChatGPT-User, Claude-SearchBot, PerplexityBot, Googlebot, Bingbot) and the ones to block (for example GPTBot, ClaudeBot, CCBot). Our robots.txt guide for AI crawlers explains what each one does.
2. Turn off the blanket toggle. Replace the one click AI block with per crawler choices in AI Crawl Control, or with your own rules.
3. Express the policy in robots.txt. Compliant bots will honor it, and it documents your intent publicly.
4. Enforce at the edge if needed. Add a WAF custom rule that blocks the training crawlers by name, and make sure no broader rule catches the search crawlers.
5. Review Bot Fight Mode. If allowed crawlers still get challenged, check whether Bot Fight Mode is the cause, since skip rules cannot override it.
6. Retest. Run the crawler checker again and watch security events for a few days.

Example WAF custom rule expressions (verify field names and bot categories in your dashboard, as Cloudflare evolves both):

# Rule 1, action: Block (training crawlers by name)
(http.user_agent contains "GPTBot") or
(http.user_agent contains "ClaudeBot") or
(http.user_agent contains "CCBot")

# Rule 2, action: Skip remaining custom rules (verified search crawlers)
(cf.client.bot and (
  http.user_agent contains "OAI-SearchBot" or
  http.user_agent contains "PerplexityBot" or
  http.user_agent contains "Claude-SearchBot"
))

Order matters: place the skip rule above any broader challenge or block rules. If you prefer category based blocking, Cloudflare exposes verified bot categories you can match on; check which category each crawler is assigned to before relying on it.

What about Vercel, Fastly, Akamai, and other hosts?

Cloudflare is not the only layer that can turn crawlers away. The details differ, but the same categories of controls exist almost everywhere:

• Vercel: the Vercel Firewall supports custom rules, attack challenge mode, and managed bot protection rules. Challenge modes stop crawlers just like Cloudflare’s do, so check firewall settings and logs if bots fail.
• Fastly and Akamai: both offer WAF and bot management products that classify AI crawlers and can block or challenge them. In larger companies these are often owned by a security team, so ask what bot policies are active.
• Managed WordPress hosts and plugins: some hosts and security plugins block or rate limit unfamiliar bots by default.
• Stacked layers: a site can sit behind Cloudflare and a host firewall at the same time. A clean Cloudflare config does not rule out a block further down.

Common mistakes when unblocking AI crawlers

• Fixing robots.txt and stopping there. robots.txt cannot undo a firewall block.
• Testing only the homepage. Rules are often path specific. Test pricing, docs, blog posts, and comparison pages too.
• Allowlisting by user agent alone. Anyone can claim to be OAI-SearchBot. Combine name matching with verified bot signals.
• Challenging robots.txt and sitemaps. If a rule challenges /robots.txt, compliant crawlers may treat your whole site as off limits. Exempt robots.txt and sitemap.xml from challenges.
• Forgetting subdomains and zones. Docs or blog subdomains may live in a separate zone with different bot settings.
• Expecting instant results. After unblocking, AI engines need to recrawl. Citations can take weeks to reflect the change.

How do I know the fix worked?

Confirm it in three layers, because each one fails for different reasons:

1. Access. The crawler checker and curl tests now return 200 with your real page for every bot you meant to allow, and the word count bots read is close to what a browser sees.
2. Traffic. Over the following days, AI Crawl Control and your logs show successful requests from those crawlers, with no new security events against them.
3. Outcomes. Over the following weeks, answer engines start citing or naming you for relevant questions. SEOSync’s AI visibility tracking on Pro and Agency plans asks ChatGPT, Gemini, and Perplexity real buyer prompts and reports whether your brand is named, which competitors are named instead, and which sources are cited, so you can see the trend instead of spot checking by hand. Plans and limits are on the pricing page.

If access is fixed but outcomes do not move, the bottleneck has shifted from infrastructure to content: the page needs clearer answers, FAQs, and structure worth quoting.

A 10 minute audit checklist

1. Run the AI crawler checker on your homepage, pricing page, and one blog post.
2. Fetch robots.txt through Cloudflare and compare it with the file in your repo.
3. Check the AI bots toggle and AI Crawl Control settings.
4. Note whether Bot Fight Mode or Super Bot Fight Mode is on, and its settings.
5. Review WAF custom rules for “bot”, “crawler”, or broad challenge rules.
6. Filter security events for GPTBot, OAI-SearchBot, ClaudeBot, and PerplexityBot.
7. Fix, then retest the same URLs.

Once bots can reach the page, the next question is whether it is worth citing. A free SEO audit gives you both an SEO and an AEO score for the URL. To see whether engines actually start naming you, our guides on Perplexity SEO and tracking brand mentions in ChatGPT cover the next steps.

FAQ

Does Cloudflare block AI crawlers by default?

It can. Since mid 2025 Cloudflare has offered or enabled AI crawler blocking by default for many newly added domains, often as an onboarding choice, and it has a one click setting to block AI bots on existing zones. Defaults change, so check the Bots section and AI Crawl Control in your own dashboard rather than assuming.

How do I know if Cloudflare is blocking GPTBot or PerplexityBot?

Request a page with the bot's user agent and look for a 403 status, a cf-mitigated: challenge header, or a page titled Just a moment. Then filter Cloudflare security events by that user agent to see which rule acted. The free SEOSync AI crawler checker also shows how many words each bot actually reads compared with a browser.

Can I block AI training bots on Cloudflare but keep AI search citations?

Yes. Instead of the blanket AI bot toggle, choose per crawler in AI Crawl Control or with WAF custom rules. Block training crawlers such as GPTBot, ClaudeBot, and CCBot, and allow search and citation crawlers such as OAI-SearchBot, Claude-SearchBot, and PerplexityBot. Mirror the same policy in robots.txt so compliant bots see it too.

Does Bot Fight Mode block AI crawlers?

It can challenge traffic Cloudflare scores as automated. Cloudflare treats verified bots differently from unknown automation, but Bot Fight Mode cannot be bypassed with WAF skip rules, which makes it a common cause of blocks that are hard to exempt. If an allowed crawler keeps getting challenged, test with Bot Fight Mode turned off.

Why does my site allow AI bots in robots.txt but they still cannot read it?

robots.txt only asks crawlers to behave; it does not open your firewall. If Cloudflare, Vercel, or another CDN challenges or blocks the request, the bot gets a 403 or challenge page no matter what robots.txt says. Client side rendering is the other common cause: the bot gets through but receives an almost empty HTML shell.

Find out if bots are being turned away

Check how GPTBot, ClaudeBot, and PerplexityBot see your page, then score it for SEO and AEO and ship the fixes.