Loading

SEOSync

Can ChatGPT see your site? Check as GPTBot, ClaudeBot & Perplexity.

We fetch your page as the real AI crawlers do (no JavaScript, their user agents, your robots.txt rules) and show you exactly what they can read. Most sites that are invisible to AI search look perfectly fine in a browser.

Free, no signup. One URL per check, results in about 15 seconds.

What you get

  • robots.txt rules for 9 AI and search crawlers
  • Live fetch as GPTBot, OAI-SearchBot, ClaudeBot, PerplexityBot and Googlebot
  • Words bots can read vs what a browser shows
  • Plain-English fixes for every problem we find

Why your site can be invisible to ChatGPT

ChatGPT, Claude, and Perplexity answer questions by fetching pages with their own crawlers: OAI-SearchBot and ChatGPT-User for ChatGPT, ClaudeBot for Claude, PerplexityBot for Perplexity. If those crawlers are blocked, or get an empty page, your brand cannot be cited no matter how good your content is.

There are three common causes. A robots.txt rule (often copied from a template) disallows AI bots. A CDN or firewall setting, like Cloudflare's AI bot blocking, refuses them. Or the page is a JavaScript app, so the crawler receives an empty shell, because most AI crawlers do not run JavaScript.

What this AI crawler checker tests

For the URL you enter, we read your robots.txt and evaluate the rules for each crawler, then fetch the page live with each crawler's user agent and without running JavaScript. We count the readable words, check the title, H1, canonical tag and structured data, and compare it with what a normal browser request gets.

  • OAI-SearchBot and ChatGPT-User: ChatGPT search and citations
  • GPTBot: training future OpenAI models (blocking it does not remove you from ChatGPT search)
  • ClaudeBot: Claude retrieval and training
  • PerplexityBot: Perplexity answers and citations
  • Googlebot and Google-Extended: Google AI Overviews and Gemini
  • Bingbot: Bing and Microsoft Copilot
  • CCBot: Common Crawl, the open dataset many LLMs train on

How to read your results

Visible means the crawler is allowed and can read real text. Thin means it gets in but sees little content, usually a client-rendered page. Blocked means robots.txt disallows it. Refused means your server or CDN returned an error or a bot challenge to our test request.

A refusal is worth confirming in your CDN logs. Some CDNs verify crawlers by IP address, so they block our test (which uses the crawler's user agent from our own servers) while still letting the real crawler through.

  1. Fix any robots.txt rule that blocks a search or citation bot
  2. Allow verified AI crawlers in your CDN or firewall settings
  3. Server-render or pre-render pages that come back thin
  4. Make sure each URL has its own title, H1 and canonical in the raw HTML
  5. Re-run the check to confirm every engine shows Visible

Blocking AI training vs blocking AI search

Many sites block GPTBot to opt out of model training. That is a reasonable choice and does not stop ChatGPT from citing you, because ChatGPT search uses OAI-SearchBot and ChatGPT-User. The mistake is blocking all OpenAI or Anthropic agents at once, which removes you from AI answers too.

If you want to be cited but not trained on, allow OAI-SearchBot, ChatGPT-User, PerplexityBot and Googlebot, and disallow GPTBot, Google-Extended and CCBot.

After crawlers can read you: getting cited

Being crawlable is the entry ticket, not the win. AI engines prefer pages with a clear answer near the top, FAQ blocks, specific facts like prices and limits, and structured data that matches the page.

SEOSync audits those signals on every page, writes the fixes, and on Pro tracks whether ChatGPT, Gemini and Perplexity actually name your brand for the questions your buyers ask.

Related

How do I check if ChatGPT can crawl my site?

Enter your URL in the AI crawler checker above. It reads your robots.txt rules for OAI-SearchBot, ChatGPT-User and GPTBot, fetches the page with their user agents, and shows how much text they can read without JavaScript.

Should I block GPTBot?

Blocking GPTBot only opts you out of training future OpenAI models. It does not remove you from ChatGPT search, which uses OAI-SearchBot and ChatGPT-User. If you want AI citations, make sure those two are allowed.

Why does the checker say my page is thin when it looks fine in a browser?

Your content is probably rendered with JavaScript. Browsers run it, but most AI crawlers do not, so they receive a nearly empty HTML shell. Server-side rendering or pre-rendering fixes this.

Does Cloudflare block AI crawlers?

Cloudflare offers a setting that blocks AI bots, and some plans enable it by default. If the checker shows Refused for several bots, check your Cloudflare (or other CDN) bot settings and allow verified AI search crawlers.

What is llms.txt and do I need it?

llms.txt is an optional plain-text file at your site root that summarizes what you do and links to key pages for AI tools. Not every engine reads it yet, so it helps at the margin but matters far less than robots.txt and server-rendered content.

Is this AI crawler checker free?

Yes. No signup, a few checks per hour per visitor. To audit whole sites, get AI-written fixes, and track whether ChatGPT, Gemini and Perplexity mention your brand, start a free SEOSync account.

More SEO tools

Start free · Pricing · Blog