AI Crawler Checker — Test Which AI Bots Can Access Your Website

An AI crawler checker reads a website’s robots.txt file and reports which AI crawlers — such as GPTBot, OAI-SearchBot, ClaudeBot, PerplexityBot and Google-Extended — are allowed or disallowed. It shows, at a glance, whether AI search and training bots may request the site. It checks robots.txt rules, not live bot behaviour.

Paste your URL into the checker above. It fetches /robots.txt and finds the rule that applies to GPTBot — its own User-agent group if one exists, otherwise the wildcard group — then reports “Allowed by robots.txt” or “Blocked by robots.txt”. GPTBot is a training crawler, so blocking it does not remove you from ChatGPT Search.

Both are OpenAI agents with different jobs. GPTBot crawls to train OpenAI models. OAI-SearchBot indexes pages so ChatGPT Search can find and link them. Blocking GPTBot keeps content out of training; blocking OAI-SearchBot removes you from ChatGPT Search sources. They are controlled by separate rules in robots.txt.

Yes. GPTBot is only the training crawler. ChatGPT Search relies on OAI-SearchBot to index pages and ChatGPT-User to fetch a page on request. A site can block GPTBot to opt out of training while still allowing OAI-SearchBot and ChatGPT-User, remaining fully available to ChatGPT Search.

ChatGPT-User is OpenAI’s user-triggered fetcher. It loads a specific page when a ChatGPT user asks about that URL or the model browses on their behalf. It is not a bulk indexing crawler. Blocking it stops ChatGPT from opening your page live during a conversation.

PerplexityBot is Perplexity’s indexing crawler that builds its answer index. Perplexity-User is the live fetch triggered when a user references your page directly. Blocking PerplexityBot removes you from Perplexity’s indexed sources; blocking Perplexity-User only stops on-demand fetches.

Neither, for ranking. Google-Extended is a training toggle: it controls whether your content is used to train Gemini and Vertex AI. It does not affect Googlebot’s crawl, your Google Search rankings, or AI Overview eligibility — those follow standard Googlebot access, not Google-Extended.

Add a group with User-agent: ClaudeBot and either no matching Disallow or an explicit Allow: /. Note that Anthropic now runs three bots — ClaudeBot (general crawl), Claude-SearchBot (search index) and Claude-User (live fetch) — so allow each one you want, not just ClaudeBot.

Yes. Cloudflare and other WAF or CDN layers can block or challenge AI bots by user-agent or with managed “block AI bots” rules, independently of robots.txt. This checker reads only robots.txt, so a site can pass here yet still be blocked at the firewall with a 403 — test that separately.

A 403 usually comes from a WAF, CDN bot-management rule, rate limiting, or a server rule that rejects the bot’s user-agent or IP — not from robots.txt. A robots.txt block simply asks a compliant bot not to request the page; it does not return 403. If AI bots get 403s, check Cloudflare or WAF settings.

robots.txt is a widely honoured standard telling compliant crawlers which paths they may request. llms.txt is an emerging, voluntary Markdown file that points AI tools to your key content; it is guidance, not an access-control or ranking mechanism, and support is not universal. robots.txt controls access; llms.txt suggests what to read.

No. Allowing a crawler is one technical prerequisite — it makes your pages fetchable. Whether an AI engine actually cites you depends on indexing, content quality, relevance, authority, structure and each engine’s own retrieval. Access removes a blocker; it does not guarantee AI visibility.

Disallow the training user-agents (GPTBot, Google-Extended, CCBot, Applebot-Extended, Meta-ExternalAgent) while allowing the search and user-fetch ones (OAI-SearchBot, PerplexityBot, Claude-SearchBot, ChatGPT-User, Claude-User). Give each its own User-agent group so the rules do not collide. See the robots.txt examples above.

No. robots.txt controls crawling, not indexing. A disallowed URL can still be indexed without content if other pages link to it. To keep a page out of an index, use a noindex meta robots tag or X-Robots-Tag header on a page that stays crawlable — not a robots.txt disallow.

Recheck after any robots.txt, CDN or WAF change, after a migration or platform switch, and whenever vendors publish new user-agents — which happens regularly. A quarterly check plus an event-based check on each deploy catches most silent blocks before they cost you citations.