AI robots.txt checker

Paste your robots.txt and see what each AI crawler is actually told: GPTBot, OAI-SearchBot, ClaudeBot, PerplexityBot, Google-Extended and eight more. For every bot you get Blocked or Allowed, whether the rule comes from a group that names it or is inherited from User-agent: *, and the line number that decides. The file never leaves your browser.

# Block AI training crawlers, keep AI search and user fetches
User-agent: GPTBot
User-agent: ClaudeBot
User-agent: Google-Extended
User-agent: Applebot-Extended
User-agent: CCBot
User-agent: Bytespider
User-agent: meta-externalagent
Disallow: /

User-agent: OAI-SearchBot
User-agent: Claude-SearchBot
User-agent: PerplexityBot
Allow: /

Training bots and search bots are not the same decision

Most vendors now split their crawlers by purpose. Blocking a training bot (GPTBot, ClaudeBot, Google-Extended, CCBot) keeps your pages out of future model training. Blocking a search bot (OAI-SearchBot, Claude-SearchBot, PerplexityBot) removes you from the answers those assistants give, and from the source links they show, which is where AI referral traffic comes from. A common choice for a site that wants visitors is therefore: block training, allow search. The ready-made block under the checker does exactly that.

User-triggered fetchers (ChatGPT-User, Claude-User, Perplexity-User) are a third case: they load a page because a person asked the assistant to read it. OpenAI and Perplexity say robots.txt may not apply to them, so do not count on a Disallow line there.

How the checker decides (RFC 9309)

• A bot follows the group whose User-agent names it, case-insensitive. Several groups naming it are merged.
• Only if no group names it does it fall back to User-agent: *. A named group replaces the * group, it does not add to it.
• Among the rules that match the path, the longest one wins; on a tie, Allow wins.
• * matches any characters, $ anchors the end of the path, and an empty Disallow: blocks nothing.

We used the same logic to read the robots.txt of 106 VPN, password manager and privacy-tool websites in August 2026: only ten had said anything about AI crawlers. The data and method are in our AI crawler robots.txt study.

robots.txt asks, the server enforces

robots.txt only works on crawlers that choose to read it. If you need a hard block, refuse the request on the server itself. On nginx, a user-agent map returns 403 to the training bots before your application even runs:

# /etc/nginx/conf.d/ai-bots.conf  (http context)
map $http_user_agent $ai_training_bot {
    default 0;
    ~*(GPTBot|ClaudeBot|CCBot|Bytespider|meta-externalagent) 1;
}

# inside your server { } block
if ($ai_training_bot) { return 403; }

A user-agent can be faked, so pair this with rate limiting if scraping is heavy. Shared hosting rarely lets you edit the nginx configuration; a small VPS where you hold root does, and the same server can also run your own WireGuard VPN.

FAQ

How do I check if my robots.txt blocks AI crawlers?

Open yoursite.com/robots.txt, copy the whole file and paste it in the checker above. For each AI bot it shows Blocked or Allowed for the path you test, whether the decision comes from a group naming that bot or from the User-agent: * group, and the exact line that decides.

Does blocking GPTBot remove my site from ChatGPT search?

No. OpenAI uses two different bots: GPTBot for model training and OAI-SearchBot for ChatGPT search results. Blocking GPTBot alone keeps you eligible for ChatGPT search citations; blocking OAI-SearchBot is what removes you from them.

Does Google-Extended stop Google from crawling my site?

No. Google-Extended is not a separate crawler, it is a name Googlebot reads to know whether your content may be used for Gemini training and grounding. Disallowing it does not change Google Search crawling, indexing or ranking.

If I block ClaudeBot, are Claude-SearchBot and Claude-User blocked too?

No. Each bot obeys only the group that names it, or the * group if none does. Anthropic documents three separate names, so each one needs its own User-agent line if you want to block it.

Why does a bot I named ignore my User-agent: * rules?

Under RFC 9309 a crawler follows the most specific group that matches it and ignores the * group entirely. If you give GPTBot its own group with only Allow: /, your Disallow: /admin/ from the * group no longer applies to it. Repeat the lines it must still respect.

Is robots.txt enough to stop AI scraping?

It is a request, not a lock. The major AI vendors say their declared crawlers honour it, but user-triggered fetchers may not, and undeclared scrapers ignore it. To enforce a block you need a rule on the server (web server, reverse proxy or firewall) that refuses the request.

Is my robots.txt sent anywhere?

No. The analysis is plain JavaScript running in your tab: no fetch, no API call, no storage. You can disconnect from the internet after the page loads and the checker still works.

Bot names and roles checked against each vendor's crawler documentation in October 2026; vendors add and rename bots, so re-check before relying on a list. Runs entirely in your browser; nothing is stored or sent. The Contabo link is an affiliate link, marked rel="sponsored".