Robots.txt LLM Policy Checker – Complete Guide to AI Crawler Rules for GPTBot, ClaudeBot and More

More and more site owners want different rules for different kinds of bots: let search engines index everything, let AI search products cite pages, but keep training crawlers out — or the opposite. The rules all live in one plain-text file, and a single misplaced group can silently let a crawler through or block search traffic you wanted to keep.

The Robots.txt LLM Policy Checker evaluates a robots.txt for each named AI and search crawler the way a standards-following crawler would: which group it reads, which rules apply, whether it may fetch the home page and a path you choose, and which gotchas — ignored * rules, misspelled tokens, conflicting rules — affect it. It does not search the file for names; it applies the matching rules per crawler.

What is a robots.txt AI crawler policy?

In simple terms

robots.txt is a file at the root of a website (https://example.com/robots.txt) that tells crawlers which paths they may fetch. Each crawler identifies itself with a short name, its product token, such as GPTBot or ClaudeBot. An AI crawler policy is simply the set of robots.txt groups that address AI-related tokens — for example, “GPTBot may not crawl anything, OAI-SearchBot may crawl everything except /members/”.

Technical background

The Robots Exclusion Protocol is standardized in RFC 9309. The rules that matter for this checker:

• Groups. One or more User-agent: lines followed by Allow: and Disallow: rules form a group.
• Group selection. A crawler uses the group(s) whose user-agent value matches its product token, compared case-insensitively. Only if none matches does it use the * group. Groups naming the same token are combined.
• Rule matching. Within the chosen group, the rule with the longest matching path wins; if an Allow and a Disallow match with the same length, Allow wins. * matches any sequence of characters and $ anchors the end of the URL.
• Missing file. A 4xx answer means no restrictions. Server errors are treated cautiously; Google treats them as “disallow all” for a while.

AI-related tokens come in different kinds. Some are training crawlers (GPTBot, ClaudeBot, CCBot, Meta-ExternalAgent, Bytespider), some feed AI search and answers (OAI-SearchBot, Claude-SearchBot, PerplexityBot, Amazonbot, DuckAssistBot), some are user-initiated fetches (ChatGPT-User, Claude-User, Perplexity-User), and some are control tokens that no crawler sends at all: Google-Extended and Applebot-Extended are read by Googlebot and Applebot to decide whether crawled content may be used for AI models. Blocking a control token does not stop any crawling.

Why use a robots.txt LLM policy checker?

• The * trap. Adding a dedicated User-agent: GPTBot group with a single rule silently drops every rule in the * group for GPTBot.
• Typos and full user-agent strings. User-agent: Claude-Search-Bot or User-agent: GPTBot/1.0 matches nothing; the crawler falls back to *.
• Search vs. AI confusion. Blocking Googlebot instead of Google-Extended removes the site from Google Search crawling.
• Partial restrictions. “Disallow: /” combined with Allow rules is neither fully blocked nor fully open; the checker says so.
• Auditing other sites. See what policy a publisher or competitor has published, without reading the file line by line.

Who should use it?

• Site owners and publishers deciding how AI systems may use their content.
• SEO specialists making sure search crawling is not affected by AI rules.
• Web developers and DevOps people who generate robots.txt from templates.
• Legal, compliance or content teams who need to verify that a decided policy is actually implemented.

How the robots.txt LLM policy checker works

Fetch mode. You enter a domain or any URL on the site. The checker takes its origin (scheme, host and port; https:// is assumed when no scheme is given) and the NetTools.tech server requests exactly /robots.txt there — never another path. The request uses the SSRF-protected fetcher (public addresses only, ports 80, 443, 8080 and 8443, every redirect re-checked), follows up to 5 redirects, times out after 8 seconds and reads at most 500 KiB, the limit Google applies. Only a 2xx text response that is not an HTML page counts as a robots.txt. The file is returned to your browser and not stored.

Paste mode. The content you paste is parsed in your browser only.

Evaluation. The file is parsed with an RFC 9309 parser. For every selected token the checker:

1. Finds the groups naming that token (case-insensitive, exact token) or, failing that, the * group.
2. Merges the rules of those groups and evaluates / (the home page) and your test path with longest-match and Allow-wins-ties.
3. Classifies the crawler: no Disallow rules → allowed; / disallowed and no Allow rules → restricted; anything in between → mixed.
4. Adds notes: the token is named in several groups; it has its own group and therefore ignores specific Disallow rules from *; Allow and Disallow for the same path; the home page is blocked but Allow rules reopen paths.

File-level checks report parser problems and user-agent lines that will not match: values with versions, spaces or brackets, and near-misses of known tokens (“did you mean ClaudeBot?”). Changing the crawler selection or the test path re-evaluates instantly without fetching again.

How to use the NetTools.tech Robots.txt LLM Policy Checker

1. Open the Robots.txt LLM Policy Checker.
2. Choose Check a website and enter a Domain or URL, for example example.com, or choose Paste robots.txt and paste the content. Load example fills in a sample policy.
3. Under Crawlers to evaluate, tick the tokens you care about. Main crawlers selects GPTBot, OAI-SearchBot, ClaudeBot, Claude-SearchBot, Google-Extended, Googlebot, PerplexityBot, Bingbot and CCBot; All and None do what they say.
4. Optionally add Other user-agent tokens (comma-separated) and a Test a path such as /blog/post-1 (default /).
5. Press Check crawler policy, then expand each crawler card for its rules and notes.

Understanding the results

Each crawler card also shows the operator, its purpose category, a short description and a link to the operator’s documentation where one is known. The crawler list states when it was last reviewed; names and purposes change, and a token only tells you which group a crawler reads, not how it behaves.

Example: search allowed, AI training restricted

The built-in example (paste mode, Load example) contains:

User-agent: *
Disallow: /admin/
Disallow: /cart

User-agent: GPTBot
User-agent: ClaudeBot
User-agent: CCBot
Disallow: /

User-agent: Google-Extended
Disallow: /

User-agent: OAI-SearchBot
User-agent: PerplexityBot
Allow: /
Disallow: /members/

User-agent: Claude-Search-Bot
Disallow: /drafts/

Sitemap: https://www.example.com/sitemap.xml

With Main crawlers selected, GPTBot, ClaudeBot, CCBot and Google-Extended are Explicitly restricted. OAI-SearchBot and PerplexityBot are Mixed rules: they may crawl everything except /members/ — and their notes point out that, because they have their own group, they ignore Disallow: /admin/ and Disallow: /cart from the * group. Googlebot and Bingbot use the * group and are Mixed rules too.

Claude-SearchBot is the interesting one. The group header says Claude-Search-Bot, which is not the real token, so the File checks report a warning — did you mean “Claude-SearchBot”? — and the crawler falls back to the * group instead of the intended /drafts/ rule. The fix is to correct the token, and to repeat /admin/ and /cart in each named group if they should apply there too.

Common use cases

• Verifying a newly decided AI policy before and after deployment.
• Checking that a CMS or plugin update did not reset robots.txt.
• Confirming that Googlebot and Bingbot are unaffected by AI rules.
• Testing specific paths (/api/, /members/, /*.pdf$) for each crawler.
• Reviewing third-party sites’ published policies for research or partnerships.
• Validating general syntax and sitemaps in the Robots.txt Checker and the Sitemap Checker.

Troubleshooting

The crawler is shown as allowed although I blocked it

The group header probably does not match the token exactly: check File checks for “more than a product token” or “did you mean” warnings. Remember that matching is case-insensitive but otherwise exact; GPTBot-Extra does not match GPTBot.

A crawler ignores rules from User-agent: *

It has its own group. Copy the shared rules into that group; the card’s note lists which * Disallow rules are being ignored.

The result says “Unavailable / unclear”

The server answered with a 5xx error or 429, or did not respond. Fix the server error — Google treats a 5xx robots.txt as “disallow all” — or paste the file to evaluate it anyway.

Fetch shows HTTP 200 but every crawler is allowed

Look at the content type in the Retrieval line. Some sites answer every unknown URL with their normal HTML page (text/html, status 200). HTML responses are not accepted as robots.txt, so the checker evaluates the site as having no robots.txt and the Retrieval section says “No usable robots.txt” (likely a soft 404). Serve a real plain-text file at /robots.txt.

The website blocks the checker

Firewalls and bot protection sometimes refuse automated requests. Paste the file’s contents instead.

Common mistakes

• Blocking Googlebot when the goal was to opt out of AI training (Google-Extended).
• Expecting robots.txt to protect private content — anything reachable without login can still be fetched, and listing a path in robots.txt makes it public knowledge.
• Writing full user-agent strings or version numbers in User-agent: lines.
• Adding a named group and forgetting it replaces the * rules for that crawler.
• Assuming a block removes content already collected or used.
• Relying on outdated token lists; operators add and rename crawlers.

Limitations

• The checker evaluates what the file says, not what crawlers actually do. Whether an operator honors robots.txt, and how it treats user-initiated fetches, is up to the operator.
• The built-in crawler list is maintained by hand and dated; new or renamed tokens may be missing. Add them in Other user-agent tokens.
• It cannot verify that a request claiming to be a crawler really comes from that operator.
• Only /robots.txt on the given origin is fetched; other hosts and subdomains have their own files.
• Content beyond 500 KiB is ignored, as Google does.
• Other opt-out mechanisms (HTTP headers, meta tags, platform settings or contracts) are outside its scope.

Security & privacy

In fetch mode the domain is sent to the NetTools.tech server, which fetches only /robots.txt on that host and returns it to your browser; neither the domain nor the file is stored. In paste mode nothing leaves your browser. Per-client rate limits apply. robots.txt is a public file by design, so no sensitive data is involved — but do not rely on it to hide anything.

Related tools

• Robots.txt Checker — syntax validation, rules per user agent and a general URL tester. Guide: Robots.txt Checker guide.
• llms.txt Checker — the proposed file that points AI systems at useful content. Guide: llms.txt Checker guide.
• security.txt Checker — another well-known site file, for vulnerability reporting contacts. Guide: security.txt Checker guide.
• Sitemap Checker — validate sitemaps listed in robots.txt. Guide: Sitemap Checker guide.
• Meta Tag Analyzer — check robots meta tags on individual pages.
• Category overview: Web & SEO tools guide.

Summary

Each crawler follows exactly one robots.txt group — the one naming its token, or * — and within it the longest matching rule. The Robots.txt LLM Policy Checker applies these RFC 9309 rules per crawler for 18 known AI and search tokens plus any you add, and flags the common traps: ignored * rules, misspelled tokens and control tokens mistaken for crawlers. robots.txt remains a voluntary policy, so the result describes what compliant crawlers should do, not what every bot will do.