Robots.txt LLM Policy Checker
AvailableCheck which robots.txt rules apply to AI and search crawlers such as GPTBot, ClaudeBot, Google-Extended and PerplexityBot, and whether each is allowed, restricted or partly restricted.
Check a robots.txt
Crawlers to evaluate
Crawler list reviewed 2026-10. A crawler’s name only tells which group it reads; how it behaves is up to its operator.
Crawler policy
Sample data, design preview. These results are made up, using example.com addresses, to show the layout. No request was sent and nothing here is a real measurement.
Working…
Fetch or paste a robots.txt to see which rules apply to each crawler.
How it works
Each crawler reads exactly one robots.txt group: the one naming its product token (matched ignoring case), or the * group when none does. Within that group the longest matching Allow or Disallow path wins (RFC 9309). The checker applies these rules per crawler instead of searching the file for names. Paste mode works in your browser; fetching a live robots.txt needs the NetTools server.
What robots.txt can and cannot do
- It is a voluntary policy (RFC 9309), not access control. Reputable crawlers follow it; nothing forces them to.
- It does not protect private content. Anything reachable without a login can be fetched, and listing a path in robots.txt makes it public knowledge.
- A crawler with its own group ignores the
*group completely. Repeat shared rules in each named group. - Blocking an AI training crawler does not remove content that was collected earlier.
Safeguards & responsible use
- Fetches only /robots.txt on public hosts through SSRF protection (every redirect re-checked), with a 500 KiB size limit and short timeouts.
- Per-client rate limits; neither the domain nor the file is stored.
Frequently asked questions
Does blocking a crawler in robots.txt stop it from accessing my site?
No. robots.txt is a voluntary policy: well-behaved crawlers follow it, but it is not access control and cannot enforce anything. Use authentication or server-side blocking for content that must stay private — and note that listing a path in robots.txt makes it public knowledge.
Why does a crawler ignore my “User-agent: *” rules?
A crawler follows only the most specific group that names it. As soon as a group for GPTBot exists, GPTBot ignores the * group entirely, so rules that should apply to everyone must be repeated in its group.
Does blocking Google-Extended remove my site from Google Search?
No. Google-Extended is a control token, not a crawler: Google crawls with Googlebot and checks this token to decide whether content may be used for its Gemini models. Search crawling is controlled by the Googlebot rules.
Are user-initiated AI fetches covered by robots.txt?
Operators treat this differently. Some say their user-triggered agents (such as ChatGPT-User or Perplexity-User) may not follow robots.txt, because a person requested the page. Check each operator’s documentation.