Open Terminal

Robots.txt Checker

Available

Check a website’s robots.txt: formatted rules per user agent, syntax validation, listed sitemaps, and a tester showing whether a URL path is allowed.

Check a robots.txt

The file is always read from the root of the host: https://example.com/robots.txt

robots.txt report

Fetch or paste a robots.txt to see its rules, problems and a URL tester.

How it works

robots.txt lives at the root of a host (https://example.com/robots.txt) and tells crawlers which paths they may fetch. Rules are grouped by User-agent; for a given crawler the most specific path rule wins, and Allow wins a tie. Paste mode parses and tests a robots.txt entirely in your browser; fetching a live file needs the NetTools server.

How crawlers read the file

  • Lines are grouped under one or more User-agent lines. A crawler uses the group naming it most specifically and ignores the rest; * is the fallback.
  • Within the group, the rule with the longest matching path decides. If an Allow and a Disallow match equally long paths, Allow wins.
  • * matches any sequence of characters, $ marks the end of the URL. Matching is case-sensitive.
  • Sitemap lines apply to the whole file and must be absolute URLs.

Safeguards & responsible use

  • Fetches only /robots.txt on public hosts, following at most 5 redirects; content beyond 500 KiB is ignored (the limit Google applies).
  • Per-client rate limits and timeouts; fetched content is not stored.

Frequently asked questions

Does Disallow remove a page from search results?

No. Disallow stops crawling, not indexing: a blocked URL can still appear in results if other pages link to it. To keep a page out of the index, allow crawling and use a noindex robots meta tag or X-Robots-Tag header.

What happens if robots.txt is missing?

A 404 or other 4xx response means crawlers may fetch everything. Server errors (5xx) are different: Google treats them as “disallow all” for a while.

Which rule wins when Allow and Disallow both match?

The rule with the longest matching path. If both are equally long, Allow wins. * matches any characters and $ anchors the end of the URL.

Some websites block the checker. Why?

Firewalls and bot protection sometimes refuse automated requests. The checker then shows the status it received; paste the file’s contents to validate it anyway.