Robots.txt Checker – Complete Guide to Crawl Rules, Validation and Testing

A single misplaced line in robots.txt can stop search engines from crawling an entire site, or let them crawl thousands of useless filter and search URLs. The file is short, plain text and easy to edit, which is exactly why mistakes slip through: a staging Disallow: / that went live, a rule that never matches because of a missing slash, a misspelled directive that crawlers silently ignore.

The Robots.txt Checker fetches a site's robots.txt (or analyzes one you paste), validates every line, lists the groups and sitemaps it contains, shows the file with line numbers and highlighted problems, and includes a tester that tells you whether a given crawler may fetch a given URL, and which line decided.

What is robots.txt?

In simple terms

robots.txt is a text file at the root of a website, such as https://example.com/robots.txt, that tells automated crawlers which parts of the site they should not fetch. Search engines and most reputable bots read it before crawling and follow it. It is a polite request, not a lock.

Technical background

The Robots Exclusion Protocol was used informally for decades and was standardized in 2022 as RFC 9309. Its main rules:

• Location. One file per host and protocol, at /robots.txt. https://example.com/robots.txt does not apply to https://shop.example.com/ or to http://example.com/.
• Groups. One or more User-agent: lines start a group, followed by Allow: and Disallow: rules. A crawler uses the group that names its product token (for example Googlebot); if there is none, it uses the * group. Several groups for the same agent are merged.
• Matching. Rules are path prefixes, matched case-sensitively against the URL path and query. * matches any sequence of characters and $ anchors the end of the URL.
• Precedence. The most specific (longest) matching rule wins. If an Allow and a Disallow rule match with the same length, Allow wins.
• Defaults. No matching rule means allowed. An empty Disallow: allows everything. /robots.txt itself is always allowed.
• Size. Crawlers must parse at least 500 KiB; Google ignores content beyond that.
• Status codes. A 4xx response means "no restrictions". A 5xx response or no answer means crawlers should assume a full disallow for a while.

Common extensions outside the RFC: Sitemap: lines (absolute URLs, valid anywhere in the file), Crawl-delay: (ignored by Google), and Yandex-specific Host: and Clean-param:. Noindex: in robots.txt is not supported.

Why use a robots.txt checker?

• Catch site-wide blocks. A Disallow: / for all agents, often left over from staging, stops all crawling.
• Find rules that never match. Paths that do not start with / or *, or that differ in case from real URLs.
• Spot ignored lines. Misspelled directives (Useragent, Dissallow), rules before any User-agent line, unsupported noindex.
• Test specific URLs. Before launching a section or after a crawling drop, see whether Googlebot may fetch a path.
• Check sitemaps. Confirm Sitemap: lines are absolute and jump to the Sitemap Checker for each.

Who should use it?

• SEO specialists auditing crawlability and diagnosing indexing drops.
• Web developers and DevOps engineers deploying sites, especially when promoting staging configuration to production.
• Site owners who want to control which crawlers fetch which sections.
• Anyone writing a robots.txt for the first time and wanting a syntax check before publishing.

How the robots.txt checker works

Paste mode runs entirely in your browser: the text is parsed, validated and tested with the same library the fetch mode uses, and nothing is uploaded.

Fetch mode uses the NetTools.tech server:

1. You can enter a domain or any URL on the site. The checker always builds the root address, <scheme>://<host>/robots.txt, keeping http:// or a port if you typed one; without a scheme it uses https://. The hint under the field shows the exact address that will be fetched.
2. The server validates the URL and runs the same destination guard as the other web tools: public addresses only, ports 80, 443, 8080 and 8443, no credentials.
3. It requests the file with Accept: text/plain, the identified user agent NetToolsBot/1.0, no cookies and an 8-second limit, following at most 5 redirects (each re-checked). Content beyond 500 KiB is cut off, matching Google's limit.
4. The body is returned only for a 2xx response with a text content type that is not HTML. Otherwise the status is reported and there is no content to parse.
5. Your browser parses the file and shows the report. Nothing is stored on the server.

The parser records each line, groups consecutive User-agent lines into one group, attaches Allow, Disallow and Crawl-delay to the current group, collects Sitemap lines, and produces errors, warnings and notes for problems it finds.

How to use the NetTools.tech Robots.txt Checker

1. Open the Robots.txt Checker.
2. Choose a source tab:
   • Fetch from website: type a domain or URL in Domain or URL, for example example.com.
   • Paste content: paste the file into robots.txt content, or press Load example for a sample file.
3. Press Check robots.txt.
4. Review the summary tiles and the Validation list.
5. In Test a URL, enter a User agent (default Googlebot) and a URL or path (default /). The verdict updates as you type; full URLs are reduced to path and query.
6. Check the Groups table, the Sitemaps list (each with a Check sitemap link) and the File content with line numbers. Lines with problems are highlighted; hover over them to read the message.

Understanding the results

Example: testing rules in a typical file

Paste this file (the rules of Load example, without its comment line):

User-agent: *
Disallow: /admin/
Disallow: /search
Allow: /search/help
Disallow: /*.pdf$

User-agent: Googlebot-Image
Disallow: /private-images/

User-agent: ExampleBot
Disallow: /

Sitemap: https://www.example.com/sitemap.xml

The summary shows 3 groups, 6 rules, 1 sitemap and no errors or warnings. Tests with User agent Googlebot, which has no own group and therefore uses *:

With User agent Googlebot-Image, /private-images/a.jpg is blocked by line 8, but /admin/ is allowed: that crawler has its own group and does not inherit the * rules.

Common use cases

• Pre-launch check: make sure production does not serve the staging file.
• Indexing drop investigation: test the affected URLs against the live file.
• Faceted navigation: verify that filter and sort parameters are blocked while category pages remain crawlable.
• Crawler-specific policies: check groups for individual bots; for AI crawlers, the Robots.txt LLM Policy Checker gives a dedicated overview.
• Drafting changes: paste an edited version and test it before deploying.

Troubleshooting

HTTP 404 and "No User-agent groups"

There is no robots.txt; crawlers may fetch everything. That is fine if intended. Otherwise, check that the file is deployed at the root of the exact host and scheme you tested.

HTTP 200 but no groups or rules

The server answered successfully but the content was not usable as robots.txt, typically because it was served as text/html (a CMS page or soft 404) or was empty. Crawlers would find no valid rules either; for an HTML response the Validation list says “No usable robots.txt” instead of “rules apply”. Serve the file as text/plain.

HTTP 5xx or 429

Google treats the whole site as disallowed until robots.txt can be fetched again. Fix the server error or rate limiting for /robots.txt urgently.

The request fails or is blocked

Some firewalls and bot-protection services refuse automated requests or block our server. Copy the file from your browser and use Paste content.

"Paths should start with / or *"

A rule like Disallow: admin/ never matches, because URL paths always begin with /.

A rule seems ignored by one crawler

That crawler probably has its own group. Crawlers use only the most specific group that names them, not that group plus *.

Common mistakes

• Using robots.txt to hide pages from search. Blocked URLs can still be indexed from links; use noindex and allow crawling.
• Blocking CSS and JavaScript. Search engines need them to render pages.
• Expecting inheritance between groups. A specific group replaces * for that crawler.
• Relative Sitemap URLs. Sitemap: /sitemap.xml is invalid; use the full URL.
• Listing secret paths. The file is public; protect private areas with authentication.
• Forgetting other hosts. Each subdomain and each scheme needs its own robots.txt.

Limitations

• The tester follows RFC 9309 with Google's documented precedence. Individual crawlers can interpret edge cases differently, and some bots ignore robots.txt entirely.
• User agents are matched by exact product token (for example googlebot, googlebot-image); the tester does not reproduce any crawler's private fallback logic.
• Fetch mode reads the file from one server location with the NetToolsBot user agent. A site that serves different robots.txt files by user agent or region may show something else to Googlebot.
• Only public hosts on ports 80, 443, 8080 and 8443 can be fetched; content beyond 500 KiB is cut off.
• The checker does not crawl the site, so it cannot tell you which real URLs a rule affects; test representative paths yourself.

Security & privacy

In paste mode, the content never leaves your browser. In fetch mode, the domain is sent to the NetTools.tech server, which requests /robots.txt from the target and returns it; neither the URL nor the content is stored. The target site sees a request from our server's IP address with the NetToolsBot user agent. robots.txt is public by design, so there is normally nothing sensitive in it; if yours lists private paths, that is a reason to move them behind authentication.

Related tools

• Robots.txt LLM Policy Checker – how a robots.txt treats AI and LLM crawlers (guide).
• Sitemap Checker – validate the sitemaps listed in robots.txt (guide).
• Meta Tag Analyzer – robots meta tags, X-Robots-Tag and canonical on a page (guide).
• Redirect Checker – see where /robots.txt redirects to (guide).
• HTTP Status – status code and headers of the robots.txt URL (guide).

Summary

robots.txt tells compliant crawlers which paths they may fetch, using per-agent groups, longest-match rules and a tie that favors Allow. The NetTools.tech Robots.txt Checker fetches the file from the root of a host or analyzes pasted content in your browser, flags syntax problems and ineffective rules, and tests individual URLs for any user agent. Remember that it controls crawling, not indexing, and is not an access control.