llms.txt Checker – Complete Guide to the llms.txt Proposal, Structure and Validation

Documentation teams and site owners are being asked whether they “have an llms.txt yet”. Some publish one because a competitor did, others because an AI tool they use reads it. Either way, the file is plain Markdown with a specific shape, and small mistakes — a missing title, a list written in the wrong form, an HTML error page served at the address — are easy to make and hard to notice.

The llms.txt Checker fetches /llms.txt from a site, or takes pasted content, and parses it according to the llmstxt.org proposal: title, summary, details, sections and links. It reports structural problems, shows a readable rendering and the raw file with problem lines marked. Keep in mind throughout: llms.txt is a proposed convention, not a standard.

What is llms.txt?

In simple terms

llms.txt is a short text file at the root of a website (https://example.com/llms.txt) that gives language models and AI tools a concise overview of the site: what it is, and links to the pages that are most worth reading. Think of it as a hand-written table of contents for machine readers, written in Markdown.

Technical background

The format was proposed in 2024 and is described at llmstxt.org. It is not an IETF RFC, has no registered well-known URI and no body that enforces it. The proposal defines a fixed order:

1. # Title — an H1 heading with the name of the site or project. It is the only required part.
2. > Summary — a blockquote with a short description containing the key information.
3. Optional free text — paragraphs or lists with more detail, without headings.
4. ## Section headings, each followed by a Markdown list of links in the form - [Name](URL): optional notes.

A section titled Optional has special meaning: its links are secondary and can be skipped when a reader has little context available. The proposal also suggests publishing clean Markdown versions of important pages (commonly by appending .md to the page URL), and some sites add an llms-full.txt with the complete documentation in one file.

The file grants no permissions and imposes no restrictions. Whether AI crawlers may fetch your site is still governed by robots.txt — see the Robots.txt LLM Policy Checker.

Why use an llms.txt checker?

• Catch structural errors. A file that does not start with an H1, or whose links are not in the list form, may be ignored or misread by tools that parse the proposal.
• Detect fake files. Many sites return their normal HTML page with status 200 for any URL; the checker recognizes that this is not an llms.txt.
• Find broken references. Invalid, non-http(s), relative or duplicate links are flagged.
• Review before publishing. Paste a draft and check it before deploying.
• Get an honest answer. If the site has no llms.txt, the checker says that this is common and optional.

Who should use it?

• Documentation and developer-relations teams who maintain docs sites.
• Site owners and SEO specialists experimenting with AI-oriented content.
• Developers generating llms.txt automatically from a docs build.
• Anyone evaluating whether a site publishes one, and how complete it is.

How the llms.txt checker works

Fetch mode. You enter a domain or any URL on the site. The checker takes the origin (assuming https:// when no scheme is given) and the NetTools.tech server requests exactly /llms.txt there — no other path, and not llms-full.txt. The request goes through the SSRF-protected fetcher (public addresses only, ports 80, 443, 8080 and 8443, every redirect re-checked), follows up to 5 redirects, times out after 8 seconds and reads at most 1 MB. Only a 2xx text response that is not HTML counts as found. The file is returned to your browser and not stored.

Paste mode. The content is parsed in your browser only. The optional File address lets the checker resolve relative links.

Parsing and validation (in your browser, in both modes):

• An empty file or HTML content is an error. A content type other than text/plain or text/markdown produces a warning.
• The first non-empty line must be an H1; further H1 headings are warnings; H3–H6 headings are noted as not defined by the proposal.
• The first blockquote after the title becomes the summary; a missing summary is a warning, a very short one (under 20 characters) an info note.
• Paragraphs before the first H2 become details. Each H2 starts a section; Optional sections are marked.
• Inside sections, - [name](url): notes items become links; list items in another form are warnings. Sections without links and files without any sections are warnings.
• Links are checked for invalid URLs and non-http(s) schemes (errors), and relative URLs, http:// links and duplicates (info).
• Files over 200 KB get a note that full content belongs in llms-full.txt or linked .md files. If no link ends in .md, an info note mentions the Markdown suggestion.

How to use the NetTools.tech llms.txt Checker

1. Open the llms.txt Checker.
2. Choose Check a website and enter a Domain or URL, for example docs.example.com; or choose Paste llms.txt and paste the content, optionally with the File address.
3. Load example (paste mode) fills in a sample file with one deliberate mistake.
4. Press Check llms.txt.
5. Read Retrieval, the summary figures and Validation, then compare the Readable view with the Raw file. Copy content copies the file.

Understanding the results

Example: the built-in sample file

Load example fills in a file with the file address https://docs.example.com/llms.txt. Line 1 is the H1 title # Example Docs, line 2 is empty, and the rest of the file continues from line 3:

> Example Docs is the documentation for the Example API: authentication, endpoints, SDKs and rate limits.

The API is REST-based and returns JSON. All examples use https://api.example.com.

## Getting started

- [Quickstart](https://docs.example.com/quickstart.md): create a key and make your first request
- [Authentication](https://docs.example.com/auth.md): API keys and OAuth

## Reference

- [Endpoints](https://docs.example.com/reference/endpoints.md)
- [Errors](https://docs.example.com/reference/errors.md): error codes and retries
- Rate limits: see the dashboard

## Optional

- [Changelog](https://docs.example.com/changelog.md)

The checker finds the title “Example Docs”, the summary, one paragraph of details and three sections, one of them Optional. It counts five links, one of them optional. The status is Valid with warnings: line 16, - Rate limits: see the dashboard, is a list item that is not in the - [name](url): notes form. Either turn it into a link or move the sentence out of the list.

Common use cases

• Validating a generated llms.txt in a docs pipeline before each release.
• Checking that a CDN or framework serves the file as plain text rather than the site’s HTML shell.
• Comparing how different documentation sites structure their files.
• Reviewing the companion policy in robots.txt with the Robots.txt LLM Policy Checker.
• Making sure linked pages are in the Sitemap Checker results too, so search crawlers find them.

Troubleshooting

“Not found” with HTTP 200, or “This is an HTML page, not a Markdown llms.txt file”

The server returned a web page at /llms.txt. When it is labelled text/html, fetch mode does not accept it and reports no llms.txt despite the 200 status; when HTML arrives with another content type (or is pasted), the parser reports the HTML error. Make sure the file exists and that your framework or single-page app does not catch the route; serve it as a static text file.

“Served as … text/plain or text/markdown is expected”

The file is found but delivered with another content type (for example application/octet-stream). Configure the server to send text/plain; charset=utf-8 or text/markdown.

“The file must start with an H1 heading”

The first non-empty line is not # Something. Front matter, a comment or a second-level heading at the top triggers this.

Links are not counted

They are not in the list form. Each link must be its own list item: - [Name](https://example.com/page.md) optionally followed by : notes. Inline links in paragraphs are not counted as section links.

Not found although the file exists

The file may be on another host (for example docs.example.com rather than example.com) — the checker only looks at the origin you entered. It may also be blocked for automated requests; paste the content instead.

Common mistakes

• Treating llms.txt as a way to block or allow AI crawlers — that is robots.txt’s job.
• Expecting publication to guarantee indexing, citation or traffic from AI products.
• Putting the whole documentation into llms.txt instead of a concise index.
• Using headings other than H1 and H2 for structure.
• Writing links as plain text or inline in paragraphs inside sections.
• Linking to pages that require a login or redirect to a consent page.

Limitations

• llms.txt is a proposal; the checker validates against the llmstxt.org description, and other tools may interpret the file differently.
• It does not fetch the linked pages, so broken targets are not detected — use the Broken Link Checker for that.
• It only checks /llms.txt on the origin you enter; llms-full.txt and .md page versions are not fetched.
• It cannot tell you which AI systems read the file or how they use it.
• Files larger than 1 MB are truncated in fetch mode.

Security & privacy

In fetch mode the domain is sent to the NetTools.tech server, which fetches only /llms.txt on that host and returns it to your browser; neither the domain nor the file is stored. In paste mode nothing leaves your browser. The readable view is built from the parsed structure, not by inserting raw HTML, so pasted content cannot run scripts on the page. llms.txt is public by design — do not put internal URLs or confidential notes in it.

Related tools

• Robots.txt LLM Policy Checker — which AI crawlers may fetch your site. Guide: AI crawler policy guide.
• Robots.txt Checker — general robots.txt validation and testing. Guide: Robots.txt Checker guide.
• Sitemap Checker — the complete list of URLs for search crawlers. Guide: Sitemap Checker guide.
• security.txt Checker — another well-known site file. Guide: security.txt Checker guide.
• Meta Tag Analyzer — metadata of the pages you link.
• Category overview: Web & SEO tools guide.

Summary

llms.txt is a voluntary Markdown index for AI readers, proposed in 2024 — useful as a curated pointer to your best content, but not a standard and not a guarantee of anything. The llms.txt Checker fetches or takes the file, checks title, summary, sections and links against the proposal, and shows exactly which lines deviate. It does not follow the links or control crawling; pair it with robots.txt and sitemap checks.