llms 1.0.0

llms.txt Check v1.0.0

Status

• Version: 1.0.0
• Check identifier: llms
• Input contract: [email protected]
• Output contract: [email protected]
• Scope: site
• Maturity: emerging recommendation

Abstract

This check validates whether a site publishes a useful root /llms.txt file for agent-readable discovery. It treats llms.txt as an emerging Markdown convention, not as an IETF, W3C, browser, search engine, or crawler-control standard. The check verifies that the root file exists, has a Markdown/text shape, contains a top-level title and useful links, follows the proposed structure where practical, and advertises links that are safe and reachable enough to help agents.

/llms-full.txt is inspected only as optional supporting evidence. It cannot replace the required root /llms.txt index.

Motivation

Language-model clients benefit from concise, curated site context before they crawl broadly. The llms.txt proposal gives site owners a simple Markdown index where they can describe the site and point agents toward important documentation, API references, policies, examples, and other high-signal resources.

The file is only useful when its links are curated, public, descriptive, and working. A syntactically present file with broken, private, misleading, or low-signal links may create more confusion than value.

Normative Model

The check uses the public llms.txt proposal as the primary model:

1. A site should publish a root /llms.txt resource.
2. The resource should be Markdown or plain text compatible.
3. The document should begin with a single H1 title naming the site, product, project, or documentation corpus.
4. A short blockquote summary after the H1 is preferred.
5. H2 sections should group important links.
6. Links should be Markdown links with descriptive labels.
7. /llms-full.txt may provide a larger complete corpus, but the concise index remains /llms.txt.

The check also recognizes adjacent standards:

• RFC 7763 registers text/markdown as a media type for Markdown resources.
• CommonMark defines a stable Markdown syntax family, but llms.txt does not require strict CommonMark conformance.
• RFC 9309 defines robots.txt crawler policy. llms.txt is not an access-control or AI-training opt-out mechanism.

Applicability

This check applies to public websites, documentation sites, API providers, SaaS products, developer platforms, open-source projects, and content sites that want to expose curated agent-readable context.

It is especially relevant when a site has public documentation, API references, changelogs, support docs, pricing, policy pages, examples, or Markdown mirrors.

Pass Criteria

The root /llms.txt resource must:

• Return a successful HTTP response.
• Use a compatible content type such as text/markdown, another Markdown-marked type, or text/plain.
• Not be an HTML error page, login wall, or JavaScript shell.
• Include an H1 title.
• Contain enough readable text to be useful.
• Include at least one usable HTTP(S), same-origin, or root-relative link.
• Avoid private, internal, credentialed, non-HTTP(S), admin, account, dashboard, checkout, preview, staging, or similar sensitive targets.
• Have at least one sampled linked target that is reachable and agent-readable.

Warnings are emitted when the file is present but weak:

• Missing blockquote summary.
• Missing H2 sections.
• Missing Markdown-formatted links.
• Vague link labels such as raw URLs or "click here".
• Few or no links to agent-friendly resources such as Markdown, API docs, reference docs, changelogs, OpenAPI, JSON, XML, schema, examples, policies, or support docs.
• A minority of sampled links are broken or not clearly agent-readable.
• /llms-full.txt is present but is not text/Markdown compatible. Absence of /llms-full.txt is informational only in this version.

Failures are emitted when:

• /llms.txt is missing or unreachable.
• The root file is not text/Markdown compatible.
• The body looks like HTML instead of Markdown/text.
• The document lacks an H1 title.
• The document is too thin to be useful.
• No usable links are present.
• Unsafe/private links are advertised.
• Most sampled links are broken, or none of the sampled links return agent-readable content.

Evidence Model

The check emits step evidence for:

• /llms.txt fetch status, content type, byte/text length, and excerpt.
• H1 title and line number.
• Blockquote summary presence.
• H2 section headings.
• Link counts by total, usable, unsafe, Markdown-formatted, raw URL, internal, external, descriptive labels, and agent-friendly hints.
• Sample link records with labels, normalized URLs, same-origin status, safety classification, and agent-friendly hints.
• Bounded link probes with method, final URL, status code, content type, reachability, agent-readability, errors, and short excerpts.
• Optional /llms-full.txt presence, status, content type, length, and likely full-corpus signal.

The check caps excerpts and does not log full files or full linked-resource bodies.

Validation And Scoring Steps

1. Fetch root llms.txt.

• Weight: 20%.
• Required.
• Fails when /llms.txt cannot be fetched successfully.

1. Validate Markdown discovery shape.

• Weight: 30%.
• Checks content type, H1 title, text length, HTML leakage, and usable links.
• Fails when the file does not meet the minimum discovery shape.

1. Score structure and usefulness.

• Weight: 20%.
• Checks blockquote summary, H2 sections, Markdown links, descriptive labels, agent-friendly hints, duplicate links, and unsafe links.
• Warns for weak structure.
• Fails for unsafe/private links.

1. Probe sampled linked resources.

• Weight: 25%.
• Probes a bounded sample of up to 25 links.
• Prioritizes links that look high-value for agents.
• Uses HEAD first and falls back to lightweight GET.
• Follows redirects with timeouts and short excerpts.
• Fails when no safe public links can be probed, most sampled links are broken, or none return agent-readable content.
• Warns when a minority of sampled links are broken or weak.

1. Inspect optional llms-full.txt.

• Weight: 0%.
• Records whether /llms-full.txt exists and appears text/Markdown compatible.
• Informational only in this version.

Standard Behavior

There is no formal llms.txt standard today. This check therefore does not claim compliance with an IETF, W3C, browser, or search engine standard.

The strongest standards-backed assertions are:

• Markdown resources should use an appropriate Markdown or text media type.
• Markdown syntax should be parseable as a broadly compatible Markdown family.
• Crawler access control belongs to robots.txt, not llms.txt.

The check accepts text/plain because production llms.txt files commonly use it, including prominent documentation sites.

Non-Standard And Real-World Behavior

Production implementations vary:

• Some documentation sites publish compact text/plain files with long lines and dense Markdown links.
• Some developer platforms publish both /llms.txt and /llms-full.txt.
• Some docs platforms expose per-section llms.txt, per-section llms-full.txt, and .md mirrors.
• WordPress and documentation generators can produce llms.txt automatically.
• Some useful real-world files do not perfectly follow the proposal's blockquote and section layout.

This version treats the proposal structure as quality evidence rather than an absolute syntax gate. It remains stricter about unsafe links, broken links, missing titles, and missing root discovery.

Non-Goals And Limitations

This check does not:

• Treat llms.txt as crawler policy, AI-training consent, or access control.
• Prove that OpenAI, Google, Anthropic, Perplexity, or another AI platform honors the file.
• Crawl every linked resource recursively.
• Validate the semantic correctness of every linked page.
• Prove strict CommonMark conformance.
• Require /llms-full.txt.
• Require every linked resource to be Markdown.
• Fetch private, credentialed, internal, localhost, or non-HTTP(S) targets.

Link usefulness is heuristic. The check uses labels, URL patterns, content types, reachability, and short response excerpts as evidence, but it cannot fully know a site's editorial intent.

References

• llmstxt.org/
• github.com/AnswerDotAI/llms-txt
• www.rfc-editor.org/rfc/rfc7763
• spec.commonmark.org/0.31.2
• www.rfc-editor.org/rfc/rfc9309
• developers.openai.com/api/docs/bots
• developers.google.com/search/docs/crawling-indexing/google-common-crawlers
• developers.cloudflare.com/docs-for-agents
• developer.yoast.com/features/llms-txt/functional-specification

llms v1.0.0 Changelog

• Migrated runtime ownership into lib/checks/llms/versions/1.0.0.
• Requires root /llms.txt discovery instead of allowing /llms-full.txt to substitute for the concise index.
• Added proposal-aware Markdown structure validation for H1 title, summary, H2 sections, Markdown links, descriptive labels, unsafe targets, and agent-friendly link hints.
• Added bounded link probing for sampled llms.txt links with warnings or failures for broken, unsafe, or non-agent-readable targets.
• Added informational inspection of optional /llms-full.txt.
• Expanded versioned documentation with standards, platform guidance, real-world behavior, pass/warn/fail boundaries, evidence, and limitations.