robots.txt 1.0.0

robots.txt Check v1.0.0

Validates that a site publishes a fetchable and parseable /robots.txt file.

This check separates the formal Robots Exclusion Protocol from popular nonstandard extensions. The RFC-defined protocol is intentionally small; search engines, AI crawlers, and site owners commonly add extra records that should be parsed and reported, but not treated as RFC-required directives.

Input Contract

[email protected]

Requires the scan origin. The check fetches ${origin}/robots.txt.

Output Contract

[email protected]

Emits the parsed robots body, sitemap directives, Content-Signal directives, and a report check result. The result is stepped: fetch, RFC core syntax, and extension classification.

Pass Criteria

• /robots.txt responds with a successful HTTP status.
• The body parses as valid robots.txt content.
• At least one valid User-agent group is present.

Failure Criteria

• /robots.txt cannot be fetched.
• /robots.txt responds with a non-2xx status.
• The body cannot be parsed as a valid robots.txt file.

Warning Criteria

• The RFC core syntax is valid, but unsupported legacy records such as Noindex

are present.

HTTP Status Semantics

The check records the HTTP status because crawlers interpret robots.txt fetch outcomes differently from ordinary page fetches.

For CanAgentUse scoring, a non-2xx response fails this check because the site did not publish a fetchable policy document at /robots.txt, even though crawler behavior for 4xx can be permissive.

RFC 9309 Core Protocol

RFC 9309 defines the core Robots Exclusion Protocol. The core records are:

Core structure:

• A file is made of one or more groups.
• A group begins with one or more User-agent records.
• A group may contain zero or more Allow or Disallow rules.
• Empty lines separate groups.
• Comments begin with #.
• Rules before the first User-agent record are ignored by compliant parsers.
• Matching is path-based.
• The longest matching rule wins.
• If an equivalent Allow and Disallow both match, Allow wins.
• * can match zero or more characters.
• $ anchors a pattern to the end of the path.
• Unknown records do not invalidate the file; parsers should ignore unsupported

records and continue processing valid lines.

Example:

User-agent: *
Allow: /
Disallow: /admin/

User-agent: ExampleBot
Disallow: /private/
Allow: /private/public/

Popular Nonstandard Extensions

These records are not part of the RFC 9309 core rule set, but are common enough that the check should parse and report them separately from RFC validity.

Extension records should not make an otherwise valid file fail. They should be classified as known extensions, unsupported legacy records, or unknown records. This version emits that classification in the extensions step evidence.

Known User-Agent Tokens

User-agent tokens are not centrally registered and change over time. The check should treat unknown tokens as syntactically valid when they fit the RFC product token shape, while classifying known tokens for reporting and AI/search policy analysis.

Search and platform crawlers:

APIs-Google
AdsBot-Google
Applebot
Applebot-Extended
Baiduspider
BingPreview
DuckDuckBot
Google-CloudVertexBot
Google-Extended
Google-InspectionTool
GoogleOther
GoogleOther-Image
GoogleOther-Video
Googlebot
Googlebot-Image
Googlebot-News
Googlebot-Video
KagiBot
Mediapartners-Google
MojeekBot
PetalBot
Qwantify
SeznamBot
Slurp
Sogou
Storebot-Google
YandexBot
YandexImages
Yeti
bingbot

AI and answer-engine crawlers:

Amazonbot
Bytespider
CCBot
ChatGPT-User
Claude-SearchBot
Claude-User
ClaudeBot
Cohere-AI
Diffbot
FacebookBot
GPTBot
ImagesiftBot
Meta-ExternalAgent
Meta-ExternalFetcher
OAI-AdsBot
OAI-SearchBot
Perplexity-User
PerplexityBot
YouBot
omgili
omgilibot

SEO and commercial crawlers:

AhrefsBot
DataForSeoBot
DotBot
MJ12bot
Screaming Frog SEO Spider
SemrushBot

Source Notes

• RFC core behavior comes from RFC 9309: https://www.rfc-editor.org/rfc/rfc9309.html.
• Google documents supported robots fields and unsupported records in its robots.txt documentation: https://developers.google.com/search/docs/crawling-indexing/robots/robots_txt.
• Google crawler and fetcher tokens are documented at https://developers.google.com/search/docs/crawling-indexing/google-common-crawlers.
• OpenAI crawler tokens are documented at https://platform.openai.com/docs/bots.
• Anthropic crawler behavior, including robots.txt and crawl-delay handling, is documented at https://support.anthropic.com/en/articles/8896518-does-anthropic-crawl-data-from-the-web-and-how-can-site-owners-block-the-crawler.
• Applebot and Applebot-Extended are documented at https://support.apple.com/en-us/119829.
• Common Crawl documents CCBot at https://commoncrawl.org/ccbot.
• IndexNow key-location behavior is documented at https://www.indexnow.org/documentation.

Scoring Steps

Current v1.0.0 Coverage

This version checks:

• Fetch success for /robots.txt.
• Presence of at least one valid User-agent group.
• Core Allow and Disallow rule grouping.
• Sitemap and Content-Signal evidence.
• Known nonstandard extension records such as Crawl-delay, Host, Clean-param, and IndexNow-Key.
• Unsupported legacy Noindex records as warnings.
• Unknown records as evidence.
• Known and unknown crawler product tokens as evidence.

This version documents, but does not yet validate:

• Full RFC longest-match path evaluation for specific URLs.
• Whether specific AI/search crawler policies are desirable for the site.
• Whether sitemap URLs advertised in robots.txt are valid; the sitemap check owns sitemap validation.

References

• www.rfc-editor.org/rfc/rfc9309
• developers.google.com/search/docs/crawling-indexing/robots/intro
• example.com/sitemap.xml
• example.com/key.txt
• www.rfc-editor.org/rfc/rfc9309.html
• developers.google.com/search/docs/crawling-indexing/robots/robots_txt
• developers.google.com/search/docs/crawling-indexing/google-common-crawlers
• platform.openai.com/docs/bots
• support.anthropic.com/en/articles/8896518-does-anthropic-crawl-data-from-the-web-and-how-can-site-owners-block-the-crawler
• support.apple.com/en-us/119829
• commoncrawl.org/ccbot
• www.indexnow.org/documentation

robots.txt v1.0.0 Changelog

Initial versioned package for robots.txt.

• Adds stepped scoring for fetch, RFC core syntax, and extension classification.
• Emits known extension records, unsupported legacy records, unknown records, orphan records, invalid lines, and known/unknown crawler token evidence.
• Warns on unsupported legacy Noindex records while preserving pass/fail behavior for the RFC core syntax.