web-bot-auth 1.0.0

Web Bot Auth Check v1.0.0

Status

• Version: 1.0.0
• Check identifier: web-bot-auth
• Input contract: [email protected]
• Output contract: [email protected]
• Scope: site
• Maturity: informational

Abstract

This check validates whether the scanned origin publicly participates in Web Bot Auth request signing as a bot or agent identity publisher. In concrete terms, it fetches /.well-known/http-message-signatures-directory and validates the public JWKS-style signing key directory that other websites would use to verify signed requests from this origin's bots or agents.

This check does not prove that the scanned website verifies incoming signed bots. That verifier-side behavior happens at the edge, WAF, or application layer and is not reliably observable from a normal external scan unless the site publicly documents or advertises it.

Motivation

Web Bot Auth is an emerging approach for cryptographically identifying automated HTTP clients. Instead of relying only on user-agent strings, IP ranges, or reverse DNS, a bot or agent can sign HTTP requests with HTTP Message Signatures and publish public verification keys in a well-known directory.

The check exists to answer a narrow question:

> Is there public evidence that this origin publishes Web Bot Auth signing keys, > and if so, is that key directory usable by verifiers?

For most ordinary websites, blogs, ecommerce sites, portfolios, and business homepages, the answer is expected to be "not applicable" because those sites do not operate public bots or agents that other sites need to verify.

Normative Model

The check follows the Web Bot Auth working-group scope and the HTTP Message Signatures Directory draft:

• Web Bot Auth authenticates automated HTTP clients, not end users.
• The public discovery resource is /.well-known/http-message-signatures-directory.
• The directory is a JSON Web Key Set containing public signing keys.
• The draft media type is application/http-message-signatures-directory+json.
• The directory should be served over HTTPS.
• JWK entries need key-type-specific public material, not only kid and kty.
• RFC 9421 Signature and Signature-Input response headers can bind the

directory response to published keys.

• Key rotation and caching matter because verifiers depend on fresh public keys.

The check treats application/jwk-set+json and generic JSON as compatibility signals, not as strict draft conformance.

Applicability

The check is applicable when at least one of these public signals is present:

• /.well-known/http-message-signatures-directory returns HTTP 200.
• The homepage headers or HTML mention Web Bot Auth, HTTP Message Signatures,

Signature-Agent, Signature-Input, or the well-known directory path.

• The scanned origin publicly claims to publish signing keys for bots or agents.

The check is not applicable when:

• The well-known path is absent and no support claim is visible.
• The site only publishes crawler policy files such as robots.txt, ai.txt,

TDMRep, RSL, or Content Signal.

• The site only authenticates humans or APIs with OAuth, OIDC, API keys, or

cookies.

• The question is whether the site verifies incoming signed bots internally.

Pass Criteria

The check passes when a public signing key directory is found and all required validation steps pass:

• The well-known directory returns HTTP 200.
• The final URL is HTTPS.
• The response uses application/http-message-signatures-directory+json.
• The body parses as a JSON object.
• The object contains a non-empty keys array.
• Each key is a public JWK with valid key-type-specific material:
• RSA: n, e
• EC: crv, x, y
• OKP: crv, x
• No private JWK fields are exposed.
• Signing intent and lifecycle metadata are usable when present.
• Cache guidance is present.
• Directory response signature headers are present and complete when used.

Warning Criteria

Warnings include:

• the site claims Web Bot Auth, HTTP Message Signatures, signed agents, or public bot signing keys but the directory is missing
• the directory is present but uses a generic JWKS or JSON content type rather than the draft media type
• cache guidance is missing or too weak for practical key rotation
• signing intent, lifecycle metadata, or key rotation hints are incomplete
• directory response signature headers are absent
• signature headers are present but incomplete
• public keys are valid but lack recommended metadata such as kid, alg, or intended use

Failure Criteria

Failures include:

• the directory is fetched but the response is not HTTPS
• the directory response is not JSON-compatible or cannot be parsed
• the root document is not a JSON object
• keys is missing, empty, or not an array
• no key contains valid key-type-specific public material
• private JWK material is exposed
• key entries are malformed or unsupported
• signature headers, when present, are malformed in a way that prevents binding the directory response
• public directory content exposes secrets, private keys, tokens, cookies, or credential-bearing URLs

Evidence Model

The check emits step-level evidence:

• Requested directory URL.
• Final URL.
• HTTP status.
• HTTPS status.
• Response Content-Type.
• Response Cache-Control.
• Whether homepage headers or HTML claimed support.
• JSON parse status.
• Key count and valid-key count.
• Per-key public metadata summary:
• kid preview, never full sensitive evidence.
• kty, alg, use, key_ops, and crv when present.
• Missing public fields.
• Private field exposure.
• JWK thumbprint match status when computable.
• RFC 9421 directory response signature header presence.
• Signature labels and required signature parameters when present.

Logs are emitted through the versioned check runtime wrapper. Logs must not include full keys, full signature values, cookies, credentials, authorization headers, or raw response bodies.

Validation/Scoring Steps

The version owns these internal steps:

• discover-directory (0.2): fetch the well-known directory and decide whether

the check applies.

• content-type (0.15): validate the draft media type and warn on compatibility

media types.

• jwks-shape (0.25): validate top-level JWKS structure and public key material.
• key-safety (0.15): detect private material, lifecycle problems, and signing

metadata issues.

• cache-lifecycle (0.1): inspect Cache-Control.
• directory-binding (0.15): inspect RFC 9421 Signature and Signature-Input

response headers.

Standard Behavior

When the directory is missing and no public claim is visible, the result is not_applicable. This is the expected outcome for most websites.

When the directory is missing but homepage evidence claims Web Bot Auth or HTTP Message Signatures support, the result is warning.

When a directory is present but malformed, unsafe, or not JSON, the result is fail.

When a directory is present and mostly usable but has interoperability gaps, such as generic JWKS content type, missing cache hints, incomplete signature response headers, or questionable key metadata, the result is warning.

Non-Standard and Real-World Behavior

Google documents an experimental Web Bot Auth rollout where some Google-Agent requests are signed and authenticated as https://agent.bot.goog. Google also states that not every request is signed and recommends fallback verification via IP addresses, reverse DNS, and user-agent strings.

Cloudflare maintains an open-source Web Bot Auth implementation with signing, verification, directory examples, Workers examples, Caddy examples, TypeScript packages, and Rust packages. That ecosystem shows practical implementation momentum, but the standard remains draft-stage.

Many major bot and AI crawler operators still rely on user-agent strings, robots.txt, reverse DNS, and published IP ranges rather than Web Bot Auth request signing. Therefore, absence of this directory is not a defect for ordinary websites.

Non-Goals and Limitations

This check does not:

• Verify that the scanned website accepts or enforces incoming signed bot

requests.

• Detect private WAF, CDN, or application-layer verification logic.
• Score crawler permission or content usage policy.
• Replace robots.txt, ai.txt, TDMRep, RSL, Content Signal, Auth.md, OAuth,

or API authentication checks.

• Assign reputation or intent to any bot.
• Prove that any live request from the scanned origin was signed.

References

• datatracker.ietf.org/wg/webbotauth/about
• datatracker.ietf.org/doc/html/draft-meunier-http-message-signatures-directory
• www.rfc-editor.org/rfc/rfc9421
• www.rfc-editor.org/rfc/rfc7517
• www.rfc-editor.org/rfc/rfc7638
• developers.google.com/crawling/docs/crawlers-fetchers/web-bot-auth
• agent.bot.goog/
• www.rfc-editor.org/rfc/rfc9421.html
• www.iana.org/assignments/http-message-signature/http-message-signature.xhtml
• www.rfc-editor.org/rfc/rfc7517.html
• www.rfc-editor.org/rfc/rfc7638.html
• developers.google.com/crawling/docs/crawlers-fetchers/verify-google-requests
• github.com/cloudflare/web-bot-auth

web-bot-auth v1.0.0 Changelog

• Clarified applicability: missing Web Bot Auth directories are not_applicable

unless the origin publicly claims Web Bot Auth or HTTP Message Signatures support.

• Validates the scanned origin as a signed bot/agent identity publisher, not as

proof that the website verifies incoming signed bots.

• Added draft media type, JWKS public key material, key safety, cache lifecycle,

and directory response signature header steps.

• Expanded versioned documentation with standards, real-world behavior, pass and

warning boundaries, evidence, and limitations.