Web Bot Auth

Web Bot Auth is an IETF working group, and the protocol it is standardizing, that lets automated HTTP clients such as crawlers and AI agents sign their requests so a website can verify which operator sent them.

Status. The IETF approved the working group’s charter on 23 October 2025. Its protocol document is draft-ietf-webbotauth-httpsig-protocol-00, published on 1 September 2026 by authors from Cloudflare and Google. It is an Internet-Draft, a work in progress that can still change.

Scope. The charter targets websites built mainly for people, and names crawlers, archivers, link checkers, AI training crawlers and AI agents acting for end users as in scope. It lists HTTP APIs and agent-to-agent interfaces as out of scope, along with authenticating the end user and assigning reputation to bots. An A2A server therefore authenticates clients with the schemes in its Agent Card’s securitySchemes. Web Bot Auth is for the agent that browses a website.

How it works. The client signs each request with HTTP Message Signatures (RFC 9421) and sends three headers. Signature holds the signature. Signature-Input lists what it covers: @authority or @target-uri, plus created, expires, a keyid equal to the key’s JWK SHA-256 thumbprint, and tag="web-bot-auth". Signature-Agent holds a URL that identifies the signer and tells the site where to fetch its keys. By default that is a key directory, a JWK Set served at /.well-known/http-message-signatures-directory. The draft recommends signatures expire within 24 hours. Its example request, unwrapped:

GET /path/to/resource HTTP/1.1
Host: origin.example.com
Signature: sig=abc==
Signature-Input: sig=("@authority" "signature-agent";key="sig");created=1700000000;expires=1700011111;keyid="ba3e64==";tag="web-bot-auth"
Signature-Agent: sig="https://signer.example.com"

What it proves. A valid signature ties the request to whoever controls the key at the Signature-Agent URL. The draft warns that a signature covering only @authority can be replayed against the same site until it expires. Whether that operator is trustworthy is a policy decision the protocol leaves to the site.

Neighbouring terms. HTTP Message Signatures is the underlying mechanism. UCP business profiles can double as a Web Bot Auth key source.