EmissarRequest early access
Menu

EmissarBot

EmissarBot is Emissar's crawler. It checks whether public websites publish an A2A Agent Card, so we can measure how many organizations offer an agent that other agents can reach.

User agent
EmissarBot/0.1 (+https://emissar.ai/bot)
robots.txt token
EmissarBot
Signed requests
Web Bot Auth, key directory at emissar.ai/.well-known/http-message-signatures-directory (how to check)
Operated by
Emissar
Updated

What it requests

Three paths per site, over HTTPS only, in this order, and no URL more than once per crawl:

PathWhy
/robots.txtAlways first. If the rules for EmissarBot, or the rules for all crawlers (*), disallow a path below, EmissarBot does not request it.
/.well-known/agent-card.jsonThe A2A v1.0 location for an Agent Card.
/.well-known/agent.jsonThe older A2A 0.3 location. Requested only when the first path did not return an Agent Card.

It follows up to three redirects, and only within the same site: example.com and www.example.com count as the same site. It never follows a redirect to another host, including a subdomain such as api.example.com, and never to a path your robots.txt disallows. It stops waiting after five seconds, and reads at most 256 KB of a card and 500 KiB of a robots.txt file. It requests nothing else: no pages, no links, no scripts, no images, no forms, and no logins.

Schedule

Once a month, EmissarBot checks the sites in the Tranco list, a public research ranking of the top one million websites. Once a week, it re-checks only the sites that published an Agent Card in the latest monthly crawl.

The first full crawl started on .

Requests are spread out. Each site's requests are made one after another, and no more than 60 requests are in flight across all sites at the same time.

What we keep

SiteWhat we keep
Sites that publish an Agent CardThe card itself, plus facts from it: agent name, provider organization, number of skills, declared capabilities, whether the card is signed, which authentication schemes it lists, which A2A version it follows, the final URL, and the Content-Type, Cache-Control and ETag response headers
Sites without an Agent CardNo record of the site. Only totals, such as how many sites in a rank range returned 404
Shown publicly, for sites with an Agent CardIn the agent directory at /directory: the agent's name, description, provider, interfaces (host names only), declared capabilities, authentication scheme types, whether the card is signed, its skills, the card's URL, its Tranco rank and when it was last fetched. Never images, skill examples or email addresses

An Agent Card is a document a site publishes for anyone to read. We use these facts to report how A2A is being adopted, and we show facts from each card in our public agent directory. To have the record of your card removed, or your directory entry corrected or removed, send us a message through the contact form. We remove it within seven days.

How to opt out

EmissarBot reads robots.txt before it requests anything else, and follows RFC 9309. To keep it away from your Agent Card paths, add this to your robots.txt:

User-agent: EmissarBot
Disallow: /.well-known/

Use Disallow: / to keep it off the whole site. If your robots.txt returns a server error, EmissarBot treats the whole site as off limits for that crawl.

You can also send your domain through the contact form. We add it to our opt-out list within seven days, and after that EmissarBot sends no requests to it, its www. form or its subdomains at all, including robots.txt.

How to check that a request came from EmissarBot

Anyone can copy a user agent. So EmissarBot signs every request it sends with a key only Emissar holds, and publishes the matching public key. It uses Web Bot Auth: HTTP Message Signatures (RFC 9421) as the IETF working group draft draft-ietf-webbotauth-httpsig-protocol-00 describes.

Signature-Agent header
sig1="https://emissar.ai"
Key directory
https://emissar.ai/.well-known/http-message-signatures-directory, served as application/http-message-signatures-directory+json and itself signed with the key it lists
Key
Ed25519. The keyid in each signature is the key's JWK SHA-256 thumbprint (RFC 7638): L-Ufc9Sso-j3ZVAcLxizDyVlvmM9KGFNgN1uvFxK1ns
Signature tag
web-bot-auth

Each signature covers the request method, the host (@authority), the path, the User-Agent header and the Signature-Agent value, and expires 5 minutes after it is made. Emissar's free tools, which fetch a site only when someone asks them to check it, identify themselves as EmissarTools/0.1 (+https://emissar.ai/tools) and don't sign: anyone can point a public tool at any site, so a signature from one would prove nothing about who asked. A valid signature from Emissar's key always means EmissarBot.

To check a request, fetch the key directory, take the key whose thumbprint equals the signature's keyid, and verify the signature as RFC 9421 section 3.2 describes. A valid signature shows that the request was signed with Emissar's key within the last 5 minutes.

Checking is optional. EmissarBot sends the same requests, follows robots.txt and honors opt-outs whether or not you verify it. Some verifiers built on earlier individual drafts expect Signature-Agent as a plain string rather than the dictionary form the working group draft requires, and may not accept these signatures yet.

Sources