Methodology

How EmissarBot looks for public A2A Agent Cards, what it requests, how each result is classified, and what the numbers cannot tell you.

By Emissar team · Published September 26, 2026 · Updated September 28, 2026

Seed list

Each full crawl checks the domains in the latest Tranco list of the top one million sites at the time the crawl starts. Tranco is a research ranking that combines several popularity sources over 30 days, which makes it harder to manipulate than any single source. Each list has a permanent ID, so anyone can download the exact seed list.

Edition	Tranco list	List created	Ranking window	Sources combined
State of Agent-to-Agent: Q3 2026	L5PZ4	September 25, 2026	August 27, 2026 to September 25, 2026	crux, farsight, majestic, radar, umbrella; dowdall rule; pay-level domains

The sources are named as the Tranco API reports them for each list. Every domain keeps its Tranco rank, and results are reported in four rank ranges:

Range	Domains
1 to 1,000	Ranks 1 to 1,000
1,001 to 10,000	Ranks 1,001 to 10,000
10,001 to 100,000	Ranks 10,001 to 100,000
100,001 to 1,000,000	Ranks 100,001 to 1,000,000

What is requested, in order

For each domain, EmissarBot sends at most three GET requests over HTTPS to the domain exactly as the list gives it (https://example.com, never an added www.), one after another:

Step	Path	Accept header	Why
1	`/robots.txt`	`text/plain, /;q=0.1`	Always first. Decides whether the two card paths may be requested.
2	`/.well-known/agent-card.json`	`application/json`	The A2A v1.0 discovery location (IANA-registered well-known URI).
3	`/.well-known/agent.json`	`application/json`	The older location used before A2A 0.3. Requested only when step 2 did not return an Agent Card.

Every request sends the user agent EmissarBot/0.1 (+https://emissar.ai/bot) and is signed with Emissar's key (Web Bot Auth), so a site can check it came from EmissarBot; how to check a signature. Crawls before signing started, including the one behind the 2026 Q3 edition, sent unsigned requests; nothing else about the requests changed. No URL is requested more than once for a domain in a crawl, and there are no retries: if a redirect from step 2 already reached the step 3 path, step 3 is not sent again. Nothing else is requested: no pages, links, scripts or logins, and no call to any agent the card describes.

robots.txt

EmissarBot follows the Robots Exclusion Protocol, RFC 9309:

It uses the rules in groups for its product token, EmissarBot (matched case-insensitively as emissarbot). With no such group, it uses the * groups. With neither, everything is allowed.
The longest matching Allow or Disallow rule wins, with * and $ supported. On a tie, Allow wins. An empty Disallow disallows nothing.
A 2xx robots.txt is parsed, up to 500 KiB (the minimum RFC 9309 section 2.5 sets). A 4xx other than 429 means there are no restrictions (section 2.3.1.3).
A 5xx means the whole site is disallowed for this crawl, as section 2.3.1.4 requires. EmissarBot also treats a 429, and a robots.txt redirect it cannot follow (including one to another host), as a full disallow.
The same rules apply to every redirect hop of the card requests: a redirect to a path they disallow is not followed.
When robots.txt gets no response at all (timeout, DNS or connection failure), the domain is counted under that failure and the card paths are not requested.

Timeouts, redirects and size limits

Limit	Value
Time per request	5 seconds for the whole chain: every redirect hop and reading the body
Redirects	Up to 3 (301, 302, 303, 307, 308), only within the same site: the domain as listed, its www. form and its name without www. count as the same site. A redirect to any other host, including a subdomain, is not followed (outcome redirect-error, detail cross-host). Every hop must also be HTTPS on port 443, with no user name or password in the URL, a public hostname that is not on the opt-out list, a path and query that robots.txt allows, and a URL not yet requested for the domain in this crawl; otherwise the redirect is not followed
Body size	256 KiB for a card, 500 KiB for robots.txt. A larger card counts as not JSON
Concurrency	At most 60 requests in flight across all domains; each domain's requests are made one after another

Changed September 27, 2026, from the next crawl on. Earlier crawls followed redirects to other hosts, as long as each hop passed the HTTPS, port, credential and public-hostname checks, and did not check hops against robots.txt or the opt-out list. The published 2026 Q3 edition (crawl full-20260926-39db43) was made with that earlier behavior: at least 7 of its 274 cards were served by a different host than the listed domain after a redirect, and each is counted as the listed domain's card. Its numbers stay as published.

How each domain is classified

Every domain checked gets exactly one outcome. If robots.txt stops the check, that is the outcome. If step 2 returns a card, that is the outcome and step 3 is skipped. Otherwise the domain gets the most informative result of the two paths, in the order of this table (JSON before not JSON, not JSON before an HTTP error, and so on).

Outcome	Meaning
Agent Card, v1.0 shape `card-v1`	A JSON object with a supportedInterfaces array (the v1.0 field that lists the agent's endpoints).
Agent Card, v0.3 shape `card-v0.3`	A JSON object without supportedInterfaces that has a url string plus a protocolVersion or preferredTransport string.
JSON, not an Agent Card `json-not-card`	A 2xx response that parses as JSON but has neither shape, or is not an object.
Not JSON `not-json`	A 2xx response that does not parse as JSON (for example an HTML page), or a body larger than 256 KiB. The media type is recorded as the detail.
HTTP error status `http-error`	A final status outside 200 to 299. The status is recorded as the detail.
Redirect not followed `redirect-error`	A redirect to another host (detail cross-host), to a path robots.txt disallows, to a URL already requested for the domain, or to a URL that fails the rules below; more than three redirects; or an unusable Location header.
Timed out `timeout`	No complete response within 5 seconds.
Connection failed `network-error`	DNS, TLS or connection failure, or HTTP 530 (the host does not resolve or its origin is unreachable).
Blocked by robots.txt `robots-disallowed`	robots.txt rules disallow the path for EmissarBot, or robots.txt returned 429 or 5xx, or its redirect could not be followed (for example, a redirect to another host).
Opted out `opted-out`	The domain, its www. form or a parent domain is on Emissar's opt-out list (names compared in lowercase, without www. and without a trailing dot). No request is sent, not even for robots.txt.
Invalid domain name `invalid-domain`	The list entry is not a public hostname (IP literal, single label, reserved suffix such as .local or .test, or invalid characters). No request is sent.

A card counts whether it was served at the v1.0 path or the older one; the report shows both. The shape test is structural: it does not validate every field. To check one card against the specification, use the Agent Card validator.

What is recorded from a card

For a domain that served a card, the crawl stores the card and these facts: its shape, the path and final URL, the HTTP status, the number of skills, the capability flags, whether a non-empty signatures array is present, the security scheme types, the provider organization, and the Content-Type, Cache-Control and ETag response headers. For every other domain it stores only totals per rank range and outcome, never the domain.

Signed means a signatures array is present. The crawl does not verify signatures.
Security schemes are counted by type: API key, HTTP (with its scheme, such as Bearer), OAuth 2.0, OpenID Connect and mutual TLS. Types outside that list count as other.
Protocol bindings are read from the stored card: supportedInterfaces[].protocolBinding for the v1.0 shape (or that entry's transport, the v0.3 field name, when protocolBinding is missing), and preferredTransport plus additionalInterfaces[].transport for the v0.3 shape.
Provider organizations are reported as counts only. Reports never name organizations or domains; a directory of public cards is a separate project.

Known limits

Apex domains only. The Tranco lists used here hold registrable (pay-level) domains. An Agent Card published only on a subdomain, such as agents.example.com, is not found. The 2026 Q3 edition still counted such a card when the apex redirected to it; from the next crawl, that redirect is not followed either.
Only public cards at the well-known paths. Cards at other URLs, cards behind authentication, extended cards and cards listed only in registries or catalogs are not counted.
Soft 404s. Some sites answer unknown paths with their HTML home page and status 200. These count as not JSON, not as a missing page.
Blocks. Bot protection and CDNs may answer 403, 429 or a challenge page, and a robots.txt that returns 429 or 5xx blocks the whole site. A card behind such a block is not counted.
One snapshot. Each edition is one pass with no retries, so a site that was briefly down counts under its failure. Figures describe the crawl dates only.
A ranking of popular sites. Tranco ranks sites by popularity signals. Shares describe the top one million sites, not all companies or all agents.
Declared, not tested. Capabilities, bindings and security schemes are what the card declares. The crawl does not call the agent.

Publication rules

An edition is published only after its crawl has finished. Each edition states how many domains in the list were checked; if a batch of domains could not be processed, those domains are not counted and are not retried, so no site receives a second request in the same crawl. Each edition lists the SQL queries behind its numbers and offers an aggregate CSV with no per-domain rows. To have your card's record removed, or to keep EmissarBot away from your site, see EmissarBot.

Sources

A2A protocol specification: Agent Card, discovery, bindings, security schemes and signatures
RFC 9309: Robots Exclusion Protocol
Tranco: Le Pochat et al., "Tranco: A Research-Oriented Top Sites Ranking Hardened Against Manipulation," NDSS 2019
IANA Well-Known URIs registry