Agent-readiness rubric
Every point the scanner can award, and the rule behind it. Nothing else counts toward the score.
By Emissar. Updated . Scan results name the rubric version they used.
Summary
| Criterion | Where the scanner looks | Points |
|---|---|---|
| A2A Agent Card | /.well-known/agent-card.json | 40 |
| security.txt | /.well-known/security.txt | 20 |
| llms.txt | /llms.txt | 15 |
| robots.txt | /robots.txt | 15 |
| Sitemap | The first Sitemap line in robots.txt, otherwise /sitemap.xml | 10 |
| Total | 100 |
Points are added, never weighted or rounded. A criterion's points are earned item by item, as listed below.
A2A Agent Card 40 points
The discovery path the A2A specification defines and IANA has registered. Other agents read it to find your agent's endpoint, skills and security requirements.
Checked at /.well-known/agent-card.json.
| Points | Earned when |
|---|---|
| 20 | The path returns HTTP 2xx with a JSON object. |
| 20 | The card has no failed checks under the Agent Card validator's rules (v1.0, or v0.3 for cards in the older shape). Warnings don't cost points. Signatures aren't checked by the scanner; run the validator for them. |
Sources: A2A v1.0 §8.2 Discovery Mechanisms; IANA Well-Known URIs registry (agent-card.json).
security.txt 20 points
Tells people, and agents acting for them, how to report a security problem. Agents that transact on a company's behalf need a way to reach its security team.
Checked at /.well-known/security.txt.
| Points | Earned when |
|---|---|
| 5 | The path returns HTTP 2xx. |
| 15 | It has at least one Contact field whose value is a URI (web URIs must start with https://), exactly one Expires field in RFC 3339 date-time format, and that date hasn't passed. |
Sources: RFC 9116 §2.5.3 Contact; RFC 9116 §2.5.5 Expires; RFC 9116 §3 Location of the security.txt file.
llms.txt 15 points
A community proposal, not a standard: a Markdown file at the site root that summarizes the site and links to the pages a language model should read.
Checked at /llms.txt.
| Points | Earned when |
|---|---|
| 10 | The path returns HTTP 2xx with text, not an HTML page. |
| 5 | The first line with content is an H1 title (# Name), the only section the proposal requires. |
Sources: The /llms.txt file (proposal).
robots.txt 15 points
Crawlers, AI crawlers included, read it before fetching. Without one (HTTP 4xx), crawlers may fetch anything; if it fails with a server error (5xx), crawlers must treat the whole site as disallowed.
Checked at /robots.txt.
| Points | Earned when |
|---|---|
| 15 | The path returns HTTP 2xx and parses as robots.txt with at least one user-agent group. |
Sources: RFC 9309 §2.2 Formal syntax (groups and rules); RFC 9309 §2.3.1 Access results (4xx and 5xx).
Sitemap 10 points
Lists the pages a site wants found, so crawlers and agents don't have to guess.
Checked at The first Sitemap line in robots.txt, otherwise /sitemap.xml.
| Points | Earned when |
|---|---|
| 5 | A sitemap is found: that URL returns HTTP 2xx. |
| 5 | It is in a format the sitemaps protocol accepts: XML with a urlset or sitemapindex root in the sitemaps.org namespace, a plain-text list of URLs, or an RSS or Atom feed. |
Sources: Sitemaps XML format (sitemaps.org protocol); RFC 9309 §2.2.4 Other records (Sitemap).
Reported, not scored
Legacy /.well-known/agent.json
A2A v0.2 used this path. Clients for v0.3 and v1.0 read /.well-known/agent-card.json instead, so an extra card here neither helps nor hurts. The scanner reports whether one exists and whether it passes the validator's rules, because an outdated copy can mislead older clients.
Rules for AI crawlers and agents in robots.txt
Reported, never scored. Blocking or allowing AI crawlers is the site owner's choice, and robots.txt doesn't govern agents calling your A2A endpoint. The scanner lists which of the tokens below your robots.txt names and whether each may fetch the site root.
MCP server discovery
Not checked. The current MCP specification (revision 2026-07-28) defines no well-known path for finding an MCP server. The well-known paths it uses, /.well-known/oauth-protected-resource (RFC 9728) and OAuth authorization server metadata (RFC 8414), only point clients to an authorization server. MCP Server Cards, a well-known file describing a server, are a draft proposal (SEP-2127) of the MCP Server Card working group. The scanner will check it if a path is ratified.
Sources: MCP specification, revision 2026-07-28: Authorization; MCP versioning (revision list); MCP Server Card working group (SEP-2127, draft); RFC 9728: OAuth 2.0 Protected Resource Metadata; RFC 8414: OAuth 2.0 Authorization Server Metadata.
AI crawler and agent tokens
The robots.txt tokens the scanner looks for, each as its vendor documents it. A token is reported when a group names it, together with whether it may fetch the site root. Checked against each vendor's page on September 26, 2026. ByteDance's Bytespider isn't listed because ByteDance publishes no page documenting it.
| Token | Vendor | What the vendor says it does |
|---|---|---|
GPTBot | OpenAI | Crawls content that may be used to train models |
OAI-SearchBot | OpenAI | Search results in ChatGPT |
ChatGPT-User | OpenAI | Fetches pages when a user asks; OpenAI says robots.txt rules may not apply |
ClaudeBot | Anthropic | Crawls content that may be used to train models |
Claude-User | Anthropic | Fetches pages when a user asks |
Claude-SearchBot | Anthropic | Search result quality |
Google-Extended | Control token for Gemini training and grounding; no separate crawler | |
Google-CloudVertexBot | Crawls requested by site owners building Vertex AI agents | |
Applebot-Extended | Apple | Control token for use in Apple's AI training; no separate crawler |
PerplexityBot | Perplexity | Search results in Perplexity |
Perplexity-User | Perplexity | Fetches pages when a user asks; Perplexity says it generally ignores robots.txt |
CCBot | Common Crawl | Open web dataset, widely used for training |
meta-externalagent | Meta | Training AI models and indexing content |
meta-externalfetcher | Meta | Fetches pages when a user asks; may not follow robots.txt |
Amazonbot | Amazon | Improves Amazon products; may be used to train Amazon AI models |
DuckAssistBot | DuckDuckGo | Fetches pages for AI-assisted answers |
MistralAI-User | Mistral AI | Fetches pages when a user asks |
MistralAI-Index | Mistral AI | Indexing for Mistral search |
MistralAI-Training | Mistral AI | Crawls content to train models |
Changes
If the points change, this page and the scanner change together, the date above moves, and results name the rubric date they were scored with.
- : first version.