The open standard for agent readiness
Agents are a new class of user. Many protocols now serve them (MCP, WebMCP, A2A, x402, llms.txt, Web Bot Auth, and more), but there is no shared definition of what it means to be ready for them. AgentReady is an open specification of what a product should implement to be usable by AI agents, from discovery to completion. It does not promote a protocol or a vendor. The practices below are based on measured agent behavior, and are revised as that behavior changes.
ora scanner
6%
of scanned sites grade B or above
each square is 1% of ~122,000 sites
D or F · 60%/C · 34%/B+ · 6%
§ essentials
The short version
Two changes broke an agent: hiding the answer behind JavaScript, and blocking access. Everything else is find, then read, then act. Find and read are the essentials every site needs. Act applies when the product should complete a task, not only be readable.
§ the guide
Make your site readable by AI agents
Together with the ora.ai research lab we traced 1,033 real agent runs and ran a controlled experiment. The sections below follow the order agents encounter a site: find, then read, then act. Each practice carries its stable identifier from the spec source (AR-FIND-01 and so on) and its strength: MUST for the changes that broke an agent in the studies, SHOULD for practices that measurably shaped where agents went, MAY for emerging conventions.
how we knowSTUDIESis ora.ai research: the traced runs and experiment, plus the reports linked from each evidence block; the dataset is public. ORA SCANNERis live reachability across every scored site. Untagged statements are recommendations.
§ find
Be reachable and citable
Agents can cite only pages they can discover and fetch. They may receive or recall a URL, follow a link, or find a page through a crawl-built search index. Pages blocked from crawlers do not appear in those indexes at all. Blocking can also stop an agent from fetching a page while answering a user. Publish the answer on one canonical, current first-party docs page, and write the crawl policy as one rule for all agents instead of an allowlist of named crawlers.
STUDIES When an agent cannot read your site, it answers anyway, from web search and other people’s pages. Across 1,056 business sites, those answers were 3.7× as likely to contain none of the facts the user asked for, and agent-ready businesses were clearly recommended 1.9× as often.
answers built without the site
backs:“When an agent could not read the site, it answered anyway, from web search and other people’s pages. Those answers were 3.7× as likely to contain none of the facts the user asked for.”
37,927 agent journeys across 1,056 business sites. Each agent-ready site was paired with a not-agent-ready one in the same industry, matched on brand fame and citations elsewhere. What the finished answer was built from:
| agent-ready | not agent-ready | |
|---|---|---|
| The site’s own pages | 78% | 58% |
| Web search | 12% | 25% |
| Training knowledge | 7% | 10% |
| Answers that clearly recommend the business | 20% | 11% |
Graded fact by fact against each business’s own site (2,499 answers, 131 businesses):
| from the site | from elsewhere | |
|---|---|---|
| Answers with none of the facts asked for | 7% | 25% |
| Pricing answers that were accurate | 60% | 37% |
In about 99% of journeys that hit a dead end on the site, the agent answered anyway. Not-agent-ready sites blocked agents 2.1× as often, and runs took 23% more turns.
Ora research, AX is the New AEO (arXiv:2609.34951), September 2026. Four agent stacks on Claude and GPT models; the two GPT stacks searched the web through Tavily. Analysis code and sample data at github.com/ora/research.
read the report →| practice | what it does | what the runs show | |||||||||||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Publish the answer as first-party docsAR-FIND-02 · SHOULD | Makes you citable: the page an agent grounds its answer on and names as its source. | STUDIES82% of answers traced back to a page the agent fetched, and that page was a docs page 47% of the time, more than any other page type. | |||||||||||||||||||||
answer provenance backs:“82% of answers traced back to a page the agent fetched, and that page was a docs page 47% of the time, more than any other page type.” Across 1,033 runs, where did the agent’s final answer come from?
Of the 844 grounded answers, the page types the wording traced to (top 3):
Docs lead every other type; the full 14-way split is in traces.csv. Reproduce over traces.csv: grounded = answer_grounded == 1; page type = answer_ground_kind. browse the dataset → | |||||||||||||||||||||||
| Allow agents in robots.txt and bot protectionAR-FIND-01 · MUST | Declares which crawlers may fetch which paths, and lets verified agents through the bot manager. This is the gate to the search indexes. | ORA SCANNERIn the April 2026 report (~50,000 scanned sites), 88% shipped a robots.txt and 80% stayed reachable to every major AI crawler. About 1 in 10 blocked them entirely. | |||||||||||||||||||||
| List your pages in a sitemap.xmlAR-FIND-03 · SHOULD | Maps every page you publish for indexing. | STUDIESFetched directly in 4% of runs, one of the files real agents touch least. It is still standard for crawlers and search engines. | |||||||||||||||||||||
sitemap reach backs:“Fetched directly in 4% of runs.”
Reproduce over traces.csv: sitemap_reached == 1. browse the dataset → | |||||||||||||||||||||||
| Publish an AI Catalog and link itAR-FIND-04 · MAY | One manifest at /.well-known/ai-catalog.json listing every agent-facing resource you ship, from MCP server card to llms.txt. | Not measured in the studies. Link it from the homepage head, since 86–100% of discovery-file visits arrive through links. ARD registries such as Ora Directory index published catalogs so agents can search across sites. | |||||||||||||||||||||
# Apply the default policy to current and future agents.
User-agent: *
# Block low-value public paths only. Do not list secret paths.
Disallow: /search/
Sitemap: https://example.com/sitemap.xmlBot protection is the second gate, and robots.txt does not control it. A WAF or bot manager that answers agents with a 403 or a challenge page blocks them however permissive the file is. Let verified agents through by the providers' published IP ranges or Web Bot Auth signatures, and throttle with a 429 instead of a challenge. In the study above, sites that were not agent-ready blocked agents 2.1× as often.
An AI Catalog is the one place an agent can learn what you ship for it: a JSON manifest at /.well-known/ai-catalog.json whose entries point to your MCP server card, OpenAPI, GraphQL schema, A2A agent card, docs, llms.txt, and skills, each typed by media type. Agentic Resource Discovery (ARD) is the registry layer on top: a registry such as Ora Directory indexes published catalogs and answers search across them. Publish the catalog, then link it from the homepage head, since files nothing points to are files nothing fetches.
<!-- In the homepage head, and as an HTTP Link header from the CDN. -->
<link rel="ai-catalog" href="/.well-known/ai-catalog.json">crawler jobs and how to test
The wildcard still leaves you one deliberate choice. The model providers run a separate crawler for each job:
- Training crawlers (GPTBot, ClaudeBot)
- Search-index crawlers (OAI-SearchBot, Claude-SearchBot)
- User-initiated fetchers (ChatGPT-User, Claude-User)
Each obeys its own robots.txt rule, so you can block a training crawler and opt out of training without dropping out of search indexes or agent answers. Two catches: a more specific group such as User-agent: GPTBot overrides your * group, so audit existing rules; and user-initiated fetchers may not consult robots.txt at all.
Agents also find pages through agent search APIs such as the Brave Search API, Exa, Parallel, and Tavily, which run their own crawlers and indexes. The wildcard rule covers them; an allowlist of named crawlers does not.
Then fetch a page with an agent user-agent string to inspect the response:
curl -iL -A 'Claude-User/1.0' https://example.com/your-pageThe request should end with a 200 response; intermediate 301 or 308 redirects are acceptable. The returned HTML should contain the answer. A 403, empty app shell, or JavaScript-only answer prevents fetch-only agents from reading the page.
The curl does not prove much either way: bot blockers weigh where a request comes from more than its user-agent string, so your laptop may pass where the real agent is blocked, or the reverse. The dependable test is your server logs. OpenAI and Anthropic publish their crawlers’ IP ranges (separate per-bot lists from OpenAI, one combined list from Anthropic). Find those requests and confirm they return 200.
§ read
Put the answer where agents read
After reaching a page, an agent has to extract the answer from the returned content. In the studies they used the same homepage and docs as human visitors. Agent-specific files help only if those pages point to them. If the site is client-rendered, the answer has to be in the initial HTML. That is the prerequisite. Content negotiation, .md endpoints, and agent detection did not change whether agents succeeded.
what agents reach · studies
- 83%docs pages
- 69%homepage
— then a cliff —
- 23%.well-known/*
- 21%openapi.json, when shipped
- 4%sitemap.xml
| practice | what it does | what the runs show | |||||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Homepage in raw HTMLAR-READ-03 · SHOULD | The front door, with real links to docs, product, and pricing. | STUDIESReached in 69% of runs, and it was the agent’s first step in 92% of them; in 59% of the runs that reached it, the next hop was a docs page. | |||||||||||||||
homepage reach and first hop backs:“Reached in 69% of runs, and it was the agent’s first step in 92% of them. In 59% of the runs that reached it, the next hop was a docs page.” The homepage funnel across 1,033 runs:
Reproduce over traces.csv: homepage_reached, homepage_first_turn, next_hop_docs, each rated against homepage_reached == 1. browse the dataset → | |||||||||||||||||
| Docs pages that answerAR-READ-04 · SHOULD | The pages agents actually read, where most tasks get resolved. Link setup and quickstart pages from the homepage. | STUDIESReached in 83% of runs and fetched more than any other page type, about 3.4 docs pages per run. | |||||||||||||||
docs reach and depth backs:“Reached in 83% of runs and fetched more than any other page type, about 3.4 docs pages per run.”
Reproduce over traces.csv: docs_reached == 1; mean of docs_pages_fetched over those runs. browse the dataset → | |||||||||||||||||
| Pricing on a first-party pageAR-READ-10 · SHOULD | Plans, free tier, and limits as text in the HTML, not an image, a calculator, or a sales form. | STUDIESBuilt from the business’s own pages, 60% of pricing answers were accurate, against 37% from elsewhere, the widest gap of any question type. | |||||||||||||||
accuracy by question type backs:“Built from the business’s own pages, 60% of pricing answers were accurate, against 37% from elsewhere, the widest gap of any question type.” Answers graded fact by fact against each business’s own site, by where the evidence came from:
Answers that punted to “contact sales for a quote” were more common when the agent could not read the site (22% against 15%). Setup is a findability problem: two-thirds of setup facts never surfaced, and sites where agents could ground setup answers scored 31% against 18%. Ora research, AX is the New AEO (arXiv:2609.34951), September 2026. 2,499 graded answers across 131 businesses. Analysis code and sample data at github.com/ora/research. read the report → | |||||||||||||||||
| Fetchable without JavaScriptAR-READ-01 · MUST | The answer is in the initial HTML, server-rendered or prerendered, rather than injected by a single-page app after load. | STUDIESA JavaScript-hidden answer was one of only two changes that broke an agent. The other was blocking it outright. | |||||||||||||||
fetchability: what broke an agent backs:“A JavaScript-hidden answer was one of only two changes that broke an agent. The other was blocking it outright.” One docs site was served under 19 configurations and probed across 5 topics by two clients, a plain non-JavaScript fetch and a JavaScript-executing one (190 probes). Only three configurations failed, and they reduce to two kinds of change:
A JavaScript-hidden answer fails only the fetch-only client; blocking (bot-block and hostile) fails both. Reproduce over fetchability.csv: success grouped by variant and client. browse the dataset → | |||||||||||||||||
| Fenced code, language-taggedAR-READ-05 · SHOULD | Turns snippets into examples an agent can lift and use. | STUDIESWhen an agent’s answer included code, 61% of the time it contained lines taken verbatim from a page it fetched on the site. | |||||||||||||||
code lift backs:“When an agent’s answer included code, 61% of the time it contained lines taken verbatim from a page it fetched on the site.” Among the 119 runs whose answer contained code:
Reproduce over traces.csv: among rows where answer_code_matched_fetch is set, share == 1. browse the dataset → | |||||||||||||||||
STUDIES Agents typically opened these resources after visiting the homepage. Depending on the file type, 86–100% of visits came through links rather than guessed paths. Link each resource from content agents already visited, such as the footer, page metadata like <link rel="alternate"> in the head, or a resources section.
discovery-file linkage
backs:“Depending on the file type, 86–100% of visits came through links rather than guessed paths.”
For each discovery file, the share of fetches that arrived through a link rather than a guessed path:
| file | via link | fetches |
|---|---|---|
| AGENTS.md | 100% | 38 |
| openapi.json | 97% | 122 |
| .well-known/* | 93% | 383 |
| llms.txt | 86% | 416 |
Reproduce over traces.csv: per file, {file}_linked / {file}_fetches.
browse the dataset →| file | what it does | what the runs show | ||||||||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| llms.txtAR-READ-07 · SHOULD | A curated, described index of your key pages. | STUDIESReached via links rather than guesses: 86% of its fetches came from a page that pointed to it. One in three agents that read it went on to fetch a page it lists, and 36% drew their final answer from its content. | ||||||||||||||||||
llms.txt behavior backs:“86% of its fetches came from a page that pointed to it. One in three agents that read it went on to fetch a page it lists, and 36% drew their final answer from its content.” How agents reached llms.txt (416 fetches):
What the 329 runs that read llms.txt did next:
Reproduce over traces.csv: via-link = llms_txt_linked / llms_txt_fetches; steering = llms_followed_listing over llms_txt_reached == 1; answer-from-llms = answer_ground_kind == 'llms_txt' over llms_txt_reached == 1. browse the dataset → | ||||||||||||||||||||
| .well-known/*AR-READ-06 · SHOULD | Protocol and metadata discovery. | STUDIESReached in about 23% of runs across the sites tested. | ||||||||||||||||||
.well-known reach backs:“Reached in about 23% of runs across the sites tested.”
Reproduce over traces.csv: well_known_reached == 1. browse the dataset → | ||||||||||||||||||||
| openapi.jsonAR-ACT-02 · SHOULD | Machine-readable API contract. | STUDIESOn the sites that ship one, used in 21% of runs. | ||||||||||||||||||
openapi reach backs:“On the sites that ship one, used in 21% of runs.” Restricted to the 7 sites that ship openapi.json (541 runs):
Reproduce over traces.csv: openapi_reached == 1 over site_ships_openapi == 1. browse the dataset → | ||||||||||||||||||||
| JSON-LDAR-READ-08 · SHOULD | Machine-readable facts and routes in the page. | Markdown converters may omit script blocks, so repeat critical facts in visible text. | ||||||||||||||||||
Inject JSON-LD at strategic places, starting with the homepage, so an agent fetching the raw HTML gets your key facts and routes in one machine-readable block.
AR-READ-02 · MUSTReturn 404 for a missing path or a valid redirect to a replacement. Do not return 200 for a missing page, even if the body suggests alternatives; agents cannot distinguish that response from a live page. In one observed case, a single-page app returned 200 for every path, making dead URLs appear valid.
The same applies when you throttle: answer with a 429 and a Retry-After header (RFC 6585) rather than a silent challenge page. Agents and crawlers treat 429 as a back-off signal and will return.
llms.txt shape and markdown mirrors
When shipping an llms.txt, make it a lean, described index:
# Acme
> Acme ships a payments API for developers.
## Docs
- [Quickstart](https://example.com/docs/quickstart): first charge in one request
- [Authentication](https://example.com/docs/auth): API keys, scopes, rotation
## API
- [OpenAPI](https://example.com/openapi.json): full request and response contractAR-READ-09 · MAYSTUDIES Agents requested markdown on about 65% of web fetches. When they specified a format, they chose markdown 96% of the time. Format did not determine task success in the studies, so ensure the HTML works before adding a .md mirror.
markdown requests
backs:“Agents requested markdown on about 65% of web fetches. When they specified a format, they chose markdown 96% of the time.”
| calls | share | |
|---|---|---|
| Web fetches requesting markdown | 2,259 | 65% of 3,470 |
| Of fetches that named any format, markdown | 2,259 | 96% of 2,361 |
Reproduce over traces.csv: markdown_calls / web_fetch_calls; markdown_calls / format_set_calls.
browse the dataset →<!-- A mirror nothing points to is a file nothing fetches. -->
<link rel="alternate" type="text/markdown" href="/docs/quickstart.md">The same relation can also be sent as an HTTP Link: header, so a CDN can advertise the mirror without touching page heads.
§ act
Let agents act, not just read
Reading documentation does not let an agent complete a task. To act, it needs an API, MCP server, WebMCP tools, SDK, or CLI, plus authentication it can finish, and, if the product charges, a way to pay. The last step happens only if the first two did.
the action gap · ora scanner · april 2026 report
80%
reachable to every major AI crawler
3%
ship a spec-grade MCP server
| practice | what it does | best practice | |||||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Auth an agent can get throughAR-ACT-01 · SHOULD | The first gate of acting. | Self-serve key generation, documented scopes. For user-delegated access, OAuth 2.0 with discoverable metadata under /.well-known (RFC 8414, RFC 9728). | |||||||||||||||
| Public API with a machine-readable contractAR-ACT-02 · SHOULD | The acting surface itself. | OpenAPI with operation IDs, typed responses and errors, /v1/ versioning, documented rate limits. | |||||||||||||||
| MCP serverAR-ACT-03 · SHOULD | Standard tool protocol for docs search and actions. | Serve over Streamable HTTP at /mcp. Keep the docs MCP public. Gate the product MCP with OAuth (protected-resource metadata at /.well-known/oauth-protected-resource per RFC 9728, with PKCE) and list it in the MCP Registry with a server.json. Server cards for pre-connection discovery (AR-ACT-04, MAY) are an emerging convention (SEP-2127, a working-group draft that major hosts have begun to support): ship one if your target hosts read it, but expect the exact path to keep moving. | |||||||||||||||
| WebMCPAR-ACT-07 · MAY | The same tool contract, registered on the page itself. | If the product has a web UI an agent should operate, register tools on document.modelContext. Feature-detect; the API is a W3C Community Group draft. Complementary to a server MCP: these tools run in the live page with the signed-in session. Only ship tools that map to actions the product actually offers. | |||||||||||||||
| MCP AppsAR-ACT-05 · MAY | Your UI, rendered inside the agent host. | If your product has a UI, ship a View (a ui:// resource) so the agent can hand a working surface back to the user. | |||||||||||||||
| NLWeb ask endpointAR-ACT-09 · MAY | A lighter alternative to a full tool surface for content-heavy sites. | Serve /ask, which takes a natural-language question and returns schema.org-shaped results, and expose the same ask tool over MCP. Suits publishers, catalogs, and docs whose value is content rather than operations. | |||||||||||||||
| SDKs and a CLIAR-ACT-06 · SHOULD | The forms agents act in: code and shell. | Publish SDKs on npm and PyPI and a CLI; show install and first call in the docs. | |||||||||||||||
| Let agents payAR-ACT-08 · MAY | The last gate of acting. Applies only if the product charges. | Per-call pricing: x402 or MPP, so an agent can settle a 402 and retry without an account. Checkout: UCP or ACP, so a shopping agent can complete a purchase. On Stripe, enable Link on checkout and accept Shared Payment Tokens, so an agent holding a Link Agent Wallet pays with a one-time credential the user approved, without filling a card form. Do not put a CAPTCHA wall in front of a verified agent.STUDIESPaying through Link, agents got 92.3% of test checkouts right; 18 of 26 agents got at least 90%. | |||||||||||||||
link checkout backs:“Paying through Link, agents got 92.3% of test checkouts right; 18 of 26 agents got at least 90%.” 26 agents (5 agent apps × 6 models), the same 33 checkout tasks for every setup. Only failures the agents caused are counted:
The most common Link miss, 27 of the 61 failed runs: the agent asked the person to approve, then stopped. Ora research, October 2026. Stripe test mode, fictional stores, no real money. Link is @stripe/link-cli 0.26.0 against a stand-in wallet that approved every request within seconds. ora research → | |||||||||||||||||
§ check your site
Check your site
Three public rankers score the site itself, each on a different surface, and one report shows how you rank when agents search your market. None of them is the spec. The rankers have a free browser score; the report runs on a free Tavily API key.
essentials
is-agentic.com
Vercel × Ora, for sites whose job is content. Essentials carry the score; capability checks sit in a recommended tier.
full ranker
ora.ai
The population ranker for products agents transact with: 100+ checks, per-layer breakdown, and a leaderboard.
docs
mintlify.com/score
Mintlify's Agent Score for documentation sites. 29 checks on the Agent-Friendly Documentation Spec.
visibility
avr.tavily.com
Tavily's report on who agents find when they search your market.
CLI
npx is-agentic example.comnpx ax audit https://example.comnpx mint score docs.example.comMethodology: is-agentic.com/docs, ora.ai/docs, and Mintlify Agent Score. Scanner checks and weights may change independently of this spec.
§ references
- What agents actually reach: the field report behind the trace numbers in this guide, drawing on part of the data used for this spec.
- AX is the New AEO: the study of 1,056 business sites behind the find-section numbers, with web search through Tavily.
- The state of agent readiness: the scanner report behind the reachability figure.
- WebMCP: the W3C Community Group draft for in-page MCP tools on document.modelContext.
- is-agentic.com: the Vercel × Ora scanner for site essentials, powered by the same engine as the Ora ranker.
- Mintlify Agent Score: the docs ranker: 29 checks on the Agent-Friendly Documentation Spec.
- ora research: the ongoing research program behind the standard.
- The agent readiness dataset: the runs and probes behind the trace and experiment [STUDIES] numbers, with a per-claim breakdown to reproduce them.
§ changelog
- v1.0 · August 2026Restructured into find / read / act with AR-FIND, AR-READ, and AR-ACT identifiers, strengths assigned from the studies, and every figure tagged by source. The earlier identifiers and /spec.json are retired.
- v1.0.0 · April 2026First published version, announced March 20, 2026 in Introducing AgentReady: five topic sections, AR-DISC / AR-CONT / AR-CAPA / AR-IDEN / AR-COMM identifiers, and a /spec.json artifact.
Full history, including every retired identifier, in the changelog.
contribute
AgentReady is developed in the open. The spec source, the dataset behind the trace and experiment STUDIES numbers, and this site live on GitHub, where discussion and proposals happen. Recommendations are revised as new agent behavior is measured.
MIT · revised as protocols emerge