Make any website agent-ready
This guide lists everything mocop looks for when it crawls a site, what each item is, and exactly how a site owner turns it on. Part 1 is a short path to follow in order (the first three steps take under an hour and cover most of the value). Part 2 is the complete reference of every location and signal mocop checks, grouped by purpose. Nothing is required: publish what fits your site and mocop shows what it found, how far it was verified, and where it came from.
Part 1: Step by step
1. Let agents in: robots.txt and a sitemap
Agents can only use what they are allowed to fetch. Publish /robots.txt, list a sitemap, and do not block the AI crawlers you want to be found by (GPTBot, ClaudeBot, PerplexityBot, Google-Extended, CCBot).
User-agent: *
Allow: /
Content-Usage: train-ai=n, search=y
Sitemap: https://example.com/sitemap.xml
Content-Usage is the IETF AI preferences vocabulary: say whether your content may be used for training (train-ai) and for search or grounding (search). Also publish /sitemap.xml (or a sitemap index) with every public page and a correct <lastmod>.
2. Say what you are: llms.txt
/llms.txt is a short markdown index written for language models: what the site is, and links to the pages and machine interfaces that matter. Optionally add /llms-full.txt with the long-form content.
# Example
> Example is a payments API for marketplaces.
## Interfaces
- [OpenAPI](https://example.com/openapi.json): REST API
- MCP server: https://example.com/mcp
## Docs
- [Quickstart](https://example.com/docs/quickstart.md)
3. Describe your API: OpenAPI
If you have an HTTP API, publish an OpenAPI 3.x document at /openapi.json (or link it from your docs and from llms.txt). Make it useful to a machine, not only to a human:
- A unique, stable
operationIdfor every operation. - A one-sentence
descriptionfor every operation and parameter. - Typed request bodies and response schemas for success and for errors (use RFC 9457 problem details).
- Examples, and a security scheme that says how to get credentials.
4. Offer tools: a remote MCP server
An MCP server lets agents call your capabilities as tools. Serve it over Streamable HTTP (for example at /mcp), keep tool names and descriptions precise, mark read-only tools with readOnlyHint, and publish a server card so it can be discovered without guessing:
/.well-known/mcp/server-card.json
{
"name": "example",
"description": "What the server does, in one sentence.",
"transport": { "type": "streamable-http", "url": "https://example.com/mcp" },
"authentication": { "required": false },
"documentation": "https://example.com/docs"
}
5. Be addressable by other agents: A2A agent card
If you expose an agent, publish an A2A Agent Card at /.well-known/agent-card.json with a name, description, version, supportedInterfaces (URL, protocol binding and version), capabilities, input and output modes, and a list of skills with examples.
6. Make everything discoverable
Agents should not have to guess where your interfaces are. Use these signals:
- **API catalog** (RFC 9727):
/.well-known/api-catalogasapplication/linkset+json, pointing at your OpenAPI (service-desc) and docs (service-doc). - **Link headers** on your home page:
Link: </openapi.json>; rel="service-desc",</llms.txt>; rel="describedby",</.well-known/api-catalog>; rel="api-catalog". - **Agent skills index**:
/.well-known/agent-skills/index.jsonlistingSKILL.mdfiles (name, description, URL, sha256 digest) that teach an agent how to do a task on your site. - **AGENTS.md** at the site root: endpoints, auth, rate limits and rules for agents, in plain markdown.
- **agents.json / ai-catalog.json** under
/.well-known/for catalogs of your agents and tools.
7. Serve content agents can read
Many AI crawlers do not run JavaScript. If your pages are rendered only in the browser, they look empty.
- Render the main content on the server (SSR or static generation) with real headings and text.
- Give every page a unique
<title>, meta description and canonical URL. - Add JSON-LD structured data (
Organization,WebAPI,SoftwareApplication,Product,FAQPage,BreadcrumbList) with stable@idvalues. - Use semantic HTML landmarks (
header,nav,main,footer) and descriptive link text. - Return markdown when a client sends
Accept: text/markdown, withVary: Accept. - Publish real About, Contact and Privacy pages: they are trust signals for both people and agents.
8. Be safe to call
- HTTPS everywhere, with
Strict-Transport-Security, a Content-Security-Policy,X-Content-Type-Options: nosniffand a sensibleReferrer-Policy. - Publish
/.well-known/security.txtwith a contact. - Rate-limit and say so in
AGENTS.md. Return429withRetry-After. - Never mark an interface read-only unless it is. Require authentication for anything that changes state, and document how.
9. Ship a CLI and SDK
A command-line tool is the easiest interface for many agent harnesses: they already run shell commands, so no MCP client or OAuth setup is needed. Scanners such as isagentready.com test for it (an MCP server or a CLI satisfies "agent task access"; SDK and CLI documentation is a separate check).
- Publish the CLI on a package registry (npm, PyPI, Homebrew, crates.io) and give it a stable install command, for example
npm install -g @example/cli. - Document it on its own page: installation, authentication, and one first call (
example --help,example search "x" --json). Support--jsonoutput and non-interactive login so agents can use it. - Publish SDKs the same way and label each registry link as an SDK.
- Link both from
llms.txtunder a Tools heading:
## Tools
- [SDK](https://www.npmjs.com/package/@example/sdk): SDK package and usage examples
- [CLI](https://example.com/docs/cli): `npm install -g @example/cli`, then `example --help`
Also mention the install command in AGENTS.md and in your docs' Install section.
10. Optional extras
- **DNS-AID**: SVCB records at
_index._agents.example.com,_mcp._agents.example.comand_a2a._agents.example.comthat point at your agent hosts, so agents can find you from the domain alone. - **WebMCP**: add
toolnameandtooldescriptionto forms (andtoolparamdescriptionto inputs), or register tools withnavigator.modelContext, so agents driving a browser can use your forms directly. - **NLWeb**: answer
GET /ask?query=...with{ "results": [ { "url", "name", "description" } ] }. - **Install path**: show a copy-paste command (
claude mcp add --transport http example https://example.com/mcp) and, if you have one, link your SDK or CLI on its package registry.
11. Check your work
Use mocop's own tools to check yourself and to see what agents see. Running a check also raises your site's priority for indexing: a site mocop has not crawled yet is queued at elevated priority, and one already in the queue moves up.
npx @mayberrytech/mocop analyze example.com # score out of 10 with a fix list
pipx install mocop && mocop analyze example.com # same, from Python
curl "https://mocop.ai/api/v1/analyze?domain=example.com"
The same check is in the SDKs (new Mocop().analyze("example.com") from @mayberrytech/mocop-sdk, Mocop().analyze("example.com") from the mocop Python package) and as the analyze_site MCP tool. To add the mocop MCP server to your own agent: curl -fsSL https://mocop.ai/setup | sh.
1. Fetch each file yourself with curl -i and confirm status 200 and the right Content-Type. A single-page app that answers 200 with its HTML shell for every unknown path is a common trap. 2. Run an agent-readiness scanner such as isagentready.com and fix the failing checks, highest weight first. 3. Search for yourself on mocop. mocop crawls the standard locations above (/openapi.json, /llms.txt, /.well-known/*, /mcp, sitemaps and Link headers) and shows what it found, how far each item was verified, and where it came from. If something you published is missing, the discovery panel on your organization page shows what was found and what was not.
Quick checklist
Step · File or signal · Why
Allow crawlers · /robots.txt, /sitemap.xml · Agents can fetch you
Describe the site · /llms.txt · Models get a map
Describe the API · /openapi.json · Agents can call it correctly
Offer tools · /mcp, /.well-known/mcp/server-card.json · Tools without glue code
Be an agent · /.well-known/agent-card.json · Agent-to-agent calls
Command line · CLI docs, install command, llms.txt Tools section · Works in any shell-based harness
Be discoverable · api-catalog, Link headers, skills index, AGENTS.md · No guessing
Readable pages · SSR, JSON-LD, markdown · Works without JavaScript
Safe · HTTPS, security headers, security.txt · Trust
Questions or corrections: support@mocop.ai.
Part 2: Everything mocop looks for
Every item below is fetched from your own domain (or a named subdomain) over HTTPS, using the exact path shown. "Status" says how settled the format is: **standard** (published specification), **convention** (widely used, no formal spec), **emerging** or **draft** (young proposals that may change). mocop shows what it found; it never treats declared metadata as verified.
APIs
OpenAPI / Swagger description
What it is: a machine-readable contract for an HTTP API: paths, parameters, schemas, auth. mocop turns each operation into a searchable capability. Where mocop looks: /openapi.json, /openapi.yaml, /swagger.json, /api/openapi.json, /v1/openapi.json, /v3/api-docs, /v2/api-docs, /api-docs, /api-json (NestJS), /swagger/v1/swagger.json (ASP.NET), /docs/openapi.json, /.well-known/openapi.json, and the same files on api., developers., developer. and docs. subdomains. How to enable: most frameworks generate it. FastAPI serves /openapi.json by default; Spring (springdoc) serves /v3/api-docs; NestJS and ASP.NET have Swagger modules. Otherwise export the file and host it at one of the paths above. Add operationId, descriptions and error schemas (see step 3).
API catalog
What it is: RFC 9727, a linkset that lists all of your APIs and where their specs and docs are. Status: standard. Where: /.well-known/api-catalog, served as application/linkset+json. How to enable: publish JSON like {"linkset":[{"anchor":"https://example.com/api","service-desc":[{"href":"https://example.com/openapi.json","type":"application/json"}],"service-doc":[{"href":"https://example.com/docs"}]}]}.
APIs.json
What it is: an index of an organization's APIs (name, description, links to specs). Where: /apis.json. How: publish the JSON file listing each API with its humanURL and properties (OpenAPI links).
AsyncAPI
What it is: the OpenAPI equivalent for event-driven and message APIs (Kafka, MQTT, WebSocket). Where: /asyncapi.json, /asyncapi.yaml. How: generate it from your event definitions with the AsyncAPI tooling and host it.
GraphQL endpoint
What it is: a GraphQL API. mocop reads the schema through introspection and lists its queries, mutations and subscriptions. Where: /graphql, /api/graphql, /v1/graphql and api. subdomain. How: expose the endpoint over HTTPS and leave introspection enabled for the public schema (or publish an SDL file and link it from your docs).
SCIM
What it is: a standard user-provisioning API. Where: /scim/v2/ServiceProviderConfig. How: enable SCIM 2.0 in your identity layer. Status: standard.
AI plugin manifest (legacy)
What it is: the retired OpenAI plugin descriptor, still found on older sites. Where: /.well-known/ai-plugin.json. How: nothing new to do; prefer OpenAPI plus an MCP server.
MCP servers (tools for agents)
Remote MCP server
What it is: a Model Context Protocol server exposing your capabilities as tools, over Streamable HTTP. mocop connects, runs the handshake and lists the tools, which makes the entry protocol_verified. Where: /mcp, /api/mcp, /v1/mcp, /mcp/v1, /sse and /mcp/sse (legacy SSE), on the root domain and on mcp., api., docs. and developers. subdomains, plus GitBook docs at /~gitbook/mcp. How to enable: use an MCP SDK (TypeScript, Python, Java, Go, Rust) and serve it over Streamable HTTP. Give every tool a precise name and description, mark read-only tools with readOnlyHint, and say in your docs whether OAuth is required.
MCP server card
What it is: a JSON descriptor so agents learn about your server without connecting. Status: emerging. Where: /.well-known/mcp/server-card.json, /.well-known/mcp-server-card, /.well-known/mcp.json, /.well-known/mcp. How: publish {"name","description","transport":{"type":"streamable-http","url"},"authentication":{"required":false}}.
DNS advertisement for MCP
What it is: records in DNS that point agents at your MCP host. Status: emerging. Where: _mcp TXT records and _mcp._agents SVCB records. How: add an SVCB record such as _mcp._agents.example.com. 3600 IN SVCB 1 example.com. alpn=h2.
MCP apps (ui:// views)
What it is: a tool can point at an interactive view that the client renders, using an MCP resource with a ui:// URI and the MIME type text/html;profile=mcp-app. Status: emerging. Where mocop looks: the MCP server's resources/list and the _meta.ui.resourceUri on its tools. How to enable: declare the resources capability, serve the HTML with resources/read, and set _meta.ui.resourceUri on the tools that have a view. The view receives the tool result by message and renders it.
WebMCP (browser agents)
What it is: lets an agent that is driving a browser call your page's forms and tools directly. Status: emerging. Where mocop looks: toolname and tooldescription attributes on forms in the home page HTML, and navigator.modelContext registrations. How to enable: add toolname, tooldescription and toolparamdescription to forms, or register tools in script with navigator.modelContext.registerTool. Label every form input (<label for> or aria-label) so agents and screen readers can use it.
Agents
A2A agent card
What it is: the Agent2Agent protocol's self-description: name, skills, supported interfaces, auth. Status: standard. Where: /.well-known/agent-card.json, and /.well-known/agent.json (older path). How: publish the card with name, description, version, supportedInterfaces (URL, protocolBinding, protocolVersion), capabilities, defaultInputModes, defaultOutputModes and skills. Then actually serve the JSON-RPC endpoint the card points at.
agents.json
What it is: a description of API flows an agent can follow. Where: /.well-known/agents.json. How: list each agent endpoint with name, description, type and a link to its card.
AI catalog / Agentic Resource Discovery
What it is: one document listing everything agent-facing on your domain. Status: emerging. Where: /.well-known/ai-catalog.json, /.well-known/ard.json. How: publish {"specVersion":"1.0","entries":[{"identifier","name","mediaType","url"}]} with one entry per MCP server, OpenAPI, agent card or skill.
Agent Skills
What it is: SKILL.md files that teach an agent how to do a task on your site, listed in an index with a digest of each file. Status: emerging. Where: /.well-known/agent-skills/index.json (also /.well-known/skills/index.json, /.well-known/skills.json), and a plain /skill.md. How: write a SKILL.md with a name and description in front matter and clear numbered steps, then list it in the index with name, type: "skill-md", description, url and digest: "sha256:<hash>".
AGENTS.md
What it is: plain-markdown instructions for agents: endpoints, auth, rate limits, do and don't. Where: /AGENTS.md and /agents.md. How: write it and serve it as text/markdown.
Agent registration (ERC-8004)
What it is: an on-chain-linked agent identity file. Status: emerging. Where: /.well-known/agent-registration.json. How: follow the ERC-8004 registration schema.
DNS advertisement for agents
Where: _agent and _a2a TXT records, and _a2a._agents SVCB. How: same shape as the MCP record, pointing at your agent host.
Content and discovery
llms.txt and llms-full.txt
What it is: a markdown index for language models, and an optional file with the full documentation text. Where: /llms.txt, /llms-full.txt, /.well-known/llms.txt. How: write a title, a one-line summary and sections of links (see step 2).
Homepage signals
What it is: what your home page says about itself. mocop reads Link response headers (rel="service-desc", rel="api-catalog", rel="describedby"), JSON-LD, Accept: text/markdown support, OpenSearch and feed links, OpenGraph and meta description (used as the site description on your mocop page). How: add the Link headers at your web server or CDN, JSON-LD in the page <head>, a unique <title> and meta description, and return markdown for Accept: text/markdown.
Server-rendered pages and structured data
What it is: the content of your pages as a crawler that does not run JavaScript sees it. mocop and most AI crawlers read the raw HTML. What mocop checks: readable text in the HTML (a few hundred characters or more), a unique <title> and meta description, a canonical link, an <h1>, JSON-LD whose entities have stable @id values that reference each other (for example an Organization referenced as the provider of a WebAPI), semantic landmarks (header, nav, main, footer), labels on form inputs, and Accept: text/markdown support. How to enable: render on the server (SSR or static generation), add JSON-LD for your Organization, WebSite and each API or app with @id, and return markdown for Accept: text/markdown with Vary: Accept.
Link headers
What it is: HTTP Link headers on your home page that point agents at your machine-readable files (RFC 8288). Where mocop looks: the response headers of /: rel="service-desc" (OpenAPI), rel="api-catalog", rel="describedby" (llms.txt) and rel="ai-catalog"; the same ai-catalog link as <link rel="ai-catalog"> in the page head. How to enable: add them at your web server or CDN, for example Link: </openapi.json>; rel="service-desc", </.well-known/ai-catalog.json>; rel="ai-catalog".
NLWeb ask endpoint
What it is: a natural-language endpoint, GET /ask?query=..., that answers with ranked results as JSON (results with url, name, description, and schema.org hints). Status: emerging. How to enable: serve it and, ideally, expose the same thing as an ask tool on your MCP server so agents discover it there too.
About, Contact and Privacy pages
What they are: real, readable pages that tell people and agents who runs the site and how to reach them. mocop treats them as trust signals. How to enable: publish /about, /contact (an email address and what to expect) and /privacy with real text, and link them from your footer and from llms.txt.
Sitemap
Where: /sitemap.xml (and /sitemap_index.xml). How: generate it from your CMS or build; list the URL in robots.txt.
ai.txt and OpenSearch
/ai.txt is an emerging file for AI usage preferences. /opensearch.xml (OpenSearch 1.1) describes your site search so other tools can query it. Publish either if you have one.
Site icon
Where: /favicon.ico and the <link rel="icon"> tags on your home page. mocop shows it next to your organization. How: provide an SVG or PNG icon of at least 64 px.
Identity and trust
OAuth metadata
What it is: how an agent obtains a token. Protected resource metadata (RFC 9728) at /.well-known/oauth-protected-resource says which authorization server protects your API; authorization server metadata (RFC 8414) at /.well-known/oauth-authorization-server describes endpoints and supported flows; OpenID Connect discovery at /.well-known/openid-configuration is the same for OIDC. A client metadata document at /.well-known/oauth-client identifies an agent client by URL (draft). How: most identity providers serve these automatically. For your own API, publish the protected-resource document with resource and authorization_servers.
JWKS, DID and Web Bot Auth
/.well-known/jwks.json (RFC 7517) publishes your public signing keys. /.well-known/did.json is a did:web identity document. /.well-known/http-message-signatures-directory publishes the keys a bot uses to sign its requests (Web Bot Auth, draft). Publish what applies to your identity setup.
security.txt
Where: /.well-known/security.txt (RFC 9116). How: add Contact:, Expires: and optionally Policy: lines. It tells researchers, and agents, who to tell about a problem.
Security headers
What they are: response headers that tell clients how to treat your pages safely. What mocop checks: Strict-Transport-Security, Content-Security-Policy, X-Content-Type-Options: nosniff, and frame protection (X-Frame-Options or a CSP frame-ancestors), plus a stated rate limit for agents (in AGENTS.md or as RateLimit / Retry-After headers). How to enable: add the headers in your web server or CDN for every response.
Account endpoints
/.well-known/passkey-endpoints (passkey enrollment and management), /.well-known/change-password (where a password manager should send users), /.well-known/host-meta (RFC 6415), /.well-known/openid-federation (OpenID Federation) and /.well-known/nodeinfo (fediverse servers). Publish the ones your platform supports.
Commerce and payments
UCP, AP2 and x402
What they are: ways for agents to discover and pay for things. The Universal Commerce Protocol manifest is at /.well-known/ucp; the Agent Payments Protocol endpoint at /.well-known/ap2; and x402 paid-resource listings at /.well-known/x402.json or /.well-known/x402. All are emerging, and the AP2 path is not settled. How: follow each protocol's manifest format and list your purchasable or pay-per-call resources, prices and settlement details. Never advertise a price or action you do not honor.
Policy
robots.txt
What it is: RFC 9309 crawl rules. mocop reads which AI crawlers you allow or restrict and any content-usage preference. Where: /robots.txt. How: allow the crawlers you want to be found by, disallow private paths, state Content-Usage: train-ai=n, search=y (or your own choice), and list your sitemap. mocop identifies itself as mocop-crawler, obeys your rules and limits its request rate.
Text and data mining reservation
Where: /.well-known/tdmrep.json (W3C TDMRep). How: publish rules that reserve or license text and data mining by path.
Global Privacy Control
Where: /.well-known/gpc.json. How: publish {"gpc": true} if you honor the GPC signal.
Platform signals
Web app manifest
Where: /manifest.webmanifest. What it gives mocop: your app name, description and icons. How: generate it with your framework's PWA tooling.
Status page
Where: status. subdomain, /api/v2/status.json (Atlassian Statuspage format). How: run a status page with a machine-readable endpoint so agents can tell whether you are up.
App associations
/.well-known/assetlinks.json (Android Digital Asset Links), /.well-known/apple-app-site-association (Apple Universal Links) and /.well-known/web-app-origin-association link your site to its apps. Publish them if you ship apps.
Agent contracts
auth.md
What it is: a short markdown file that tells an agent how to authenticate: how to get a credential without manual approval, the scopes and what each allows, and an example request. Where mocop looks: /auth.md (and /.well-known/auth.md). How to enable: write the flow you support. For OAuth, name the authorization and protected-resource metadata, require PKCE S256, list scopes and token lifetime. For API keys, say where to create one and which scopes exist. Include an authenticated curl example, then link the file from llms.txt.
pricing.md
What it is: a markdown description of plans, prices, limits and billing period so an agent can budget before it calls you. Where mocop looks: /pricing.md. How to enable: a small table (plan, price, limit) is enough. If everything is free, say so and state the rate limit.
AGENTS.md
Covered under Agents above: the offer, endpoints, auth and rules for agents. Reference auth.md from it.
Developer tooling
CLI
What it is: a command-line tool that wraps your API. Many agent harnesses already run shell commands, so a CLI works without an MCP client or OAuth app. What mocop looks for: an install command in llms.txt, AGENTS.md or the documentation pages llms.txt links to (npm install -g, npx, brew install, pipx install, uvx, pip install, cargo install, go install, gem install, winget install, scoop install), a link labeled CLI, a first-call example (--help, --version, --json), and a separate CLI documentation page. How to enable: publish the CLI on a package registry, document it on its own page with the install command and one real first call, support --json output and non-interactive login, and link it from llms.txt under a Tools heading.
SDK and package registries
What it is: a client library for your API in the languages your users write. What mocop looks for: a registry link labeled as an SDK in llms.txt or the documentation pages it links to: npmjs.com/package/, pypi.org/project/, crates.io/crates/, pkg.go.dev/, rubygems.org/gems/, formulae.brew.sh/formula/, hex.pm/packages/. How to enable: publish the package and link it from llms.txt:
## Tools
- [SDK](https://www.npmjs.com/package/@example/sdk): SDK package and usage examples
- [CLI](https://example.com/docs/cli): `npm install -g @example/cli`, then `example --help`
Claim your domain on mocop
What it is: prove you own a domain to manage how it appears in mocop. Sign in, add the domain under My Account, then use any one method. Verified domains show a blue check in search and listings and are indexed straight away.
- **HTML file:** host
/.well-known/mocop-verify.htmlcontaining your verification code. - **Meta tag:** add
<meta name="mocop-verification" content="…">to your home page<head>. - **DNS TXT record:** add a TXT record
_mocop-verify.yourdomainwith the verification code. mocop detects where your DNS is hosted and links to the right dashboard. - **Well-known file:** serve
/.well-known/mocop-verify.txtcontaining the code.
Once verified you can hide the site from the index, edit its display name and description, re-scan on demand and see its readiness trend. Everything is also available from the CLI (mocop domains), the SDKs and the API.
How mocop treats what it finds
- **Provenance first.** Every entry shows the file or URL it came from and when it was last checked.
- **Verification levels.**
declaredmeans a file says so;observedmeans mocop fetched and parsed it;protocol_verifiedmeans mocop completed a live handshake, for example listing an MCP server's tools. - **Changes are tracked.** New and removed operations get first-seen and last-seen dates, and versions follow the version your API declares.
- **Nothing is invented.** Prices, auth and capabilities are shown only when a source asserts them.
Questions or corrections: support@mocop.ai.