What is the default when an agent wants a page?

It fetches a web page. Typically HTML. There is no official markdown-only default and no official API-only default. Most sites can leave a note called robots.txt. If that note is unavailable, RFC 9309 says a crawler may access resources on the server. Reachable is not the same as appointed.

Is robots.txt a lock?

No. The standard says the rules are not a form of access authorisation. OpenAI says the rules may not apply to ChatGPT-User when a person asked. Anthropic lets a site block Claude-User through robots.txt and says its bots honour do-not-crawl signals. A note is not a lock.

Jeremy Howard proposed it on 3 September 2024. Version 2 is dated 10 August 2026. It is a map of pages an agent can use when a person has asked. It is not a lock and not an internet standard. In the HTTP Archive July 2025 sample, about two sites in a hundred had a valid file.

What is a public site MCP?

A socket the site named. A visitor's own agent can ask for public pages instead of guessing from the HTML. Authorisation is optional in the spec. This volume relies on four live examples. It is not a census.

Does a public page equal an agent corpus?

Not as a public standard. Two worked sites run that as a house rule, so a draft in a publishing system does not become an agent page. Publisher bodies are moving the other way: content used on terms, not merely because the page is public.

Who names the reader?

The site, if it cuts a door and says what may be read. Availability is not appointment. A reachable page is not a named reader.

The Appointed Door

A page is fetched.

That is where this starts. Not with a protocol. Not with a product. With a request for a URL.

If you run a site, this is already happening. If you use a chat tool that goes and looks at a page, you are already on one side of it.

The site already has a face for people. HTML. Navigation. A title. A paragraph you can sit with.

Agents have been coming through the wall. They take that page unless something stops them.

In this volume, I use scrape for a machine taking a page through the ordinary web surface rather than through a surface the site deliberately appointed for agents. That is a narrower use of the word than the industry uses. It is useful here because the distinction is the point.

You can leave the hole. Or you can cut a door and say what may be read.

The site can appoint a door.

The wall is the default

The default is still HTML.

An agent that wants a page fetches a web page. Typically HTML. There is no official markdown-only default. There is no official API-only default.

Most websites can leave a small text file at a known address. The file is called robots.txt. It is a note to crawlers about which pages the site would rather they leave alone.

An RFC is a numbered internet standard. Engineers treat it as a rule they agreed. RFC 9309 is the Robots Exclusion Protocol. It is a standard. It also says the rules in robots.txt are "not a form of access authorisation."

That sentence matters.

If the file is unavailable — for example, the server answers with a 404 — the crawler may access resources on the server under the Robots Exclusion Protocol.

If the file is unreachable because of a server or network error — a 5xx error, for example — the crawler must assume complete disallow.

If the file is there but no group matches this crawler, and there is no wildcard group, no rules apply.

So a missing robots.txt leaves no Robots Exclusion Protocol restriction in place. The crawler may proceed.

That is not the same thing as authorisation.

robots.txt is a note. It is not a lock.

The labs also draw different lines around that note. The useful distinction is not only the crawler name. It is what the crawler is doing.

A training crawl is a lab's machine walking the web to collect material that may contribute to model development. A user-triggered fetch is different. A person asked a question. The model goes to get a page.

OpenAI draws that line explicitly. ChatGPT-User is used for certain user actions. OpenAI says that because those actions are initiated by a user, robots.txt rules may not apply.

Anthropic draws the line differently. Claude-User is the bot used when a person asks Claude to retrieve a page. Anthropic says disabling Claude-User in robots.txt prevents that retrieval, and says its bots honour do-not-crawl signals.

So the same file does not have the same operational meaning across every agent provider.

Google separates common crawlers, special-case crawlers, and user-triggered fetchers. Search crawling is one thing. Other uses sit beside it.

Grounding is when a model is told to answer from named pages, not only from what it already remembers. Google-Extended is a special token a site can write in robots.txt. It is meant to control certain Gemini and Vertex uses without removing the page from ordinary Google Search. The official Google crawler page does not say that Google-Extended removes a page from AI Overviews.

Cloudflare is the edge, not a crawler. Many sites sit behind it. It is the porch in front of the server. Cloudflare says robots.txt compliance is voluntary. Its products can publish a managed file and can separately enforce bot policy. The note and the enforcement mechanism are not the same object.

That is the wall.

A page can be reachable. A crawler may be allowed under the robots protocol to fetch it. Neither fact, by itself, says the site appointed that route for an agent.

Figure 1. The wall is the default. An agent fetches a web page unless a note or an edge stops it. A missing robots.txt is not a lock, and not an appointment.

A map is not a lock

Some sites now leave a second small file. It is called llms.txt.

Jeremy Howard at Answer.AI proposed /llms.txt on 3 September 2024. The current proposal is v2, modified 10 August 2026.

It is a curated markdown map. Markdown is plain text a model can read without the menus and layout of a web page. An agent can use the file to find pages that are likely to matter while helping a person.

It is not robots.txt. It is not a crawl ban.

The official proposal describes /llms.txt as information to help agents use a website. Version 2 also recommends clean markdown versions of pages and standard link relations that point between the ordinary page, its markdown alternative, and the llms.txt file that describes the area.

There is no IETF RFC for it. It is a proposal and convention, not an internet standard.

The proposal now says thousands of sites publish one. That is useful evidence of adoption, but it is not a web-wide census.

The strongest measured primary I found remains the HTTP Archive July 2025 crawl, published in the Web Almanac in January 2026. In that sample, about two sites in a hundred had a valid file. The figures were 2.13 per cent of desktop pages, 2.1 per cent of mobile pages, and 324,184 valid files on mobile sites. Many came from systems and plugins that generate the file automatically.

Two worked sites in this volume serve one. thierry-gilgen-ict.ch/llms.txt is a heading, a blockquote, a line that names the public door, a Field Notes list, and optional contact. theoldhandofasia.ch/llms.txt is bilingual lists, a line that names the public door, and a note that public agent resources are read-only. Both also serve /agents.

A map is not a lock.

It tells an agent where the rooms are. It does not stop another machine taking HTML through the ordinary web surface. It does not create a licence. It does not appoint every possible reader.

There can be a third object.

Not a note. Not a map.

A door the site deliberately exposes.

Figure 2. Three objects, three jobs. robots.txt is a note. llms.txt is a map. A public MCP is a door the site named.

A door you can name

There is a common plug for this. People call it MCP, the Model Context Protocol.

Think of it as a socket. A client can ask a server for structured resources or tools instead of scraping presentation HTML and guessing what matters.

The current Streamable HTTP specification is simpler than earlier revisions. The 28 July 2026 revision removed the standalone GET stream endpoint and removed protocol-level sessions. A modern HTTP server exposes the MCP endpoint and receives requests by POST. Server-sent events can still appear inside an individual response where appropriate, but the old long-lived stream is gone.

Authorisation is optional for MCP implementations. When an HTTP implementation uses it, the specification defines a modern authorisation flow based on OAuth 2.1 and related standards.

That means an unauthenticated MCP is not automatically wrong. A public read-only site MCP is a choice the site can make.

This volume uses four worked examples I confirmed. It is not a census of every public website MCP on the web.

The ICT site at thierry-gilgen-ict.ch.

The Old Hand of Asia at theoldhandofasia.ch.

Joost de Valk, who writes about the web, on joost.blog. On 4 May 2026 he described his endpoint as intentionally public and unauthenticated.

And cocktail.glass, another site exposing a public MCP surface.

There are others. These four are simply the examples this volume relies on.

The Last Interface already said software can become something you call, the way you call a plug. This is not that architecture note. This is a website.

The Unappointed Operator already said a process you started can act on its own. You stop. You look. You name it. This is not that test. This stays on the site.

I sat with the two worked sites closest to this volume.

Both serve HTML for people. Both serve markdown. Both serve /llms.txt. Both serve /agents. Both expose /mcp as read-only and without authentication.

What a visitor sees in the browser is the public page. What the agent can ask the public door for, on these two sites, is that same public corpus.

That is the appointment they chose.

On the ICT site the public page says bring your agent. The public MCP needs no key. There is a second door, the Operator Library at /api/mcp/library, that needs a key. It is not the public surface. robots.txt disallows /api/mcp and allows /mcp.

On Old Hand the public pages — German and English — say the same thing: public MCP, no authentication, read-only. robots.txt disallows /api/. The conversation stays with the visitor's chosen agent.

Engawa is one toolkit that can cut those surfaces. It is a case, not the subject. A different project, tsuru0805/engawa-mcp, uses the same Japanese word for a local Python porch. Same name. Different object.

The toolkit can cut the door.

The interesting question is what the site chooses to put behind it.

Equivalence is a house rule

The two worked sites run a house rule.

Here is why.

A CMS holds more than the public page. It holds drafts. It holds notes that were never published. A page marked draft is not on the public site. A stranger who arrives in a browser does not see it.

If you hand an agent the database, you have handed it the draft.

The house rule says no.

The public agent corpus equals what an anonymous human visitor already sees. The Engawa README writes it as product law: "Engawa's public corpus must match what anonymous human visitors see—not merely what exists in a CMS or database."

That is a house rule. It is not an IETF fact. It is not a W3C fact. It is not an IAB fact.

The nearest analogues are not the same object.

RFC 9309 says how crawlers are asked to behave when requesting resources. It does not define a public agent corpus.

Google's cloaking policy is about another split: showing materially different content to a search engine and a human to manipulate ranking. That is not the same as deliberately exposing an agent-readable representation of the same public information.

WCAG 2.2 has a rule for a conforming alternate version. A second version must satisfy specific accessibility requirements and provide equivalent information. That is accessibility. It is not an agent corpus rule.

Cloudflare Content Signals let a site express preferences around search, training, and other AI use. Those signals are policy marks. They do not define which pages an agent may treat as its public set.

Publisher organisations are moving in a different direction.

IAB Tech Lab finalised the CoMP specification in April 2026. CoMP is not itself a bot blocker, licensing system, or marketplace. It is a protocol framework for communicating with authorised AI systems. Its documented workflow assumes that when there is no agreement, the owner can deny the request and direct the bot toward licensing.

IAB Australia, in July 2026, put the broader principle plainly: content used on terms, not by default. Its own decision matrix still distinguishes search, training crawlers, live AI agents, and operational bots rather than treating every machine request alike.

So there is no single industry rule that says public HTML automatically equals an agent corpus.

Among publishers and publisher bodies, a different principle is emerging: access on terms, not merely because the page is public.

Retrieval-augmented generation, or RAG, is one form of the same underlying move as grounding: fetch relevant material, then use it to produce the answer. The New York Times terms, effective 20 January 2026, restrict AI training, grounding, and RAG without written consent.

That is a contractual position, not a protocol rule.

And that distinction matters.

robots.txt is not a licence. llms.txt is not a licence. MCP is not a licence.

Each object answers a different question.

Availability is not appointment

A fetch is not an appointment.

The web page may be reachable. The map may make it easier to find. The door may make it easier to ask.

None of those facts grants a right by itself.

The wall is the ordinary web surface. The note expresses crawler preferences. The map points toward useful material. The door is a surface the site deliberately exposes.

On two sites the door is cut. What comes through it is what a stranger already sees. That is a choice those sites made. It is not a right the agent brought with it.

A site can decide what it exposes, through which surface, and on what terms.

The toolkit can cut the door. It cannot make that decision.

Who named the reader?

The site did.

Or it did not.

Sources

Library

Contested, and therefore unused as fact: llms.txt as an IETF or W3C standard, or as a crawl or training ban; any adoption percentage other than the HTTP Archive July 2025 measurement cited above; a 2026 web-wide census of llms.txt; Cloudflare defaulting to a block of all AI bots; ChatGPT-User always honouring robots.txt; Google-Extended stopping AI Overviews; any claim that the four public MCP examples in this volume constitute the whole market; tsuru0805/engawa-mcp as this toolkit; the Engawa public-corpus rule as an industry standard; agents being entitled to any anonymous-HTML corpus; bot market-share or traffic percentages; Google using or ignoring llms.txt.

This volume is not legal advice.

The Appointed Door

Key takeaways

Questions this volume answers

The wall is the default

A map is not a lock

A door you can name

Equivalence is a house rule

Availability is not appointment

Sources

The default fetch

The map

The door

Publication and licence

Library