Treating llms.txt as a Distribution Plan

Shipping the machine-readable layer is cheap and worth doing. Expecting it to bring readers is a different claim, and the evidence is against it.

Pattern

You are building something agent-first, so you ship the machine layer properly: llms.txt, a .md mirror of every page, a JSON catalogue, a permissive robots.txt. It takes an afternoon.

Then the plan quietly becomes: models will read this, models will cite us, readers will follow. The machine layer stops being a feature and becomes the strategy. Nobody writes that down, which is why nobody checks it.

Why it looks right

Because most of the individual beliefs are true, and the layer is genuinely good work.

Machines really are the majority reader now — automated requests crossed 57.5% of HTML traffic in mid-2026. Blocking AI crawlers really does cost you: publishers who blocked lost 7% of weekly traffic in 6 weeks with no measurable protection in exchange.

Both are reasons to build the layer. Neither is evidence that the layer brings readers. The step from "machines read the web" to "machines will send me people" is the one that never gets examined.

Why it fails

Three independent measurements point the same way.

Nobody fetches the file. Google's position, stated plainly:

None of the AI services have said they're using LLMs.TXT, and you can tell when you look at your server logs that they don't even check for it.

An independent scan of 300,000 domains found about 10% had adopted llms.txt with no measurable lift in citations. Adoption is not consumption.

Crawling is not traffic. One AI crawler logs on the order of 24,000 crawled pages per single referral. Roughly half of AI crawl requests serve training, under a tenth serve live search. A crawler arriving is not a reader arriving, and a dashboard that counts bot hits as interest will feel wonderful and mean nothing.

The timescale is wrong for a new site. Practitioners put the first meaningful result from generative-engine work at around 16 months — fine as a background investment, useless as a plan for your first 90 days.

What does correlate with being cited is off-site. In one study of 75,000 brands, mentions on video and in ordinary web text outranked backlinks by two to three times as predictors of AI visibility, and one large forum accounted for 40% of citations sampled. The machine layer is on your server. The signal is not.

Receipt: the first 48 hours

This page was published on 14 August 2026, three days before the site it sits on went live. The argument above was assembled from other people's measurements. Then this site launched, and its own logs settled the question faster than expected.

Between 17 and 18 August 2026 the site logged 62,000 HTTP requests. A single crawler collecting training data accounted for 47,000 of them — three quarters of everything. It arrived 29 minutes after the domain went live, at a point when no link to the site existed anywhere. Nobody announced it. A public certificate log is enough.

Two caveats about that number, and they matter more than the number does. 26,000 of the 62,000 requests returned 404, and the logs available here do not break status codes down by client. The count of pages actually read is therefore somewhere below the request count, and there is no honest way to say where from this data. A post that rounded 47,000 requests up into "47,000 pages read" would be committing the exact error this page is about.

The other side of the ledger is not in doubt. Browser-class traffic in the same window was about 4,000 requests, essentially all from the operator's own country and machines. Identified outside human readers: none.

The figure predicted in this article was 24,000 crawled pages per referral. Nothing here refines that ratio — the sample is small and the denominator is zero. The point was never the ratio:

The crawl happened. It brought no one. Both halves were predictable, and only the first half feels like progress.

Two details are worth more than the headline number.

The crawlers that came are not the crawlers that cite. The clients that arrived collect training data. The client that feeds search citations — the one that produces a visible mention with a working link — did not appear once. They are separate crawlers with separate names, and only the second is a distribution channel. A dashboard that reports "AI traffic" as one bucket hides exactly this distinction.

91% of the crawl hit a staging hostname marked noindex. For training that is irrelevant; the text is read either way. For search it means the corpus was collected from an address no index will ever surface. The machine layer worked perfectly and pointed at the wrong door.

The correction this forces is small and unwelcome: the number to watch was never how much of the site machines read. It is whether anything a machine read ever comes back as a person.

Instead

Ship the layer. Demote the expectation. Concretely:

• Keep llms.txt, keep the .md mirror, keep robots.txt open to AI crawlers. The cost is hours and blocking is measurably worse than allowing.
• Serve the mirror as text/markdown, and put the date and version in the body text, not only in metadata. Cited content skews fresher, and freshness has to survive being pulled out of context.
• Move the machine layer out of your success metrics. Percentage of agent traffic becomes a diagnostic you look at, not a number you steer by.
• Do not point anyone at one giant file as the recommended fetch. Longer context measurably degrades extraction accuracy. Give a catalogue and let the agent pick.
• Then go where people already talk — the part the afternoon of file-writing lets you feel you have already done.

The machine layer is a product feature serving the moment someone hands your link to their agent. It is a good feature. It is not a plan.

[Treating llms.txt as a Distribution Plan](https://stillvalid.dev/patterns/llms-txt-as-a-plan) — stillvalid, SV-3809 (anti-pattern, verified 2026-08-18)