llms.txt Explained: What It Is, How to Write One, and Whether It Actually Works

What is llms.txt?

llms.txt is a plain markdown file placed at the root of a website — yoursite.com/llms.txt — that gives AI systems a curated index of a site's most important pages. It was proposed by Jeremy Howard of Answer.AI in September 2024, with the specification published at llmstxt.org. It is a voluntary convention, not a web standard.

The reasoning behind it is sound. A rendered web page is mostly navigation, scripts, cookie banners and ads. A language model working with a limited context window has to spend budget parsing all of that before it reaches the paragraph that answers the question. A clean markdown map that says *read these pages, in this order* solves a real problem — at least in theory.

Whether it solves that problem in practice is a different question, and the honest answer as of August 2026 is: mostly not, for most sites. Here is what the evidence shows, what Google has said on the record, who is genuinely using the file, and how to decide whether it's worth twenty minutes of your time.

Do AI crawlers actually read llms.txt?

Largely, no. Three independent lines of evidence point the same direction.

The SE Ranking result is the one worth sitting with. Removing llms.txt as a variable from their model made the model *more* accurate — meaning the file was contributing noise rather than signal.

Adoption on the publishing side is similarly thin. By 31 March 2026, 37 of the Fortune 500 — 7.4% — had shipped one. A separate analysis found that roughly 40% of the files that do exist are empty plugin stubs rather than curated indexes.

What has Google said about llms.txt?

Google's position is the clearest of any major provider, and it is negative.

In July 2025, Gary Illyes confirmed Google does not support llms.txt and has no plans to. John Mueller compared it to the keywords meta tag — a comparison SEOs recognise as shorthand for *a signal that lost its value through overuse and was eventually set aside*. Google's own guidance on AI Overviews and AI Mode states that those surfaces rely on the same signals as ordinary search.

There is one wrinkle worth knowing, because it gets cited as evidence of a reversal: Google added llms.txt files to its own developer and Search Central documentation in December 2025. Google's Search team has been explicit that publishing one for their own docs does not imply endorsement for citation purposes. Both things are true at once — the file is useful for developer documentation, and Google does not use it for search.

No major AI provider — OpenAI, Anthropic, Google, Meta or Mistral — has publicly committed to reading or acting on llms.txt in a production search or answer system.

So who is actually using it?

Coding agents. This is the real use case, and it is genuinely real.

Tools like Cursor, Claude Code, GitHub Copilot, Cline and Windsurf can be pointed at an llms.txt or llms-full.txt to pull clean, current documentation into a coding session. For a developer-tools company, that means a user's AI assistant reads your actual API documentation instead of hallucinating a method signature from an outdated training snapshot. That is a concrete, measurable benefit.

Note the mechanism, because it is the opposite of the SEO pitch. The file is fetched on demand by a tool the user configured — not discovered by a crawler deciding what to cite.

Adoption follows that logic exactly. Mintlify rolled the file out across every documentation site it hosts in November 2024, which is why Anthropic, Cursor, Coinbase, Pinecone and Windsurf all have one. Stripe, Vercel, Cloudflare and Supabase ship one on their developer docs. Framework documentation is where the convention actually lives.

Media sites, by contrast, have close to zero adoption — and media is the sector most exposed to AI crawling. That gap tells you the file is not being adopted as a defensive or visibility measure. It's being adopted where it serves a tool.

What goes inside an llms.txt file?

The specification is deliberately minimal. It is markdown, and the structure is:

# TapasSEO

> Productized SEO and GEO services with fixed per-deliverable pricing.
> No retainers. Every service is scoped, priced and delivered as a
> defined output.

## Services

- [Site SEO Audit](https://tapasseo.com/site-audit/): Full technical,
  content and authority audit across 22 sections.
- [GEO/AEO Discovery Audit](https://tapasseo.com/geo-audit/): Measures
  brand visibility across ChatGPT, Perplexity, Claude and AI Overviews.
- [llms.txt Config](https://tapasseo.com/llms-txt-config/): Curated
  llms.txt and AI crawler configuration.

## Guides

- [What SEO Actually Costs](https://tapasseo.com/guides/seo-cost/):
  Real per-deliverable pricing, not ranges.

## Optional

- [Sample audit](https://tapasseo.com/site-audit/sample/): A complete
  worked example.

The rules that matter:

• One H1 with the site or brand name.
• A blockquote summary immediately after it — one or two sentences describing what the site is. This is the single highest-value line in the file, because it is the part most likely to be read as an identity statement.
• H2 sections grouping links. Each link gets a one-sentence description after a colon.
• An ## Optional section at the end. The spec treats this as content that can be skipped when context is tight.
• Ten to forty links, not four hundred. The entire value is curation. A file that lists every URL on the site is just a sitemap in a worse format.

llms-full.txt is a companion file containing the full markdown text of your key pages rather than links to them. It is worth publishing if you run documentation. For a marketing site it is usually unnecessary bulk.

llms.txt vs robots.txt vs sitemap.xml

These three get conflated constantly, and the confusion causes real mistakes.

The critical row is the fourth. llms.txt does not block anything. A number of articles describe it as a way to opt out of AI training, complete with User-agent: * and Disallow: directives. That is factually wrong — those are robots.txt directives, and putting them in llms.txt achieves nothing.

If you want to control AI crawler access, that belongs in robots.txt, where every major provider documents their user agents:

• GPTBot — OpenAI, foundation model training
• OAI-SearchBot — OpenAI, indexing for ChatGPT search
• ChatGPT-User — OpenAI, fetches a page when a user asks ChatGPT to read a specific URL
• ClaudeBot — Anthropic, training
• Claude-SearchBot — Anthropic, indexing for Claude search
• PerplexityBot — Perplexity
• Google-Extended — Google, Gemini training and grounding (separate from Googlebot)

Blocking any of these has no effect on your Google rankings. Googlebot is a separate crawler with separate rules.

Common mistakes

Generating a markdown copy of every page. Some plugins do this automatically. If those markdown files are indexable, you have just created a duplicate of your entire site at scale — which dilutes crawl budget and can suppress the originals. If you publish page-level markdown, make sure it is noindexed.

Shipping a plugin stub and calling it done. Roughly 40% of existing files are auto-generated placeholders with no curation. A stub delivers none of the benefit, because the benefit was always the curation.

Treating it as access control. Covered above. It controls nothing.

Letting it go stale. A file that lists pages you deleted eight months ago is worse than no file. Treat it like a sitemap: review it when your important pages change.

Prioritising it over work that has evidence behind it. This is the expensive one. If you have limited hours this quarter, crawl accessibility, answer-first content structure and original data all have measurable effects on AI citation. llms.txt does not, yet.

How to check whether anything reads yours

Two methods, both cheap:

1. Filter your server access logs for requests to /llms.txt and /llms-full.txt, then filter by known AI user agents. Cloudflare's bot analytics will break this down by user agent without touching raw logs.
2. Plant a honeypot. Include one URL inside the file that appears nowhere else on your site — no internal links, not in your sitemap. Any traffic to that URL came from something that read the file. Watch it for ninety days.

Do the second one. It is the only way to get an answer specific to your site rather than an industry average.

Should you publish one?

The honest framing: llms.txt costs almost nothing, carries no risk, and forces a useful exercise — deciding which twenty pages actually represent your expertise. Most businesses find that exercise more valuable than the file.

What it is not is a citation lever. If llms.txt is presented as the reason your brand will start appearing in ChatGPT, the current data does not support that claim — so it's worth understanding what else the work involves and where the visibility gains are expected to come from.

Frequently asked questions

Is llms.txt an official web standard? No. It is a community proposal with no backing from the W3C, IETF or any recognised standards body, and no enforcement mechanism. AI providers adopt it, or don't, entirely on their own terms.

Does llms.txt affect my Google rankings? No. Google has stated it does not use the file, and it is not a ranking input on any surface, including AI Overviews and AI Mode.

Does llms.txt stop AI companies training on my content? No. It has no access-control function whatsoever. Training opt-outs belong in robots.txt, using the specific user agents each provider documents.

Is llms.txt the same as robots.txt? They are close to opposites. robots.txt tells crawlers what they may not access and is broadly honoured. llms.txt suggests what an AI system should read first and is rarely fetched.

Will llms.txt get me cited in ChatGPT or Perplexity? There is no evidence it will. Citation appears to follow entity strength, answer-first content structure, third-party mentions and freshness. The file makes your content marginally easier to ingest *if* something visits it.

How often should I update it? Whenever your important pages change — the same cadence you'd apply to a sitemap. A quarterly review is a reasonable default.

Should I publish llms-full.txt as well? Only if you run documentation that coding agents consume. For a marketing site it is bulk without a reader.

Get your llms.txt configured properly

TapasSEO's llms.txt Config service delivers a curated llms.txt, a matching llms-full.txt where it makes sense, and a complete AI crawler configuration in robots.txt covering GPTBot, ClaudeBot, PerplexityBot, Google-Extended and the search-specific agents — with the training-versus-retrieval split set deliberately rather than by accident.

We will also tell you plainly if your site is one of the ones that won't benefit. That conversation is free.

llms.txt Config — $150 →

If AI visibility is the actual goal, the GEO/AEO Discovery Audit measures where your brand currently appears across ChatGPT, Perplexity, Claude and AI Overviews — which is the number llms.txt is usually sold as improving.