llms.txt: what it is, how to write one, and whether it works

Who proposed llms.txt, and when?

Jeremy Howard, co-founder of Answer.AI and fast.ai, published the proposal on 3 September 2024. The specification lives at llmstxt.org with a reference implementation in the AnswerDotAI/llms-txt repository.

His argument was about context, not rankings. A model trying to understand a site has to decide whether to crawl the whole sitemap, how to strip navigation and ads out of HTML, and whether to pull in source code. Howard’s position was that site authors already know which pages matter, so they should just say so.

It is worth being precise about the origin, because it shapes everything else. llms.txt was designed to help a model that has already decided to read your documentation. It was never proposed as a ranking signal, and it has never gone through the IETF, the W3C, or any other standards body.

What is the exact llms.txt file format?

The spec defines Markdown, deliberately chosen over XML, in this order:

1. An H1 with the name of the project or site. This is the only required element.
2. A blockquote containing a short summary with the key context needed to understand the rest of the file.
3. Zero or more Markdown sections of free-form detail. These can use paragraphs, lists, or code blocks, but no headings above H2.
4. Zero or more H2 sections containing lists of Markdown links, each with an optional note after a colon.

One H2 name carries special meaning. A section titled ## Optional marks URLs that can be skipped when a shorter context is needed. Everything else is treated as core.

Can I see a real llms.txt example?

Here is a complete, spec-compliant file you can adapt. Replace the domain and descriptions with your own.

# Northwind Analytics

> Northwind Analytics is a self-serve product analytics tool for B2B SaaS
> teams. This file lists the pages that best explain what the product does,
> how it is priced, and how to integrate it.

Northwind has been running since 2021. Pricing is per tracked user, billed
monthly. Every page linked below is maintained by the team that owns it and
is updated on release.

## Product

- [What Northwind does](https://example.com/product): One-page overview of event tracking, funnels, and retention reports.
- [Pricing](https://example.com/pricing): Current plans, per-seat costs, and what the free tier includes.
- [Security and data handling](https://example.com/security): Storage locations, retention periods, and our GDPR position.

## Documentation

- [Quickstart](https://example.com/docs/quickstart): Install the SDK and send your first event in under ten minutes.
- [JavaScript SDK reference](https://example.com/docs/sdk-js): Full method list with parameters and return values.
- [Webhooks](https://example.com/docs/webhooks): Payload format, retry behavior, and signature verification.

## Company

- [About](https://example.com/about): Founding story, team size, and funding history.
- [Changelog](https://example.com/changelog): Dated list of shipped features, newest first.

## Optional

- [Engineering blog](https://example.com/blog): Longer posts on our storage architecture.
- [Conference talks](https://example.com/talks): Recorded talks from 2023 onward.

For a production example on a site with real traffic, look at the file Anthropic serves at platform.claude.com/docs/llms.txt.

Where do you put the llms.txt file?

At the root: https://yourdomain.com/llms.txt. The spec permits a subpath, but root is the convention and it is where anything looking for the file will look first.

Three technical details matter more than they sound. Serve it as plain text or Markdown, not HTML. Return a real HTTP 200, not a soft 404 with your error page. And keep it as actual Markdown, because Ahrefs had to filter out files that failed exactly these checks when studying 137,210 domains.

Google’s John Mueller has also suggested adding a noindex header to the file, since other sites can link to it and you probably do not want a raw Markdown index turning up in your own search results. Search Engine Journal covered that recommendation in July 2025.

What is the difference between llms.txt and llms-full.txt?

llms.txt is a map. llms-full.txt is the territory: your entire documentation set flattened into one enormous Markdown file that somebody can paste into a context window.

llms-full.txt is not part of Howard’s specification. Mintlify says it developed the format with Anthropic and rolled llms.txt support across every documentation site it hosts in November 2024, which is why thousands of sites acquired the file overnight. The spec’s own variants are named differently: llms-ctx.txt and llms-ctx-full.txt, generated by a command-line tool that either drops or keeps the Optional section.

How is llms.txt different from robots.txt and sitemap.xml?

They are often lumped together, and they should not be. Only one of the three is an actual standard.

The clearest way to see the gap: OpenAI’s crawler documentation tells webmasters to use robots.txt to manage GPTBot and OAI-SearchBot. Perplexity’s crawler documentation says the same for PerplexityBot. Both companies publish an llms.txt for their own developer docs. Neither says its crawlers read yours.

Do ChatGPT, Perplexity, Google and Claude actually read llms.txt?

There is no public evidence that they do, and a fair amount of measured evidence that they do not.

Google has been the most direct. John Mueller wrote in June 2025 that no AI system currently uses llms.txt, comparing it to the long-dead keywords meta tag. A month later, at a Search Central event on 23 July 2025, Gary Illyes said Google does not support llms.txt and does not plan to, and that normal SEO is what gets content into AI Overviews. Search Engine Land reported the session.

The server-log data points the same way. Ahrefs checked 137,210 domains with live traffic in May 2026 and found that 97% of the llms.txt files it found received zero requests that month, from bots or humans. Of the roughly 38,000 domains with a valid file, about 1,100 got any request at all. Of those requests, 96% came from bots, and the largest identifiable groups were SEO audit tools and tech-profiling crawlers rather than AI assistants. Retrieval bots such as OAI-SearchBot, PerplexityBot and Claude accounted for 1.1%.

A smaller independent measurement agrees. Evil Martians logged two months of AI traffic from early May to early July 2026 and counted roughly 770 fetches of llms.txt and llms-full.txt combined. Only 37 came from named AI assistants.

Two caveats, in fairness. Ahrefs notes that its figures measure requests, not whether anything acted on the contents, so every number is a ceiling rather than a floor. And its sample skews technical, since roughly 28% of the domains it looked at had a file at all, far above what HTTP Archive data shows for the wider web. None of that changes the direction of the finding.

Is llms.txt worth doing in 2026?

It depends entirely on who you are trying to reach.

If you publish developer documentation, yes. This is the use case the file was designed for, and it is where the traffic that does exist shows up. In the Ahrefs breakdown, the named AI categories that fetched llms.txt most were agents and infrastructure at 10.5% and training crawlers led by GPTBot at 5.3%, both of them coding-adjacent. Developers point coding assistants at documentation deliberately, and a clean index genuinely helps.

If you are chasing citations in AI search, no, or at least not as a priority. Nothing in the current data supports the idea that publishing llms.txt gets you into ChatGPT answers, Perplexity results, or AI Overviews. Adoption is still climbing, from 1.04% of the top 10,000 sites in July 2025 to 5.61% in June 2026 by HTTP Archive’s measurement, but adoption is not the same as effect. A large share of that growth is platform defaults rather than deliberate choices: Shopify alone shows 78.1% adoption because the platform ships the file.

The pragmatic position: it costs an afternoon, the downside is small if you maintain it, and the standard could still get picked up. Treat it as cheap insurance, not as a strategy. If somebody sells you llms.txt as the reason you will get cited by AI, they are selling you something.

What are the most common llms.txt mistakes?

• Auto-generating it from your sitemap. The entire premise is curation. A dump of every URL you own is a sitemap with extra steps, and it gives a model no signal about what matters.
• Letting it go stale. A file full of dead links and last year’s pricing is worse than no file. If you cannot commit to updating it when the site changes, skip it.
• Writing descriptions that say nothing. "Our pricing page" is filler. "Current plans, per-seat costs, and what the free tier includes" tells a model whether the page answers the question in front of it.
• Expecting it to block anything. llms.txt has no exclusion syntax. It cannot stop scraping, opt you out of training, or restrict access. That is robots.txt, and increasingly it is a WAF.
• Pasting your whole site into it. That is what llms-full.txt is for, and even then only for documentation.
• Serving it as HTML. A Markdown file rendered through your CMS template, wrapped in navigation and a cookie banner, defeats the point.

What actually gets you cited by AI engines?

The mechanics that hold up under measurement are unglamorous. AI answer engines decompose one question into many sub-questions and retrieve passages, not pages, so pages that answer the specific sub-questions around a topic get pulled in more often than pages optimized for a single head term. Concrete statistics, named sources, and explicit citations make a passage more quotable. Keyword density does not help and appears to hurt.

That is the same work that has always made a page worth reading, which is why Google’s position on AI Overviews is that ordinary SEO is what gets you there. If you want the longer version, we cover it in what generative engine optimization actually is and where GEO and traditional SEO diverge.

The harder problem is knowing whether any of it worked. Optimization for AI tracks how often your domain gets cited across ChatGPT, Perplexity, Google AI Overviews, Gemini and Claude, shows you which questions you are missing, and re-measures after you publish. You can see how the measurement works, browse the public visibility leaderboard, or start on the free plan and check where you stand today. Either way, measure the outcome rather than the file.