What is llms.txt, and does it help AI assistants find you?

What is llms.txt?

An llms.txt file is a short text file that tells an AI tool what a website is and where its best pages are. It lives at the root of the site, at an address such as example.com/llms.txt, in the same place as robots.txt.

The file is written in Markdown, a simple way of marking up text with symbols such as a hash for a heading. It holds the site's name, a one-paragraph summary, and lists of links to pages with a short note on each. The idea is that an AI tool can read one small file instead of crawling a whole site and guessing what matters.

LLM stands for large language model, the kind of AI behind ChatGPT, Claude and Gemini. So the name means, roughly, a text file for language models.

Think of it as the floor plan by the entrance of a museum. It does not change what is in the rooms. It helps a visitor who stops to read it find the right room faster, and a visitor who walks straight past it loses nothing.

Where llms.txt came from

Jeremy Howard proposed llms.txt on 3 September 2024, on the site llmstxt.org. The site says a second version of the proposal was published on 10 August 2026, updated from two years of seeing how it was used.

The proposal gives two reasons for the file. Context windows, the amount of text a model can read at once, are still too small for most websites in their entirety. And converting a page of HTML, with its menus, scripts and adverts, back into clean text is difficult and imprecise.

The proposal also says the file was expected to be useful mainly for inference rather than training. Inference is when a model is answering a question right now. Training is when a model learns from text in advance. So llms.txt is meant for a tool reading your site at the moment someone asks about it, not for teaching a model in general.

It also suggests that pages agents might need offer a clean Markdown copy at the same address with .md added. That part is optional, and many sites skip it.

What changed in version 2

The changes page lists what version 2 altered. Its author writes that thousands of sites now publish the file, that documentation platforms generate one automatically, and that coding agents use them reliably. Those are his statements, not a measurement this post can check.

• A page can now point to its Markdown copy and to the llms.txt file that covers it, using standard link tags in the page or a Link header.
• A Markdown copy may be named page.html.md or page.md. Version 1 allowed only the first.
• A file placed in a folder covers the pages under that folder, and the most specific file applies.
• The tool for expanding a file into one long block of text is no longer part of the proposal.
• The Optional section is still allowed, but it no longer tells any tool what to leave out. It is a convention for secondary links.

The page also states how the file is expected to be used: an agent reads or searches the llms.txt file, then follows the links that are relevant to its task. Those links should lead to pages that are easy for a model to read.

What goes in an llms.txt file

Only the first part is required: a heading with the name of the site or project. Everything else is recommended in a fixed order.

Our companion post, an llms.txt example, line by line, walks through a complete file and explains each line. This post stays with the bigger question: does anyone read it?

What llms-full.txt is

Some sites also publish a second file, llms-full.txt, which holds the full text of their documentation in one place rather than links to it. It is a common practice rather than part of the proposal: the version 2 text at llmstxt.org does not define it.

These files can be very large. On 10 October 2026 the llms-full.txt files of Vercel and Svelte were each over a megabyte, and the one Anthropic publishes for its developer documentation was over 40 megabytes. The largest is far more than any model can read at once, which is the problem the original proposal set out to solve.

Does llms.txt work? What the evidence says

There is no public evidence that llms.txt helps a site get named or cited in AI answers. The largest published study found that almost no AI crawler asks for the file, and Google says its search ignores it.

What Google says

Google's guide to optimizing for generative AI features, last updated on 10 July 2026, says you do not need to create new machine readable files, AI text files, markup or Markdown to appear in Google Search. It says Google Search itself does not use them and ignores them. It adds that keeping such files for other services is fine, and that they will neither help nor harm a site in Google Search.

Google's page on AI features and your website says the same for AI Overviews and AI Mode. A page needs to be indexed and eligible for a snippet, and there are no additional technical requirements.

What the server logs show

Ahrefs published a study on 15 June 2026 of 137,210 domains that use its web analytics, covering May 2026. Of those, 38,360 sites, about 28 per cent, published a valid llms.txt file. Of those files, 97 per cent received no requests at all that month.

The retrieval bots row is the one that matters for being cited. Those are the bots an assistant sends to read a page while it writes an answer, and they made 1.1 per cent of requests to files that were requested at all. The study also found that no AI bot asked for an llms.txt file on sites that did not have one.

Who does support it

Google's browser team treats llms.txt differently from its search team. Chrome's Lighthouse tool, which audits web pages, added an llms.txt check in May 2026. It describes the file as an emerging convention for giving LLMs and AI agents a machine-readable summary of a site. The Lighthouse scoring page says the category it sits in is experimental and based on proposed standards.

The audit itself is narrow. Its page says a missing file is marked not applicable, and a page is flagged when a server error occurs while fetching the file. Passing it shows the file can be fetched. It says nothing about whether any assistant reads it.

The AI companies also publish llms.txt files for their own developer documentation. The proposal names OpenAI, Anthropic and Gemini as examples. Publishing one is not the same as using other sites' files to choose what to cite, and none has said its assistant does that.

An llms.txt file is a helpful map for a tool that chooses to read it. No major assistant has said it reads the map before deciding whom to recommend.

llms.txt, robots.txt and sitemap.xml compared

The three files sit side by side at the root of a site and are often confused. Each does a different job, and only one of them can stop an AI crawler.

The robots.txt claim comes from OpenAI's crawler page, which says sites opted out of OAI-SearchBot will not be shown in ChatGPT search answers, though they can still appear as navigational links. If you have an hour to spend on AI visibility, robots.txt is a far better use of it than llms.txt. Our page on AI crawler access explains what to check.

Should you add an llms.txt file?

Add one if it takes you less than an hour and you have the basics done, but do not expect it to change what assistants say about you. Do the work that has evidence behind it first.

1. Check that robots.txt allows the search crawlers, such as OAI-SearchBot and PerplexityBot.
2. Make sure your important pages are indexed and readable as plain text, not locked in PDFs or images.
3. Make sure the pages answer the questions buyers actually ask, near the top.
4. Then, if you like, add an llms.txt file that names your site, sums it up in two sentences and links your key pages.
5. Keep it short, keep it true, and update it when a price or a product changes.

There is one clear case for it: documentation for developers. Coding tools read documentation directly. In the Ahrefs study, Claude Code, a coding tool, requested the files more often than any AI retrieval bot or assistant. That may be why documentation platforms generate the file automatically, as llmstxt.org notes. For a marketing site, the case is weaker.

There is also a small risk to watch. If your llms.txt says something your pages no longer say, such as an old price, any tool that does read it gets the wrong fact. A file that is generated from the same data as your pages avoids that.