llms.txt

llms.txt is a proposed standard for a Markdown file at the root of a website (/llms.txt) that gives AI models a short, structured overview of the site and links to its most useful content.

llms.txt is a plain text file in Markdown format that a website places at its root, for example example.com/llms.txt. The idea is to offer language models a clean summary of what the site is about and a curated list of its most important pages, without the menus, scripts and ads that make normal web pages hard to process. The format was proposed in 2024 by Jeremy Howard of Answer.AI and has since been adopted by many documentation sites and software companies.

What the file contains

The proposal describes a simple structure:

  • A heading with the name of the site or project.
  • A short summary in a blockquote explaining what the site offers.
  • Optional extra paragraphs with key context.
  • Sections with lists of links to important pages, each with a brief description.

Some sites also publish an llms-full.txt file containing the full text of their documentation in one document, so an AI tool can load everything at once.

Does it work?

This is where expectations need managing. llms.txt is a community proposal, not an official web standard, and the large AI search providers have not confirmed that they use it to choose or rank sources. Google has stated that it does not use llms.txt for Search or AI Overviews. In practice the file is most useful for developer tools and coding assistants that are pointed at documentation, and for AI agents that are explicitly told to read it.

It is also not a way to control access. llms.txt does not block or allow crawlers. That remains the job of robots.txt.

Should you add one?

For most sites it is a low-effort, low-risk addition. If you run a software product, an API or a knowledge-heavy site, an llms.txt can make it easier for AI tools to find the right pages. For a typical business or publisher site, the effect on visibility in AI answers is likely small. Your time is better spent on crawlability, clear content and mentions on trusted sites.

Tips if you create one

  • Keep it short and focused on pages that answer real questions.
  • Use descriptive link texts and one-line descriptions.
  • Update it when you add or remove important content.
  • Make sure the linked pages are accessible to crawlers and not blocked in robots.txt.

Common mistakes

Treating llms.txt as a ranking factor, copying an entire sitemap into it, or using it to try to steer AI tools with promotional claims.

Questions about llms.txt

What is llms.txt used for?

It gives AI tools a short, structured summary of a website and links to its key pages in Markdown, making the content easier for language models to process.

Does Google use llms.txt?

No. Google has said it does not use llms.txt for Search or AI Overviews.

Is llms.txt the same as robots.txt?

No. robots.txt tells crawlers what they may access. llms.txt only offers a content overview and does not control access.

Related terms