What is llms.txt, and should your site have one?
llms.txt is a small Markdown file that gives AI agents a curated map of your site. Here's what goes in it, how it differs from robots.txt and sitemaps, and how to write one.
llms.txt is a Markdown file at your site's root that gives AI agents a short summary of your site and links to its most useful pages. It doesn't control crawling the way robots.txt does; it's a curated guide that agents read on demand. It takes minutes to write, so most content sites should have one.
- Jeremy Howard proposed /llms.txt on 3 September 2024; version 2 of the proposal was published in August 2026.
- The file is Markdown: an H1 with the site name (the only required part), a short blockquote summary, then H2 sections listing links.
- It complements robots.txt and sitemap.xml rather than replacing them: robots.txt sets access, the sitemap lists everything, llms.txt curates.
- Chrome's Lighthouse checks for it as part of its agentic-browsing audits, and OpenAI, Anthropic and Google publish llms.txt files for their developer docs.
AI agents read the web constantly now, but web pages are built for people: navigation, scripts and ads wrapped around the part an agent actually needs. llms.txt is a simple fix, a plain Markdown file that hands agents a short, curated map of your site.
Where it came from
Jeremy Howard of Answer.AI proposed /llms.txt on 3 September 2024. Version 2 of the proposal, updated after two years of real-world adoption, was published in August 2026. Documentation platforms generate it automatically, WordPress SEO plugins such as Yoast and AIOSEO can create one, and Chrome’s Lighthouse audits sites for it.
What goes in the file
The format is Markdown, in this order:
- An H1 with your site or project name. This is the only required part.
- A blockquote with a short summary of what the site is.
- Optional paragraphs or lists with extra context.
- H2 sections that each hold a list of links:
[Page name](url): what it covers.
A section called ## Optional is, by convention, for links an agent can skip when it’s short on context.
How it differs from robots.txt and sitemaps
| File | Job |
|---|---|
| robots.txt | Says which bots may access which paths |
| sitemap.xml | Lists every indexable page for search engines |
| llms.txt | Curates the pages an agent should read first, with context |
How to write a good one
- Lead with a one-sentence summary a stranger would understand.
- Link to your best pages, not all of them.
- Give each link a few words on what it answers.
- Test it: give an AI assistant only your llms.txt and ask it questions about your site.
Add one. It costs almost nothing, it helps agents find your best pages instead of guessing, and it's now checked by mainstream tools. Just don't expect it alone to change rankings; what you publish still does the heavy lifting.
Questions people ask
Is llms.txt an official web standard?
No. It's an open proposal maintained on GitHub, not a W3C or IETF standard, though it's widely adopted.
Does llms.txt block AI crawlers?
No. Access rules live in robots.txt. llms.txt only describes your site and points to useful content.
Where should the file go?
At the root (yoursite.com/llms.txt), or at a subpath such as /docs/llms.txt to cover only the pages under it.
Sources
- The /llms.txt file, v2, llmstxt.org
- llms-txt repository, Answer.AI on GitHub
Spotted something wrong? We fix errors fast and note every correction. How corrections work or Tell us