llms.txt is a proposed file that sits at the root of your website and gives AI systems a short, curated guide to your most important content. It's written in markdown, it's easy to create, and it's popular with documentation-heavy products. But it isn't an official standard, and the major AI search providers haven't said they use it to choose sources. Add one if it's cheap for you; don't expect it to change your visibility on its own.
Where it came from
Jeremy Howard of Answer.AI proposed llms.txt in September 2024. The idea: websites are built for browsers, full of navigation, scripts and layout. Language models work better with concise, structured text. A single markdown file listing the key pages, with short descriptions, gives AI tools a quick way to understand a site.
What a file looks like
The proposal suggests a simple structure:
# Example Analytics
> Example Analytics is a product analytics platform for mobile app teams. This file lists our key product, pricing and documentation pages.
## Product
- [Overview](https://www.example.com/product): What the platform does and who it's for
- [Pricing](https://www.example.com/pricing): Plans, limits and the free tier
## Docs
- [Quickstart](https://docs.example.com/quickstart): Install the SDK and send your first event
- [API reference](https://docs.example.com/api): Endpoints, authentication and rate limits
## Optional
- [Blog](https://www.example.com/blog): Guides on product analytics
The elements are an H1 with the name, a short blockquote summary, and sections of links with one-line descriptions. Some sites also publish llms-full.txt with the full text of their key docs in one file.
What it can do
- Help AI tools that look for it. Some AI coding assistants, agents and developer tools fetch llms.txt or llms-full.txt when given a site to work with.
- Give a curated map. It points AI tools to your canonical, most useful pages instead of everything.
- Force useful clarity. Writing a one-paragraph summary and a list of key pages is a good exercise in itself.
What it can't do
- It doesn't control access. That's robots.txt's job. See our AI crawlers guide.
- It isn't a confirmed ranking or citation signal. Major AI search products haven't said they use it, and Google representatives have said Google Search doesn't.
- It doesn't replace your pages. AI search answers are built from retrieved page content. If the pages are hard to read, llms.txt won't fix that.
- It doesn't fix rendering. If your pages are client-rendered, AI crawlers still can't read them. See JavaScript rendering and AI crawlers.
Who should add one
| Site type | Recommendation |
|---|---|
| Developer tools, APIs, SDKs | Yes; AI coding tools may use it. Consider llms-full.txt for docs. |
| SaaS with substantial docs | Worth adding; low effort |
| Content publishers | Optional |
| Ecommerce | Optional; focus on product data and pages first |
| Local businesses | Low priority |
How to write one in 15 minutes
- Write the summary: one or two sentences on what you are and who you serve, matching your brand description elsewhere.
- List 10 to 30 key pages: product, pricing, use cases, comparisons, docs, policies.
- Add one-line descriptions that say what each page answers.
- Group into sections such as Product, Docs, Company, Optional.
- Publish at
/llms.txtas plain text, and keep it updated when key pages change.
Some site generators and documentation platforms can create llms.txt automatically.
Priorities before llms.txt
If you're deciding where to spend time, these come first:
- AI search crawlers allowed in robots.txt and your firewall
- Key content in server-rendered HTML
- Answer-first, specific content on key pages. See how to write content AI assistants cite.
- Accurate structured data. See schema markup for AEO.
- Indexing in Google and Bing
Then add llms.txt as a small extra.
Common questions from teams
Will llms.txt hurt anything? No. It's a static file that doesn't change how your site works or how crawlers access it.
Should it list every page? No. It's a curated guide. List the pages that best explain your product and answer common questions.
Does it need to match robots.txt? Don't list pages you block from crawlers; it sends a mixed signal and the links won't be fetchable anyway.
Who maintains it? Whoever owns your site navigation or docs. Update it when key pages are added, renamed or retired.
Should llms-full.txt include everything? For docs, include the content an AI tool needs to answer common developer questions. Keep it current, because outdated instructions in a convenient file are worse than none.
For a shorter take, see our explainer, what is llms.txt?.
The bottom line
llms.txt is a sensible, low-cost idea with limited confirmed impact on AI search today. Developer and documentation-heavy products should add one, since AI tools in that space may use it. Everyone else can add it in a few minutes, as long as it doesn't distract from the work that actually determines AI visibility: access, rendering, clear content and third-party authority.
