Technical AEO

    llms.txt: What It Is and Whether You Need One

    What the proposed llms.txt standard is, what it can and can't do for AI visibility, who it's useful for, and how to write one in a few minutes.

    Saarthak Srivastav

    Saarthak Srivastav

    · 4 min read

    Key takeaways

    • llms.txt is a proposed markdown file at your site's root that gives AI systems a curated summary and map of your most important content.
    • It was proposed in 2024 by Jeremy Howard of Answer.AI. It's not an official web standard.
    • Major AI search providers haven't said they use it for ranking or citations, and Google representatives have said Google Search doesn't use it.
    • It's most useful for documentation-heavy and developer products, where AI coding tools and agents may read it.
    • It's cheap to add, but it's no substitute for crawlable HTML, clear content and robots.txt access.

    llms.txt is a proposed file that sits at the root of your website and gives AI systems a short, curated guide to your most important content. It's written in markdown, it's easy to create, and it's popular with documentation-heavy products. But it isn't an official standard, and the major AI search providers haven't said they use it to choose sources. Add one if it's cheap for you; don't expect it to change your visibility on its own.

    Where it came from

    Jeremy Howard of Answer.AI proposed llms.txt in September 2024. The idea: websites are built for browsers, full of navigation, scripts and layout. Language models work better with concise, structured text. A single markdown file listing the key pages, with short descriptions, gives AI tools a quick way to understand a site.

    What a file looks like

    The proposal suggests a simple structure:

    # Example Analytics
    
    > Example Analytics is a product analytics platform for mobile app teams. This file lists our key product, pricing and documentation pages.
    
    ## Product
    
    - [Overview](https://www.example.com/product): What the platform does and who it's for
    - [Pricing](https://www.example.com/pricing): Plans, limits and the free tier
    
    ## Docs
    
    - [Quickstart](https://docs.example.com/quickstart): Install the SDK and send your first event
    - [API reference](https://docs.example.com/api): Endpoints, authentication and rate limits
    
    ## Optional
    
    - [Blog](https://www.example.com/blog): Guides on product analytics
    

    The elements are an H1 with the name, a short blockquote summary, and sections of links with one-line descriptions. Some sites also publish llms-full.txt with the full text of their key docs in one file.

    What it can do

    • Help AI tools that look for it. Some AI coding assistants, agents and developer tools fetch llms.txt or llms-full.txt when given a site to work with.
    • Give a curated map. It points AI tools to your canonical, most useful pages instead of everything.
    • Force useful clarity. Writing a one-paragraph summary and a list of key pages is a good exercise in itself.

    What it can't do

    • It doesn't control access. That's robots.txt's job. See our AI crawlers guide.
    • It isn't a confirmed ranking or citation signal. Major AI search products haven't said they use it, and Google representatives have said Google Search doesn't.
    • It doesn't replace your pages. AI search answers are built from retrieved page content. If the pages are hard to read, llms.txt won't fix that.
    • It doesn't fix rendering. If your pages are client-rendered, AI crawlers still can't read them. See JavaScript rendering and AI crawlers.

    Who should add one

    Site type Recommendation
    Developer tools, APIs, SDKs Yes; AI coding tools may use it. Consider llms-full.txt for docs.
    SaaS with substantial docs Worth adding; low effort
    Content publishers Optional
    Ecommerce Optional; focus on product data and pages first
    Local businesses Low priority

    How to write one in 15 minutes

    1. Write the summary: one or two sentences on what you are and who you serve, matching your brand description elsewhere.
    2. List 10 to 30 key pages: product, pricing, use cases, comparisons, docs, policies.
    3. Add one-line descriptions that say what each page answers.
    4. Group into sections such as Product, Docs, Company, Optional.
    5. Publish at /llms.txt as plain text, and keep it updated when key pages change.

    Some site generators and documentation platforms can create llms.txt automatically.

    Priorities before llms.txt

    If you're deciding where to spend time, these come first:

    1. AI search crawlers allowed in robots.txt and your firewall
    2. Key content in server-rendered HTML
    3. Answer-first, specific content on key pages. See how to write content AI assistants cite.
    4. Accurate structured data. See schema markup for AEO.
    5. Indexing in Google and Bing

    Then add llms.txt as a small extra.

    Common questions from teams

    Will llms.txt hurt anything? No. It's a static file that doesn't change how your site works or how crawlers access it.

    Should it list every page? No. It's a curated guide. List the pages that best explain your product and answer common questions.

    Does it need to match robots.txt? Don't list pages you block from crawlers; it sends a mixed signal and the links won't be fetchable anyway.

    Who maintains it? Whoever owns your site navigation or docs. Update it when key pages are added, renamed or retired.

    Should llms-full.txt include everything? For docs, include the content an AI tool needs to answer common developer questions. Keep it current, because outdated instructions in a convenient file are worse than none.

    For a shorter take, see our explainer, what is llms.txt?.

    The bottom line

    llms.txt is a sensible, low-cost idea with limited confirmed impact on AI search today. Developer and documentation-heavy products should add one, since AI tools in that space may use it. Everyone else can add it in a few minutes, as long as it doesn't distract from the work that actually determines AI visibility: access, rendering, clear content and third-party authority.

    FAQ

    Frequently asked questions

    What is llms.txt?

    llms.txt is a proposed file, placed at yoursite.com/llms.txt, written in markdown. It contains a short description of the site and links to key pages, so large language models and AI tools can quickly understand what the site offers.

    Does llms.txt help with ChatGPT or Google AI Overviews?

    There's no public confirmation that major AI search products use llms.txt for ranking or citations. Google representatives have said Google Search doesn't use it. Treat it as optional.

    Is llms.txt the same as robots.txt?

    No. robots.txt tells crawlers what they may access. llms.txt is a guide to your content. It doesn't grant or block access.

    What is llms-full.txt?

    Some sites also publish llms-full.txt, a single file containing the full text of their key documentation, so AI tools can load it in one request. It's mainly used by developer documentation sites.

    • llms.txt
    • technical SEO
    • AI crawlers
    • documentation
    Saarthak Srivastav

    Written by

    Saarthak Srivastav

    AEO & GEO, AEO-Rank

    Saarthak Srivastav writes and reviews articles on the AEO-Rank blog, drawing on what the AEO-Rank team sees tracking AI answers across ChatGPT, Gemini, Claude, Perplexity and Grok.