llms.txt is a proposed standard where you place a simple markdown file at /llms.txt that points large language models to the most important and most readable content on your website. The idea was launched by Jeremy Howard at Answer.AI in September 2024. But the big question isn't what it is — it's whether it actually works.
What llms.txt actually does
An llms.txt file is a curated table of contents written in markdown: a short description of what the site is, followed by links to the most important pages, often with clean markdown versions of the content. The idea is to save the AI model the work of parsing a cluttered website, and instead serve the core directly.
llms.txt vs. robots.txt: not the same thing
They're often confused because both are text files at the root of a site, but they solve different problems:
- robots.txt controls access — which parts crawlers are allowed to fetch.
- llms.txt controls emphasis — it points AI toward what you consider most important.
Does it work? What the data says
Here we need to be honest. None of the major AI providers — OpenAI, Anthropic, or Google — have publicly confirmed that they read llms.txt. Industry surveys of large numbers of websites suggest the vast majority of llms.txt files are never fetched by an AI crawler at all. In other words: the standard is promising, but so far largely unadopted on the provider side.
That doesn't mean it's worthless — it doesn't hurt, and adoption could change. But it's a low-risk addition, not a shortcut to visibility.
What actually drives AI visibility
Prioritize what's documented to work, and add llms.txt at the top of your list if you have time to spare:
- Technical accessibility and indexability for AI crawlers.
- Chunk-friendly content where each paragraph answers one question precisely.
- Clear entity authority and structured data.
- Mentions on third-party sources the models trust.
This is the core of AEO and GEO. Want to know how visible you are in AI answers today? Start with a free AI visibility check.
Frequently asked questions
What is llms.txt?
What's the difference between llms.txt and robots.txt?
Does llms.txt work? Do AI models actually use it?
Should I create an llms.txt file?
Definitions used in this article
- llms.txt
- llms.txt is a proposed standard where a markdown file at /llms.txt points large language models to the most important and most readable content on a website. It was proposed by Answer.AI in September 2024.
- robots.txt
- robots.txt is a file at the root of a website that tells crawlers which parts they're allowed to fetch. Unlike llms.txt, it controls access rather than which content is highlighted.
- AI crawler
- An AI crawler is a bot that fetches web content on behalf of an AI service — either to train models or to retrieve real-time sources when the model answers.