llms.txt is a proposed plain-text file placed at your site’s root that gives AI systems a curated, structured guide to your most important content conceptually similar to robots.txt or sitemap.xml, but aimed at language models rather than crawlers.
The proposal originated in the developer community in 2024. The idea is reasonable: web pages contain navigation, ads, scripts and boilerplate that make extraction harder, so a clean markdown index of your key content would in principle help a model find and use the right pages.
The format is markdown a heading with your site name, a short summary, then curated sections of links with one-line descriptions.
Does it actually work?
There is no confirmed evidence that major AI providers read llms.txt, and no major provider has committed to supporting it.
This is the answer most content on this topic avoids giving directly, so it is worth stating plainly.
What is true?:
- It is a community proposal, not an adopted standard
- No major AI provider has publicly committed to reading it
- Some smaller tools and documentation platforms have adopted it
- Server logs generally do not show major AI crawlers requesting the file with any consistency\
What you will read elsewhere?: A great deal of content asserting it is essential, that it is “the new robots.txt,” or that implementing it will improve your AI visibility. Most of that content is published by companies selling implementation services, tools that generate the file, or agencies looking for a differentiator.
The honest position: The proposal addresses a real problem, it may be adopted, and there is currently no evidence it does anything. Both halves matter.
Should you implement it?
Yes, if you have an hour spare. No, if it is displacing work that demonstrably matters.
The case for implementing:
- It takes about an hour
- It does no harm nothing breaks, nothing is penalised
- If it becomes a standard, you are ready
- The exercise of curating your most important content is mildly useful in itself
The case against spending real time on it:
- No confirmed effect
- The hour is better spent on comparison content, extractable facts or third-party presence, all of which demonstrably matter
- It creates a false sense of having done something about AI visibility
Where it becomes a problem?:
when it appears as a significant deliverable on an agency invoice, or when a company implements it and concludes they have addressed AI visibility.
How to implement it:
One file, one hour, and you can stop thinking about it.
Create a file at `yourdomain.com/llms.txt`:
markdown
Xperno Digital
SEO and Generative Engine Optimization for B2B and SaaS companies.
AI visibility audits, comparison content architecture, and technical
SEO built for AI retrieval.
Core pages:
[What is GEO](https://example.com/what-is-geo): Complete guide to
Generative Engine Optimization for B2B companies
[Services](https://example.com/services): SEO and GEO services,
scope and approach
[Pricing](https://example.com/pricing): Engagement models and
what drives cost
Guides:
[AI visibility measurement](https://example.com/measure): How to
track citation rate and share of voice
[Comparison pages](https://example.com/comparison-pages): How to
build content that gets cite.
Optional:
[Blog](https://example.com/blog): Ongoing writing on AI search
The structure:
- Heading with your name
- Blockquote with a one- or two-sentence summary
- Sections grouping links
- Each link with a short description of what the page contains
- An Optional` section for lower-priority content.
Some implementations also add`llms-full.txt` the full markdown content of key pages rather than links. More work, same absence of evidence.
Platform notes: WordPress needs a plugin or a file at the web root. Web flow and HubSpot generally need it hosted as a static file. Custom builds can serve it from a route.
What people get wrong about it?
Four misconceptions worth correcting:
“It’s the new robots.txt.” robots.txt is a decades-old standard that all major crawlers honour. llms.txt is a proposal with no confirmed adoption. The comparison flatters it considerably.
“It controls what AI can access.” It does not. Access control is robots.txt. llms.txt, if read at all, is a suggestion about what is important not a permission system.
“It will improve your AI visibility.” No evidence supports this. What improves AI visibility is extractable content, comparative framing and third-party corroboration.
“You need a tool to generate it.” You need a text editor. Several services sell generation, which for a twenty-line markdown file is not a service worth paying for.
What actually affects AI visibility?
The unglamorous things, which is why speculative quick fixes get disproportionate attention.
- Effort Evidence it works.
- Comparison and alternatives content High Strong
- Third-party corroboration High Strong
- Extractable facts Medium Strong
- Entity consistency Low Good
- Schema markup Low Moderate removes ambiguity
- Crawler access Very low Strong where it applies
- llms.txt Very low None confirmed
Why speculative fixes attract attention?: They are cheap, fast, and feel like progress. The things that demonstrably work are slow, require organisational buy-in, and produce nothing visible for two months. Given the choice, people prefer the file.
A useful test for any AI visibility recommendation: Ask what evidence supports it and what the mechanism is. If the answer is “it makes sense that it would help,” treat it as speculation including this one.
How to tell if a proposal is overselling it?
Three signals, and this is where the post is practically useful:
- It appears as a significant line item.** A twenty-line markdown file should not be a meaningful proportion of an engagement. Its presence at that level tells you something about how the rest was scoped.
- It is described as essential or as a standard. It is neither.
- It appears alongside other speculative deliverables “AI-optimised meta tags,” “LLM-friendly formatting,” “prompt optimisation.” That cluster usually indicates a proposal built on plausible-sounding activity rather than on evidence.
What a defensible proposal looks like instead?: Diagnosis with actual screenshots from your category, comparison content, technical fixes, off-site outreach with a start date, and a stated measurement method. If llms.txt appears at all, it should be a footnote.
Will it be adopted?
Possibly: The problem it addresses is real, and adoption would cost providers little.
Arguments for adoption: It is cheap for providers to support, it improves extraction quality, and site owners have an incentive to maintain it accurately.
Arguments against: Providers already extract from HTML successfully, a self-declared index is trivially gameable, and adopting it creates an expectation of honouring it.
What to watch?: A public statement from OpenAI, Anthropic, Google or Perplexity. Until one of those exists, treat the file as insurance rather than as strategy.
And if adoption happens, you will have an hour of work to do at that point. There is no first-mover advantage in a file that nobody currently reads.
Frequently asked questions:
Do AI crawlers read llms.txt?
There is no confirmed evidence that major providers do. Some smaller tools and documentation platforms have adopted it.
Is llms.txt required for AI visibility?
No. What matters is extractable content, comparative framing and corroboration across independent sources.
Should I pay someone to create it?
No. It is a twenty-line markdown file you can write in an hour.
Is it the same as robots.txt?
No. robots.txt is an established standard controlling crawler access. llms.txt is a proposal suggesting which content is important, with no confirmed adoption.
What is llms-full.txt?
A variant containing the full markdown content of key pages rather than links. More work, same absence of evidence.
Could implementing it hurt me?
No. It does no harm. The risk is opportunity cost time and budget spent here rather than on work that demonstrably matters.