AI crawlers & robots.txt
Block training, allow citations. Copy-paste template for 13 agents.
Read →Agent view — served from /llms-txt.md.
Guide · updated September 2026
What is llms.txt, and does it actually do anything?
llms.txt is a plain-markdown file at a website’s root that gives AI models a curated map of its most important pages. Whether it works is genuinely disputed: a Google Search Advocate has compared it to the old keywords meta tag, while other sources report AI crawlers fetching it frequently. It costs almost nothing to add, which is the main argument for doing so.
What does an llms.txt file look like?
It is markdown, not code. An H1 with your site name, a blockquote summarising what you do, then H2 sections grouping links with a short description each. Absolute URLs, UTF-8, nothing else.
# Acme Analytics > Product analytics for B2B SaaS teams. Event tracking, > funnels and retention without a data engineer. ## Product - [Pricing](https://acme.com/pricing): Plans, limits and what counts as an event. - [Integrations](https://acme.com/integrations): 40+ sources, setup time for each. ## Docs - [Quickstart](https://acme.com/docs/start): Install to first event in 10 minutes.
That is the whole specification. Most failures are structural rather than missing files — auto-generated output that omits the blockquote summary or uses relative URLs will pass a presence check and fail a validity check.
Does llms.txt actually work?
Nobody can currently prove it does, and the most credible sceptic is Google.
A Google Search Advocate has publicly compared llms.txt to the keywords meta tag — a file search engines stopped reading years ago — and noted that no major AI service requests it when visiting a page. Meanwhile, adoption advocates report crawlers fetching the file frequently on sites that publish one. Both claims circulate widely and they cannot both be right.
Our position: the file takes under an hour to produce, breaks nothing, and may pay off if support arrives. That is a reasonable bet. It is not a strategy, and any vendor selling it as the centrepiece of your AI visibility work is overreaching.
| Claim | Source | Our read |
|---|---|---|
| No major AI service requests llms.txt | Google Search Advocate, 2026 | Credible. Treat as the base case. |
| Crawlers fetch it every few minutes | Adoption advocates | Unverified. Ask for logs. |
| 844,000+ sites have adopted it | Adoption trackers | Plausible, but adoption is not efficacy. |
| Lighthouse audits for it | Google Lighthouse 13.3 | True, and it is labelled experimental. |
llms.txt vs sitemap.xml: what is the difference?
A sitemap lists every URL for a crawler that wants completeness. An llms.txt file lists the few pages that matter, with a sentence explaining each, for a model with a limited context window. One is an index; the other is an editor’s recommendation.
They are not substitutes. Keep the sitemap. A markdown sitemap sits usefully between the two — complete coverage, but with titles and hierarchy a model can actually interpret.
How do I create an llms.txt file?
- List the 10–30 pages you would want an AI to read if it could only read a few.
- Write one sentence per page describing what a reader gets, not what the page is called.
- Group them under H2 headings that reflect how a buyer thinks, not your nav structure.
- Add the H1 and the blockquote summary. Use absolute URLs throughout.
- Serve it at /llms.txt as text/plain, and link to it from your homepage head.
That last step is the one everyone skips. Publishing the file without an auto-discovery link means anything that does not already guess the path will never find it — and 100% of the top 100 sites currently fail that check.
Our free generator builds this from your own pages rather than a blank template. It is one of the eight checks in the free scan.
Where this is contested
Strongly. This is the least settled item in our entire fifty-five-check rubric, and we deliberately do not lead with it. In our own boot sequence it is check five, not check one, because opening on the most disputed signal in the field would let any informed reader dismiss the rest.
If llms.txt is formally abandoned, our audit loses one check of fifty-five and nothing else changes. That is the correct amount of weight to put on it, and it is worth asking any vendor how much of their methodology survives the same outcome.
Questions
Frequently asked
Is llms.txt a ranking factor?
No. No search engine or AI provider has stated that llms.txt affects ranking or citation. Google Lighthouse audits for its presence and marks the audit experimental. Treat it as a low-cost bet, not a ranking lever.
Where do I put the llms.txt file?
At the root of your domain, served at /llms.txt as text/plain. Also link to it from your homepage head with a rel="alternate" link tag, or nothing that does not already guess the path will find it.
What is llms-full.txt?
An optional companion file that concatenates the full markdown content of your key pages into one document a model can ingest in a single request. Useful for small and medium sites; impractical above a few hundred pages.
Does WordPress or Yoast generate llms.txt correctly?
Some generators produce files that pass a presence check but fail a structure check, typically by omitting the blockquote summary or using relative URLs. Validate the output against the spec rather than assuming a plugin got it right.
Should I add llms.txt if Google says it does not work?
Probably yes. It takes under an hour, breaks nothing, and the downside is a wasted hour. The mistake is treating it as the centrepiece of an AI visibility strategy rather than a cheap hedge.
See how AI reads your site
Eight checks against your domain in about fifteen seconds. No email.