llms.txt is a markdown file at your site root that gives AI systems a curated, token-efficient map of your most important content. Here’s the honest status: some AI agents fetch it, no major engine has committed to using it, and most of what’s written about it is either hype or dismissal. Neither camp is doing the analysis.
I’ll do it here. What the file is, what the evidence actually shows, and where it should sit in your priority stack.
What Is llms.txt?
llms.txt was proposed by Jeremy Howard of Answer.AI in September 2024. The idea is simple. LLMs working with your site at inference time face two problems: context windows are finite, and your HTML is full of navigation chrome, scripts, and boilerplate that wastes tokens. So you publish a plain markdown file at /llms.txt that says, in machine-friendly terms: here’s who we are, here’s what matters on this site, here are direct links to the pages worth reading.
A typical file has a title, a short summary of the site or company, and organized link lists with one-line descriptions. Some sites also publish llms-full.txt, which inlines the full content of key pages into one large document an agent can ingest in a single fetch.
That’s the whole spec. No protocol handshake, no registration, no tooling required. You could write one in a text editor before lunch.
How Is llms.txt Different From robots.txt and Sitemaps?
They solve different problems, and conflating them is the most common mistake in coverage of this topic.
robots.txt is access control. It tells crawlers what they may and may not fetch. It’s a fence.
An XML sitemap is inventory. It lists every URL you want indexed, with no judgment about which ones matter. It’s a phone book.
llms.txt is curation. It tells a model which handful of pages best represent your company and what each one contains, formatted for a machine that pays by the token. It’s a briefing document. robots.txt manages crawl-time access; llms.txt is aimed at inference-time consumption, when a model or agent is actively trying to answer a question using your site.
None of the three replaces the others. If llms.txt matters at all, it matters as a third layer, not a substitute.
Does llms.txt Actually Work? What the Evidence Says
This is where I have to be straight with you, because almost nobody else is.
The adoption story on the publishing side is real. Developer-tools companies moved first. Anthropic publishes llms.txt files for its documentation. Mintlify rolled the format out across the docs sites it hosts, which put it on thousands of sites at once. If you spend time in developer documentation, llms.txt is close to table stakes.
The adoption story on the consumption side is much thinner. There is no public confirmation that OpenAI, Anthropic, Google, or Perplexity systematically fetch and use llms.txt for retrieval in their consumer products. Server logs at various companies show some AI-agent fetches of these files, which tells you the file gets read sometimes, by something. It does not tell you the file influences what ChatGPT says about you when a buyer asks who solves your problem.
Google’s John Mueller has publicly compared llms.txt to the old keywords meta tag: a signal sites want to send that engines may simply ignore. That comparison stings because it’s apt. The keywords tag failed precisely because it let sites describe themselves, unverified, and self-description is cheap. Any format that says “trust my summary of my own content” carries the same structural weakness.
So the honest scorecard reads like this. Publishing adoption: real and growing in developer ecosystems. Consumer-engine adoption: unconfirmed. Measured citation impact: nobody has credibly demonstrated one. If a vendor tells you llms.txt moved their AI visibility numbers, ask for the mechanism. They won’t have one.
Why Ship llms.txt Anyway?
Given all that, my position might surprise you: ship it. Three reasons.
The cost is near zero. This is an afternoon of work, most of which is thinking, not typing. There’s no build pipeline to maintain and no downside risk. A static file that engines ignore harms nothing. Very few bets in marketing offer optionality this cheap.
The exercise itself has value. Writing a good llms.txt forces you to answer questions most B2B teams have never answered crisply. What is this company, in one machine-readable paragraph? Which ten pages actually define us? What does each one say? That’s entity-definition work, and it pays off everywhere AI systems evaluate you, whether or not this particular file gets read. Half the companies I audit couldn’t produce a clean answer to “what are your ten most important pages” today.
If adoption lands, you’re early. The file format is trivial to consume, agentic browsing is growing, and the problem llms.txt addresses is real. Standards sometimes go from ignored to assumed quickly. If that happens, you spent an afternoon. If it doesn’t, you spent an afternoon.
Strategnik practices this. There’s a live file at strategnik.com/llms.txt: entity description, the core framework, the service pages that matter. It took less time to write than this post.
Why You Shouldn’t Rely on llms.txt
Here’s the failure mode I actually worry about, because I’ve watched it happen with every shiny AEO tactic: a team ships llms.txt, checks the “AI optimization” box, and stops.
llms.txt does nothing for the actual drivers of AI visibility. When an engine decides whether to cite you, it’s weighing entity authority: whether you exist as a recognized, consistent entity across the sources it trusts. It’s weighing citations, whether third parties reference you, because AI engines run on consensus, not self-description. And it’s weighing whether your content is answer-shaped, leading with direct extractable answers instead of burying them under throat-clearing intros.
A curated site map influences none of that. You can have a perfect llms.txt and be completely invisible in ChatGPT, because the model has no independent evidence you belong in the answer. The work that builds that evidence is slower and harder, which is exactly why teams reach for the file instead. I keep the full list of what actually moves citations in the GEO checklist; llms.txt is one line item near the bottom, and it’s near the bottom on purpose.
If you want a read on where you currently stand before deciding where to invest, run your site through the AI Visibility Grader. It measures the signals engines demonstrably use, not the ones we hope they will.
So Should You Ship llms.txt? The Verdict
Yes. This afternoon, ideally. Write the entity summary, curate the links, publish the file, and treat the thinking it forced as the real deliverable.
Then be equally clear about what you just did: you sent a signal that no engine has promised to receive. The visibility your buyers actually experience, whether ChatGPT names you when they ask who solves their problem, is built from entity authority, third-party citations, and answer-first content. That’s the actual LLM SEO discipline, and it doesn’t fit in an afternoon.
Ship the file. Spend your real effort on the work that moves citations. Any vendor selling you the reverse priority is selling the cheap half of the job.