Discoverability & Structure

What is llms.txt and why your site needs one

By · · · 6 min read

The problem llms.txt solves

When an AI assistant answers a question using your website, it doesn't experience your site the way a person does. It doesn't see your navigation, your hero image, or your carefully art-directed homepage. It ingests text — and it has a limited budget for how much it will read. If that budget gets spent on cookie banners, nav menus, and boilerplate, the substance of your site may never get read at all.

llms.txt is a direct answer to that problem: a single, human-curated file that says "here is what matters on this site, and here is where to find it."

What goes in it

The format is deliberately simple — it's just Markdown. A minimal llms.txt has:

  • An # H1 with your site or organisation name
  • A > blockquote one-liner summarising what the site is
  • Optional sections (## Core pages, ## Guides) listing important URLs with a short description after each
# Acme Analytics

> Self-serve product analytics for B2B SaaS teams.

## Core pages
- [Pricing](https://acme.com/pricing): Plans, limits, and what each tier includes.
- [Docs](https://acme.com/docs): API reference and integration guides.

That's it. The companion file llms-full.txt carries the full text content for engines that want to ingest everything in one request.

Why it matters now

AI-mediated discovery is no longer a fringe channel. When a marketing manager's prospects ask an assistant "which tools do X?", the assistant assembles its answer from whatever it can read and trust. A clean llms.txt:

  • Curates — you decide what's authoritative rather than leaving the model to infer it.
  • Compresses — the model spends its attention budget on substance, not chrome.
  • Signals effort — like a good sitemap, it's a quiet credibility marker.

The honest caveat

llms.txt is a convention with growing adoption, not a universally honoured standard. Not every crawler reads it today. But it costs almost nothing to publish, it can't hurt, and it positions you correctly for where discovery is heading. This very site serves one at /llms.txt — view it, then publish your own.

What to do this week

  1. Draft an llms.txt listing your 5–10 most important pages with one-line descriptions.
  2. Place it at https://yourdomain.com/llms.txt.
  3. Reference it from robots.txt as a comment so crawlers can find it.
  4. Add a fuller llms-full.txt if your content library justifies it.