llms.txt — from curiosity to working convention

What changed. The llms.txt convention — a root-level Markdown file that tells AI crawlers what a site is about and which pages are authoritative — has steadily moved from curiosity to common practice across documentation sites, tools, and content publishers.

Why it matters. It remains a convention, not a universally honoured standard: not every crawler reads it today. But the economics are lopsided. It costs almost nothing to publish, it can't hurt, and it positions you correctly for where AI-mediated discovery is heading. Pairing it with a clean machine-readable substrate is the point.

What to do. If you haven't already, publish an llms.txt listing your most important pages with one-line descriptions, reference it from robots.txt, and add a fuller llms-full.txt if your library justifies it. This very site serves both — verify them at /llms.txt.

Tracking: llms.txt, llms-full.txt, AI crawler directives.

← All Radar updates RSS