What It Does
llms.txt is a community-proposed file format, published at /llms.txt on a website’s root, that provides a structured Markdown summary of the site’s content specifically formatted for large language model consumption. Proposed by Jeremy Howard of Answer.AI in September 2024, the format is explicitly analogous to robots.txt (access control) and sitemap.xml (content discovery) but serves a different purpose: it helps LLMs assemble useful context about a project without crawling the full site.
The file follows a simple structure: an H1 with the project name (required), an optional blockquote with a brief summary, optional sections with H2 headings, and bullet-point links to key resources — preferably to Markdown versions of content rather than HTML pages. The intent is to give agents a curated navigation map into documentation they might otherwise have to infer through multiple crawl requests.
A companion format, llms-full.txt, concatenates all documentation into a single file for models to consume in one shot — useful for smaller documentation sets that fit within a context window.
Key Features
- Simple Markdown format: Human-readable, no special tooling required to create or maintain.
- Token-efficient navigation: Points agents to the most important documentation sections rather than requiring full-site crawl.
- llms-full.txt companion: Optional full-content concatenation file for models that want everything in one request.
- Python tooling: The
llms_txtPython package (from AnswerDotAI) automates generation from existing documentation. - Wide grassroots adoption: Over 844K sites have implemented the file as of October 2025 per BuiltWith tracking, including Anthropic, Cloudflare, Stripe, and Cursor.
- Documentation generator plugins: Mintlify, Docusaurus, and other documentation platforms have added native llms.txt generation.
Use Cases
- Developer documentation sites: Add
/llms.txtto help AI agents find key API reference, quickstart guides, and SDK documentation without full-site crawl. - Open-source libraries: Provide a summary of library structure, changelog, and migration guides for agents assisting developers with the library.
- SaaS vendor portals: Surface API capabilities, rate limits, and authentication patterns in a single agent-readable entry point.
- Personal/portfolio sites: Give AI assistants a curated summary of your work and background for accurate citations.
Adoption Level Analysis
Small teams (<20 engineers): Low-effort to add — a single Markdown file. Worth adding to any documentation site as a no-cost bet on future agent adoption. Many documentation platforms (Mintlify) generate it automatically.
Medium orgs (20–200 engineers): Worth standardizing in documentation templates. Does not replace structured API documentation or MCP servers, but is a lightweight complement.
Enterprise (200+ engineers): Implement as part of broader AEO strategy. Do not treat as a substitute for MCP server endpoints on high-stakes APIs. Monitor server logs to assess whether AI platforms actually request the file.
Alternatives
| Alternative | Key Difference | Prefer when… |
|---|---|---|
| MCP server for docs | Structured, queryable API with schema and tool-calling | You need agents to query documentation programmatically, not just read it |
| sitemap.xml | Lists all pages for crawlers; no LLM-specific formatting | Standard web crawler discovery rather than agent-specific |
| AGENTS.md | Repo-level instructions for AI coding agents | Providing instructions to development agents rather than documenting a website |
| RAG over docs | Dynamic chunk retrieval based on query | Documentation is too large and dynamic for static files |
Evidence & Sources
- /llms.txt proposal — Jeremy Howard, Answer.AI (original post, September 2024)
- Is llms.txt Dead? — llms-txt.io analysis of adoption reality
- What is llms.txt? Breaking down the skepticism — Mintlify
- llms.txt GitHub repository — AnswerDotAI
Notes & Caveats
- No confirmed LLM provider adoption: As of April 2026, no major LLM provider (OpenAI, Google, Anthropic, Meta) has publicly confirmed they read llms.txt files at inference time. Google’s Gary Illyes explicitly stated Google does not use it and is not planning to. Engineers have noted that server logs show AI crawlers do not even check for the file consistently.
- SEO-anxiety-driven adoption: A significant portion of adoption is driven by SEO tools (Rank Math, SEMrush) flagging missing llms.txt as a site health issue, creating demand without evidence of value — a dynamic similar to early meta-keywords adoption.
- Confusion with “AI training” crawlers: llms.txt is for inference-time use, not training data. Many site operators conflate the two and may implement it while simultaneously blocking AI training crawlers in robots.txt — an incoherent combination.
- Google’s A2A protocol mention: Google included llms.txt in their Agent-to-Agent (A2A) protocol specification, suggesting some continued interest despite the public statements against it. This signal is ambiguous.
- Speculative investment: Low cost to implement, uncertain payoff. Teams with limited documentation engineering bandwidth should prioritize MCP server endpoints or structured API references over llms.txt.
- Historical parallel: The
keywordsmeta tag was widely adopted (>90% of sites) before search engines stopped using it. llms.txt faces the same risk: wide adoption without confirmed inference-time value.