llms.txt

llms.txt is an emerging convention in which a website explains its own structure, core content and licensing terms to models in Markdown format.

llms.txt is a site description file proposed in 2024, placed in the domain root directory, listing a site's most important pages, summaries and purposes in Markdown format, for large language model crawlers to understand the site structure.

llms.txt was formally proposed by Jeremy Howard (founder of fast.ai) in 2024; websites place a Markdown-format file at the path /.well-known/llms.txt to describe the site's core content.

llms.txt is an emerging website standard proposal, similar to robots.txt but specifically targeting AI crawlers, written in Markdown format.

There is currently no public benchmark showing that llms.txt alone can improve citation rates.

llms.txt has a different positioning from robots.txt (which controls crawler access permissions) and sitemap.xml (which lists URLs); its goal is to let models quickly understand a site's structure and core content.

llms.txt draws on the concept of robots.txt and aims to provide large language models with a clear and concise content index

claude-seo includes Agent Readiness-related capabilities, specifically involving Lighthouse Agentic Browsing, WebMCP, and llms.txt.

Common supplementary practices in the industry to improve Agent readability include schema.org Product/Offer markup, llms.txt, and APIs or MCP endpoints provided separately for Agents.

The llms.txt V2 revision adds standard link relations, specifically rel="alternate" type="text/markdown" and rel="describedby".

OpenAI, Anthropic and Gemini already publish their developer documentation as llms.txt.

AI search bots rarely fetch the llms.txt file, and Google said in 2025 that it does not support the community proposal.

Google recommends ignoring llms.txt, Chrome Lighthouse audits it, and four independent studies show it has no effect on AI citations.

llms.txt cannot solve the rendering problem: if the HTML is empty, AI bots following the links still cannot obtain the content.

To let AI read the website, the priorities in order are: ensure robots.txt does not block crawlers and that the host/firewall does not intercept them, maintain a complete sitemap and declare it in robots.txt, ensure each page's pure HTML does not depend on script loading, and treat llms.txt as a courtesy to users and tools rather than a step toward gaining exposure.

OpenAI, Anthropic, and Perplexity each provide documentation for their own crawlers, all pointing to robots.txt as the control entry point, and none declares that its crawler reads a website's llms.txt; but all three companies published llms.txt for their own developer documentation.

Jeremy Howard proposed the llms.txt proposal at llmstxt.org on September 3, 2024, and the proposal's original wording states that it is primarily intended for inference rather than training.

When building human-in-the-loop (HITL) agent systems, the request side can block most spam requests through an llms.txt endpoint with proof of work (PoW), but the reply side is the truly tricky part.

llms.txt and OpenAPI are complementary and together form the practical direction of "documentation as interface."

Vendors such as OpenAI and Anthropic have not publicly committed to parsing the llms.txt file, so its actual crawling effect is limited, and it is currently more of a community-level convention.