New /llms.txt Standard Proposed to Help AI Models Navigate Website Content
Key Takeaways
- ▸The /llms.txt proposal standardizes how websites can provide AI-friendly content through markdown files in a /llms.txt format at the root directory
- ▸The standard includes creating .md versions of web pages by appending .md to URLs, making content more accessible to language models
- ▸FastHTML, nbdev, and all Answer.AI and fast.ai projects have already adopted the standard, demonstrating practical implementation
Summary
Jeremy Howard, founder of Answer.AI and fast.ai, has introduced a standardized proposal for website owners to create /llms.txt files that help large language models access and understand website content more efficiently. The proposal addresses a critical challenge in AI development: LLMs struggle to process entire websites due to context window limitations and the complexity of converting HTML pages with navigation, ads, and JavaScript into usable plain text. The /llms.txt file would provide LLM-friendly markdown content with brief background information, guidance, and links to detailed documentation.
The proposal includes two key components: a root-level /llms.txt file containing structured information about the website, and .md versions of individual pages accessible by appending .md to any URL. The FastHTML project has already implemented this standard, demonstrating how the approach can streamline AI access to documentation. The format uses markdown rather than traditional structured formats like XML, specifically because it's designed to be easily read by both language models and humans.
All nbdev projects, including Answer.AI and fast.ai software projects, now generate .md versions of pages by default and have regenerated their documentation with this feature. The standard is intentionally flexible, allowing different applications to process the llms.txt file according to their needs—from helping developers navigate software documentation to enabling businesses to outline their structure or making legislation more accessible to stakeholders.
- The format uses markdown specifically because it's readable by both LLMs and humans, while still being parseable by traditional programming tools
- Applications range from software documentation navigation to business structure outlines, legislative breakdown, and e-commerce product information
Editorial Opinion
This proposal represents a pragmatic solution to a growing challenge in the AI ecosystem—how to make web content more accessible to language models without sacrificing human readability. By choosing markdown over more rigid formats like XML, Howard demonstrates an understanding that AI tooling needs to remain flexible and human-centric. The rapid adoption across Answer.AI's ecosystem suggests this could become a de facto standard, potentially influencing how other organizations structure their web content for the AI era.
