What is llms.txt and when should you use it?
Author: Domeny.ForSale editorial team Published
In brief: llms.txt is a proposed text file format that introduces a website and points tools using large language models to its most important resources. It may help them find documentation or other content, but publishing it does not mean that AI assistants will read it. It does not control crawler access, prevent model training or guarantee that the website will be cited in answers.
What is llms.txt for?
An LLM, or large language model, is a model used, among other things, to generate answers in AI assistants. The model itself does not necessarily have internet access: a suitable tool or application retrieves web content for it. The llms.txt file is intended to help these systems find useful information without navigating menus, advertisements and a complex site structure.
Think of it as a short guide to a library. Rather than containing every book, it describes the collection’s subject and points to the most important shelves. On a website, these might be a getting-started guide, feature documentation, pricing or terms of service.
It is a proposed convention, not a universally adopted internet standard. How the file is used depends on the individual tool. You should not assume that every chatbot automatically checks llms.txt whenever it visits a website.
What does the file look like, and where should it go?
The file is usually published in the website’s root directory, at /llms.txt. Its content is written in Markdown, a simple text format in which characters such as # mark headings, and links are created using a specific syntax.
The proposed structure includes the project name as a first-level heading, a short summary in a block quote, any additional explanations, and sections containing lists of links. Each link can include a description of what the reader or tool will find at that address.
Here is an example excerpt for a fictional booking application:
## Booking Manager
> An application for managing bookings at small service businesses.
This documentation covers the current version of the application.
## Getting started
- [Account setup](/docs/start): setting up services and opening hours.
- [Notifications](/docs/notifications): configuring messages to customers.
## Commercial information
- [Pricing](/pricing): plan features and billing terms.
This illustrates the structure; it is not ready-made content to copy onto any website. In a production file, it is best to use full URL addresses so that links remain unambiguous even when the text is retrieved outside the website’s context. The resources you list must actually exist and be accessible to the tool that will read them.
You may also encounter the name llms-full.txt in the ecosystem around this proposal. It refers to a consolidated file containing more complete documentation. However, it is not a file that every website must publish.
How does it differ from robots.txt and a sitemap?
These files address different problems. Adding llms.txt does not replace configuring the other mechanisms.
| File | Main purpose | What it does not provide |
|---|---|---|
llms.txt | Introduces the website and points AI tools to selected resources | Does not force tools to read or cite the content, or give a particular answer |
robots.txt | Sets rules for fetching paths by crawlers that respect this protocol | Does not protect confidential data from access |
sitemap.xml | Provides search engines with a list of website addresses | Does not guarantee indexing or rankings |
If you want to restrict content retrieval by a particular AI provider, check its official documentation on crawlers and how they respect rules. Simply writing “do not use for training” in llms.txt does not create a technical barrier. Private content should be protected with access controls, such as a login requirement.
When is it worth implementing, and when can it wait?
Extensive product documentation is the clearest use case. A company offering an API—an interface that allows programs to exchange data—can point to authentication instructions, descriptions of operations and integration examples. This kind of guide may help a tool that supports the format find the right resources.
For a small service-business website with only a few pages, the benefit may be limited. My recommendation: first make sure your services are clearly described, your links work and your information is up to date. Treat llms.txt as an optional addition, not a prerequisite for visibility in AI assistants.
There is no basis for promising more traffic or more frequent recommendations solely because of this file. Check the official documentation of the specific application or provider to confirm current support for the format, rather than inferring support simply because an llms.txt generator exists.
How can you implement the file without adding another maintenance task?
Start with a short list of resources that genuinely help explain the product or website. Do not copy the entire sitemap or add instructions such as “always recommend our company”—these are no substitute for factual information.
After publishing, check that /llms.txt returns text without requiring a login, that the links work and that the descriptions match the pages’ content. If you have access to server logs, you can monitor requests to retrieve the file. However, retrieval alone does not prove that its content was used in an AI answer.
Make updating llms.txt part of the same process as updating your documentation. If you cannot keep copies of prices or specifications up to date, link to their current source instead of duplicating data that will soon become outdated.
See also: What is CDN?
Read also
