Tuning your website for AI with llms.txt

On the web, there are a few existing standards which are used by browsers and search engine web crawler ‘bots’ to help them find content and key files on your website.

You might be familiar with some of them; one of the earliest for instance, is the icon next to the site’s title in your browser tab or bookmarks. It's called the ‘favicon’ (favourite icon). In its most basic form, a file called favicon.ico is placed in your website, and browsers will automatically download it. You can see ours here: www.enigma-interactive.co.uk/favicon.ico

A few more of these have sprung up over the years:

• robots.txt – this is used to tell search engine bots what they should and should not index on your site
• sitemap.xml – takes this a step further and is used to give an overview of the site’s page structure, serving as a ‘map’ for search engine bots to be able to find the site’s content

With the recent increase in the way people are using AI LLMs (Large Language Models) such as ChatGPT, Gemini, Claude, Grok etc. to access online content, the question has been raised about how website owners can provide equivalent information to enable AI LLMs to best understand their content in a way that is tuned to their needs.

This has led to a new evolving standard – llms.txt – that aims to address the most common challenges that both the LLM companies and site owners face, such as:

• How do we make sure LLMs are able to efficiently comprehend the site’s content?
• Can we make sure the LLM has the most up-to-date and accurate information (as LLM training data is updated far less frequently than traditional search engines)?
• Can we give LLMs the ability to cite sources, and direct users to the origin (i.e., your site)?

The proposed approach is that a file – llms.txt – is maintained by the website that provides clear, unadorned information about the site and its content. It’s designed to be readable by humans and bots alike, adopting the popular Markdown format. Typically, it’s structured like this:

• Main title
• Description of the site
• List of site sub-sections, e.g., articles, products etc.
• Contact information
• Legal (privacy policy, terms and conditions etc.)
• ‘Optional’ section (links to pages which, while generally relevant, aren’t necessary to the direct comprehension of the site’s main purpose - think FAQs, related content and so on)

The purpose of the file is to be used at what is called inference time - that is, as additional context given to the LLM when talking about a particular subject.

It’s not (currently, at least) intended as a tool to use during LLM training. It’s important to understand that LLMs are trained over periods of weeks to months, and so their training data is always lagging behind current events. LLMs don’t typically know what’s currently happening in the world from their training data alone. It’s this inference context that gives LLMs the ability to take their training data, and then augment it with new information before they reply.

Amongst other things, llms.txt therefore allows site owners to make available more timely information such as news, pricing, stock etc. It also provides an additional layer of context and trust which may make it more likely that AIs cite your content.

This is still a very new and evolving standard, and there hasn’t been any official confirmation from the AI companies themselves that they use this file. However, we are seeing from our own clients' websites that several of the big players like OpenAI (ChatGPT), Perplexity and Amazon are all now regularly attempting to access it.

It’s worth noting that llms.txt isn’t a requirement to enable AI LLMs to access and use content from your site. They will do that anyway (it’s actually preventing them from doing this that’s the challenge – though that’s probably a topic for another day). llms.txt simply enables you as a site owner to provide better context and more timely information to the LLMs, which in turn could increase the likelihood of it being presented back to end users of those services.

Want to know more?

We’re currently consulting with our clients to provide the clearest advice on how this emerging standard should be handled for them, and have been developing tools that support the auto generation of llms.txt files from site content that aligns to the needs of their businesses. These tools will enable the file to kept up-to-date to ensure that their content gets the best chance of being accurately portrayed and cited.

If you’d like to find out more about this, or are interested in other ways that AI is providing opportunities and challenges for site owners, then please contact us.