llms.txt and AI Crawlers: What Actually Matters

A GEO guide by Apurv Singh, HQ Digital

llms.txt is a proposed file that hands AI models a tidy map of your best content. It is a reasonable idea. It is also not, today, something the major AI engines have confirmed they use to decide who gets cited. So treat it as a cheap hedge, not a strategy, and put your energy where it actually moves the needle.

The thing that genuinely controls whether AI can read and cite your site is older and less exciting: which crawlers you allow, and whether your site is clean enough for them to use. Let me separate the signal from the noise.

It was proposed in 2024 as a markdown file that summarises and links your most important pages, so a model does not have to wade through your navigation, banners and footers. Good intention. But after more than a year, no major answer engine has confirmed it uses the file to choose citations. Google has said on the record that it does not use it, and large studies across hundreds of thousands of domains have found no link between having an llms.txt and being cited. The crawlers barely request it.

Where it does earn its keep is developer tooling. AI coding assistants and documentation agents use it as a routing layer. So if you run a docs-heavy or developer product, it is genuinely useful. For everyone else, it is a thirty-minute, forward-compatible hedge. Yoast or Rank Math can generate one. Just do not build duplicate markdown copies of every page, which creates a duplicate-content mess, and do not expect a single citation to come from it.

Forget the magic file. The real lever is whether you allow the crawlers that power AI answers. If you block OAI-SearchBot, Claude-SearchBot or PerplexityBot, you have quietly opted out of being cited on those platforms. For GEO, you allow them. The training crawlers, GPTBot and ClaudeBot, are a separate question about whether your content trains future models. Blocking those does not help your citations and may simply remove you from the conversation.

This is the Core pillar of the CITE framework in its most literal form. A crawlable, fast, well-structured site that lets the right bots in beats any speculative file. See how structure decides liftability ›