TL;DR – Google’s Search Advocate, John Mueller, believes llms.txt files are unlikely to help AI systems decide which websites to surface for a specific query. Since llms.txt is self-reported information created by website owners, it doesn’t provide a reliable way for LLMs to distinguish one site from another. Instead, Mueller says traditional web signals such as HTML content and internal linking remain essential for discovery, while llms.txt may only be useful once an AI agent has already landed on a website.
John Mueller Explains Why llms.txt Has Limited Value for AI Discovery
During a recent episode of the Search Off the Record podcast, Google’s Search Relations team discussed whether publishers should create Markdown-based content for AI systems. Both John Mueller and Martin Splitt agreed that HTML remains the primary format for web crawling, indexing, and content discovery.
The discussion eventually shifted to llms.txt, a proposed file designed to provide AI systems with structured information about a website.
According to Mueller, the concept faces a fundamental challenge: AI systems cannot rely on self-promotional files to determine which website deserves visibility for a particular query.
He explained that every website could claim to be the best resource, making it impossible for an AI model to use those claims as a meaningful ranking signal. As a result, llms.txt offers little value when it comes to helping LLMs choose between competing sources.
Why AI Systems Can’t Differentiate Websites Using llms.txt
Mueller’s argument centers on differentiation.
If every website publishes a file stating that its content is authoritative, comprehensive, or highly relevant, AI systems gain no objective way to compare those claims. An LLM still needs external signals to determine which source best answers a user’s question.
In practice, websites would continue competing based on factors such as:
- Content quality and relevance
- Website structure
- Internal linking
- External references and citations
- User experience and accessibility
- Other trust and authority signals
Because llms.txt is created by the website itself, it lacks the independent validation needed to influence discovery decisions.
What Mueller Means by “By Design”
Mueller stated that LLM systems “by design” cannot trust llms.txt for discovery purposes. While he did not elaborate further, the statement can be interpreted in two ways.
1. Architectural Limitation
One interpretation is that AI systems are designed to evaluate actual web content rather than self-declared descriptions of that content. In this model, llms.txt simply isn’t considered a reliable discovery signal.
2. Signal Saturation Problem
Another interpretation relates to the history of search ranking signals.
A useful comparison is the now-obsolete meta keywords tag. Once website owners realized it could influence visibility, many began stuffing it with keywords. As adoption increased, the signal lost value because search engines could no longer distinguish quality websites from low-quality ones.
The same issue could affect llms.txt. If every website provides similar self-promotional information, the file ceases to offer meaningful differentiation.
Regardless of which interpretation is correct, the outcome remains the same: llms.txt is unlikely to help AI systems decide which websites to surface.
Where llms.txt Could Still Be Useful
Although Mueller dismissed llms.txt as a discovery tool, he did identify a scenario where it could provide value.
If an AI agent has already arrived on a website, llms.txt could help guide navigation and task completion.
For example, an AI shopping assistant visiting a photography marketplace could use the file to understand:
- How to browse products
- How to complete a purchase
- Where important resources are located
- Which pages contain specific information
In this context, llms.txt functions more like a website guide or directory rather than a ranking signal.
The distinction is important:
- Discovery: Choosing which website to visit.
- Navigation: Understanding how to interact with a website after arriving.
Mueller believes llms.txt may have value for navigation but not for discovery.
The Debate Around llms.txt Continues
The skepticism surrounding llms.txt is not new.
Mueller has previously criticized the idea of creating Markdown versions of websites specifically for AI systems and has compared llms.txt to the old meta keywords tag.
Others in the SEO industry have raised similar concerns. Critics argue that the format is inherently difficult to trust because website owners can add whatever claims they want without independent verification.
Industry research has also questioned its practical impact. Studies examining hundreds of thousands of domains have found little evidence that adopting llms.txt increases the likelihood of being cited in AI-generated responses.
However, Mueller’s latest comments introduce a broader concern. The issue is not just potential manipulation—it is the absence of a mechanism that helps AI systems objectively select one website over another.
Why This Matters for Website Owners
For publishers and SEO professionals looking to improve visibility in AI-powered search experiences, Mueller’s comments suggest that traditional web fundamentals remain the priority.
Rather than relying on llms.txt as a shortcut to AI visibility, website owners should continue focusing on:
- High-quality, authoritative content
- Strong internal linking structures
- Clear HTML architecture
- Crawlability and accessibility
- Topical expertise and trustworthiness
Even a perfectly written llms.txt file cannot instruct an AI model to favor one website over competing sources.
What’s Next for AI Website Standards?
Mueller acknowledged that standards for AI agents and automated web interactions are still evolving.
He referenced emerging initiatives such as WebMCP and other experimental approaches aimed at improving how AI systems interact with websites.
However, no universal standard has emerged yet. According to Mueller, it could take anywhere from six months to a year—or even longer—before the industry settles on a preferred framework for agent-based website interactions.
Until then, the discovery layer of the web is expected to continue relying on established technologies such as HTML, structured site architecture, and internal linking.
FAQs
-
What is llms.txt?
llms.txt is a proposed file that provides structured information about a website for AI systems and autonomous agents. It is intended to help AI tools better understand a site's content and navigation.
-
Does llms.txt improve AI search rankings?
According to Google’s John Mueller, llms.txt does not help AI systems decide which websites to surface for a query because it contains self-reported information that cannot reliably differentiate one website from another.
-
Can AI systems trust information in llms.txt?
Mueller suggests they cannot use it as a discovery signal because website owners can make any claims they want. Without independent validation, the information lacks ranking value.
-
Is llms.txt completely useless?
Not necessarily. Mueller believes it may be useful for AI agents that have already reached a website and need guidance on navigation, purchases, or task completion.
-
What should website owners focus on instead?
Website owners should prioritize high-quality content, HTML optimization, internal linking, technical SEO, and overall site authority rather than relying on llms.txt for visibility.
-
Could llms.txt become important in the future?
Possibly. AI navigation standards are still evolving, and llms.txt or similar formats may find a role in helping AI agents interact with websites. However, there is currently no evidence that it improves discovery or rankings in AI systems.