Google Engineers Expose Limits of LLMs.txt Proposal
Google's John Mueller said on the company's "Search Off The Record" podcast (episode 111) that the creator of the proposed LLMs.txt standard told him the file was never meant to help sites get discovered by search engines or AI systems, contradicting how many publishers are using it, Search Engine Journal reported June 18, 2026 (updated June 23). Mueller said Discovery, the crawler step where a search engine first learns a URL exists, is not part of the LLMs.txt proposal and remains tied to standard HTML pages, meaning an LLMs.txt file alone cannot make a site more visible to AI systems or search engines. He suggested a different standard, WebMCP, is a better fit for helping AI agents interact with a site once they're already there, such as completing a purchase.
For publishers and SEO teams, the useful signal here isn't just "LLMs.txt doesn't do what people think," it's that Mueller is drawing a sharper line than the industry usually does between two different jobs: getting found at all (Discovery) versus being trusted or acted on once an AI system is already looking at your content. Conflating those two is the actual mistake to fix.
What happened
Search Engine Journal reports that Google's John Mueller and Martin Splitt discussed the proposed LLMs.txt standard and markdown on the company's "Search Off The Record" podcast, episode 111. Mueller said he had spoken with one of the people who created the LLMs.txt proposal, who told him the file was never intended to make a site's content easier for search engines or LLM systems to discover, contrary to how many site owners are using it. Mueller also said LLMs.txt is inherently untrustworthy as a ranking or citation signal because it's simply a site owner asserting what their own content is about, which may not match the actual HTML.
Technical context
Mueller framed Discovery, finding that a URL exists and queuing it for crawling, as the first step in a five-part pipeline (Discovery, Crawling, Indexing, Ranking, Serving) that determines whether a page shows up anywhere, in search or in an LLM's citations. Because LLMs.txt isn't part of that pipeline, per Mueller's account of the creator's own framing, publishers spending time generating it for visibility are optimizing for something the standard was never built to deliver. Mueller pointed instead to WebMCP as a better fit for agentic use cases, letting an AI agent already on a site figure out how to complete a task like buying a product, rather than helping it find the site in the first place.
For practitioners
The practical implication is that content discovery for both search engines and LLMs still runs through ordinary, crawlable HTML pages, not through a supplementary LLMs.txt file. Practitioners who have invested resources in generating and maintaining LLMs.txt for AI-visibility purposes should redirect that effort toward standard crawlability and technical SEO fundamentals, and watch WebMCP or similar agentic-interaction standards if their site needs to support AI agents completing tasks, not just being found.
What to watch
Whether other search engines or AI platforms publish similar guidance on LLMs.txt's actual scope; whether the competing draft standards Mueller mentioned, LLMs.txt, WebMCP, and other JSON/well-known file proposals, converge on one agentic-interaction standard over the next six to twelve months, as Mueller suggested; and whether publisher behavior shifts away from LLMs.txt generation toward core SEO investment.
Key Points
- 1Google's John Mueller said the creator of the LLMs.txt proposal told him it was never meant to help sites get discovered by search engines or AI systems.
- 2Discovery, the crawler step that determines whether a page is found at all, is not part of the LLMs.txt proposal and remains tied to standard HTML.
- 3Mueller pointed to WebMCP as a better-suited standard for AI agents completing tasks on a site, rather than for initial discovery or visibility.
Scoring Rationale
Directly sourced from Google's own official podcast (Search Off The Record ep. 111) via a detailed SEJ transcript, giving high confidence despite single-article coverage. Solid, practitioner-relevant clarification of a widely misunderstood standard; score unchanged as already well-calibrated.
Sources
Public references used for this report.
Practice with real Ad Tech data
90 SQL & Python problems · 15 industry datasets
250 free problems · No credit card
See all Ad Tech problems
