Publishers Reconsider Blocking Google as AI Search Changes the Traffic Bargain
Digiday reported on July 30 that publishers are again discussing whether to block Google's crawlers as AI-generated answers reduce the perceived value of search referrals. A broad Googlebot block can also remove content from ordinary Search and Google News, while a UK requirement imposed on June 3 is intended to provide separate controls over publishers' content in generative AI.
Publishers are again discussing whether to block Google's crawlers as AI-generated search answers keep more activity on Google's own surfaces, Digiday reported on July 30. The debate reflects a weakening version of the traditional exchange: publishers allow indexing and receive referral traffic, but AI answers can use publisher material without always producing a visit.
Digiday reported that publishers still see a full Google block as a high-cost step because conventional search referrals remain commercially important. Its report did not identify a publisher that had fully blocked Google at publication.
A broad block also affects ordinary discovery
Google's documentation says blocking Googlebot through robots.txt can prevent content from appearing in both Google Search and Google News. Publishers can target Googlebot-News separately for News, or use page-level indexing controls, but those tools do not turn a broad Googlebot block into a narrow generative-AI restriction.
That coupling explains why a threat to block Google can function as leverage without becoming an immediate operating plan. A publisher that cuts off the crawler used for ordinary search risks losing discovery, audience, advertising impressions, and subscription opportunities before an alternative distribution channel replaces them.
The UK rule is imposed, but implementation is still unfolding
The UK Competition and Markets Authority imposed a publisher conduct requirement on Google on June 3; it is no longer merely a proposal. The requirement directs Google to provide effective controls over the use of publisher search content in generative AI, explain how that content is used, report engagement metrics, and ensure clear attribution and access links.
The CMA said Google has nine months to implement all changes and must publish compliance reports. That means the regulatory direction is settled in the UK, while the practical controls and the evidence publishers receive from them are still being implemented and monitored.
For publisher data and SEO teams, the central distinction is between blocking discovery and controlling AI-specific uses. Decisions need separate measurements for ordinary search referrals, AI-feature citations, clickthroughs, and revenue. Without that visibility, a crawler block can be easy to announce but difficult to price.
Key Points
- 1Digiday reports that publishers are revisiting Google crawler blocks, but conventional search referrals still make a full block commercially risky.
- 2Google documents that blocking Googlebot can remove content from Search and Google News, so a broad block is not an AI-only restriction.
- 3The UK CMA imposed its publisher conduct requirement on June 3; implementation and compliance reporting are still in progress.
Scoring Rationale
The story concerns publisher access controls for search and generative AI, with direct implications for AI search, SEO, attribution, and content economics. The UK requirement is now imposed, but implementation is ongoing and the July 30 publisher discussion is not itself a new Google product change.
Sources
Public references used for this report.
Practice interview problems based on real data
1,625 SQL & Python problems across 15 industry datasets — the exact type of data you work with.
Try 250 free problems
