Cloudflare Targets Mixed Crawlers as UK Adds Google AI Controls

Cloudflare announced on July 1 that it would update its managed robots.txt service on September 15. For sites using the service, Cloudflare says crawlers that combine search and generative-AI purposes will be disallowed by default on pages that contain advertisements, unless the crawler separates those functions or the site owner changes the policy.

Cloudflare framed the change as a response to mixed-purpose crawlers. Its announcement said the largest search engine had roughly twice as much information as the next-largest crawler, creating a practical disadvantage for publishers that depend on search referrals. Business Standard separately attributed sharper figures to Prince: Google crawled Cloudflare customer sites 3.2 times more than Microsoft and 4.8 times more than OpenAI in the preceding six months. Those measurements come from different sources and should not be treated as the same statistic.

The policy landscape moved beyond a proposal on June 3. The UK's Competition and Markets Authority imposed a publisher conduct requirement on Google Search. For eligible UK publishers, the measure requires controls to exclude content from generative-AI search features while retaining eligibility for ordinary search results. It also covers source attribution and a separate control over use of publisher content for fine-tuning Google's generative-AI models.

Google's documentation distinguishes these controls. Google-Extended governs whether crawled content may help improve Gemini models or provide grounding for Gemini apps, but it does not affect inclusion or ranking in Google Search. The CMA-specific search control addresses AI Overviews and AI Mode while preserving eligibility for other search features.

Cloudflare's September default and the CMA requirement tackle adjacent problems through different mechanisms: one changes site-side crawler permissions, while the other imposes conduct obligations on Google for eligible publishers in the UK. For publishers and platform teams, the operational task is to map each control to the use it actually governs rather than assuming one robots token covers search, model training, and AI-generated search answers alike.