Policy & Ethicsweb crawlingcontent licensinggoogleregulation
UK CMA Forces Google Separate AI Crawlers
9.8
The UK’s Competition and Markets Authority has forced Google to separate its AI training crawler from its standard search crawler, a change implemented this month and described in Cloudflare’s analysis. Google now exposes a distinct Google-Extended agent that sites can block for AI training while preserving Google Search indexing. The move grants publishers new control over data licensing and may reshape global AI training practices.
Key Points
- 1Separates Google-Extended crawler from standard Googlebot, enabling AI-training-specific blocking for website owners
- 2Addresses bundling concerns by preventing forced content access, following CMA investigation into Google’s market dominance
- 3Enables publishers to retain search visibility while denying AI training, improving licensing negotiation leverage
Scoring Rationale
High regulatory novelty and broad industry impact; limited by incomplete public CMA report details and enforcement uncertainty.
Sources
Public references used for this report.
Practice with real Ad Tech data
90 SQL & Python problems · 15 industry datasets
Used by DS/ML engineers at top companies
Active Search Campaigns by BudgetEasyHigh CPC Clicks & Poor Landing PagesMediumCampaign ROAS by Attribution ModelHard
250 free problems · No credit card
See all Ad Tech problems
