Cloudflare and GoDaddy Shield Websites from AI Crawlers
Cloudflare and GoDaddy announced a partnership to give website operators new controls over AI crawlers from large AI firms, framing the effort as a fix for the web’s “broken business model” in the AI era. The collaboration aims to let site owners decide how automated AI agents access, index, or reuse their content — a direct response to large-scale scraping by major AI companies. For practitioners, this changes the negotiation around training data access, web-scale crawling ethics, and infrastructure-level bot management. Expect product integrations that surface crawl policies and enforcement at the edge, shifting some control back to publishers and site operators.
Scoring Rationale
The partnership matters for practitioners because it directly affects web data access and bot-management at scale, which are core to data collection and infrastructure. It's not a fundamental AI research breakthrough, but it changes operational realities for ML teams and publishers.
Practice with real Ad Tech data
90 SQL & Python problems · 15 industry datasets
250 free problems · No credit card
See all Ad Tech problemsStep-by-step roadmaps from zero to job-ready — curated courses, salary data, and the exact learning order that gets you hired.
Sources
- Read Original?Cloudflare and GoDaddy team up to help websites fend off Big Tech's AI bot swarm
- Read Original?Cloudflare, GoDaddy team up to curb AI bot brigades