Industry Newsweb crawlinggenerative aianthropicopenai
Anthropic and OpenAI Crawl Web With Low Referrals
8.1
Cloudflare data for the first week of January shows Anthropic and OpenAI sent far fewer referral visits relative to their web crawls, a ratio that worsened since early September 2025. Cloudflare, which services about 20% of the web, computes a crawl-to-refer metric indicating these firms extract content while sending limited traffic back, sometimes increasing costs for site owners.
Key Points
- 1Cloudflare data shows Anthropic and OpenAI crawled sites many times for each referral in early January.
- 2Indicates companies harvest web content without sending equivalent referral traffic, breaking the prior referral 'grand bargain'.
- 3Raises operational costs for some site owners and reduces downstream ad/subscription revenue and traffic referrals.
Scoring Rationale
High relevance and credible Cloudflare data drive the score, tempered by limited novelty and mainly observational reporting.
Sources
Public references used for this report.
Practice interview problems based on real data
1,625 SQL & Python problems across 15 industry datasets — the exact type of data you work with.
Try 250 free problems
