Anthropic Intensifies Web Crawling Affecting Websites
Anthropic has dramatically increased web crawling activity in recent months, prompting complaints from website operators, SEO professionals, and publishers who say high request volumes strain servers and ignore robots.txt directives. Reports identify ClaudeBot and anthropic-ai user-agents making tens to hundreds of thousands of requests, leading site owners to use IP blocks, CAPTCHAs, legal action, or EU opt-outs; Anthropic says it provides blocking guidance and is reviewing practices.
Scoring Rationale
High industry impact and actionable implications drive the score, limited by primarily Search Engine Land reporting and incomplete Anthropic transparency.
Practice interview problems based on real data
1,500+ SQL & Python problems across 15 industry datasets — the exact type of data you work with.
Try 250 free problems
