Anthropic Updates Crawler Documentation With Three Bots
This week Anthropic updated its crawler documentation to formalize three distinct web crawlers — ClaudeBot, Claude-User, and Claude-SearchBot — each with separate robots.txt user-agent strings. The page explains how blocking each bot affects training data collection, retrieval, and search indexing, aligning Anthropic with similar splits at OpenAI and Perplexity. Publishers must audit robots.txt to distinguish training, search, and user-initiated fetchers.
Key Points
- 1Lists three distinct bots: ClaudeBot, Claude-User, and Claude-SearchBot with unique user-agents.
- 2Explains blocking impacts, noting search and retrieval bots can reduce site visibility in AI results.
- 3Advises publishers to audit robots.txt and separate entries for training, search, and user fetchers.
Scoring Rationale
Official documentation change with clear publisher impact supports a high score, but the update is incremental within an existing industry trend.
Sources
Public references used for this report.
Practice interview problems based on real data
1,625 SQL & Python problems across 15 industry datasets — the exact type of data you work with.
Try 250 free problems