Anthropic Updates Crawler Documentation With Three Bots

This week Anthropic updated its crawler documentation to formalize three distinct web crawlers — ClaudeBot, Claude-User, and Claude-SearchBot — each with separate robots.txt user-agent strings. The page explains how blocking each bot affects training data collection, retrieval, and search indexing, aligning Anthropic with similar splits at OpenAI and Perplexity. Publishers must audit robots.txt to distinguish training, search, and user-initiated fetchers.