ClaudeBot
A web crawler used by Anthropic to fetch web content for training Claude models, identified by its user-agent.
Core Facts
Timeline (last 90 days)
The verified AI bot request count metric counts requests separately by verified AI bot, and verified AI bots usually refer to crawlers whose identity is confirmed through reverse DNS or IP range verification, such as GPTBot, Google-Extended, ClaudeBot, and others.
ClaudeBot made 128 requests to /sitemap-posts.xml during the measurement period, accounting for two-thirds of the 192 requests to that file.
Among roughly 3000 AI crawler requests in the same period, the homepage was requested 207 times, with the breakdown being Googlebot 39, GPTBot 29, OAI-SearchBot 25, BingBot 20, and ClaudeBot 11.
GPTBot, PerplexityBot, and ClaudeBot are the crawlers used by OpenAI, Perplexity, and Anthropic, respectively, to crawl public web content.
A misspelling in the user-agent name will cause the rule to fail silently; the correct names include GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, Claude-SearchBot, Claude-User, PerplexityBot, and Perplexity-User.
Anthropic has three crawlers: ClaudeBot for training, Claude-User for user-triggered crawling, and Claude-SearchBot for improving search result quality.
GPTBot (OpenAI training and retrieval), OAI-SearchBot (ChatGPT search), ClaudeBot, PerplexityBot, and Google-Extended (Google's AI training, independent of Googlebot) are currently the most critical AI crawler identifiers
After excluding the 14 sites rejected solely due to 429 rate limiting, 64 sites (11.7%) were still rejected by the server despite robots.txt allowing them.
Mainstream AI crawlers such as GPTBot, ClaudeBot and PerplexityBot currently largely do not execute JavaScript, so for sites that depend on JavaScript rendering, even if both robots.txt and the WAF let them through, the content they get is still empty.
GPTBot's server rejection rate was 14.4%, of which 11.1% occurred while robots.txt allowed it; ClaudeBot was rejected 14.8%, with 10.9% under allowance; PerplexityBot was rejected 11.3%, with 8.4% under allowance; OAI-SearchBot was rejected 9.8%, with 8.0% under allowance.
14 more timeline events
All Facts (20)
AI crawlers include GPTBot, ClaudeBot, PerplexityBot, and others.
65%Partially VerifiedClaudeBot is Anthropic's crawler
65%UnverifiedMainstream AI crawlers include GPTBot (OpenAI), ClaudeBot (Anthropic), Google-Extended (Google DeepMind), and Bytespider (ByteDance).
60%UnverifiedThe verified AI bot request count metric counts requests separately by verified AI bot, and verified AI bots usually refer to crawlers whose identity is confirmed through reverse DNS or IP range verification, such as GPTBot, Google-Extended, ClaudeBot, and others.
50%UnverifiedGPTBot, PerplexityBot, and ClaudeBot are the crawlers used by OpenAI, Perplexity, and Anthropic, respectively, to crawl public web content.
50%UnverifiedA misspelling in the user-agent name will cause the rule to fail silently; the correct names include GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, Claude-SearchBot, Claude-User, PerplexityBot, and Perplexity-User.
50%UnverifiedAnthropic has three crawlers: ClaudeBot for training, Claude-User for user-triggered crawling, and Claude-SearchBot for improving search result quality.
50%UnverifiedGPTBot (OpenAI training and retrieval), OAI-SearchBot (ChatGPT search), ClaudeBot, PerplexityBot, and Google-Extended (Google's AI training, independent of Googlebot) are currently the most critical AI crawler identifiers
50%UnverifiedMainstream AI crawlers such as GPTBot, ClaudeBot and PerplexityBot currently largely do not execute JavaScript, so for sites that depend on JavaScript rendering, even if both robots.txt and the WAF let them through, the content they get is still empty.
50%UnverifiedAmong the 549 top-1000 Tranco sites that return 2xx to ordinary browsers, 78 (14.2%) rejected at least one of GPTBot, OAI-SearchBot, ClaudeBot or PerplexityBot, even though their robots.txt explicitly allows these crawlers.
50%UnverifiedAfter excluding the 14 sites rejected solely due to 429 rate limiting, 64 sites (11.7%) were still rejected by the server despite robots.txt allowing them.
50%UnverifiedGPTBot's server rejection rate was 14.4%, of which 11.1% occurred while robots.txt allowed it; ClaudeBot was rejected 14.8%, with 10.9% under allowance; PerplexityBot was rejected 11.3%, with 8.4% under allowance; OAI-SearchBot was rejected 9.8%, with 8.0% under allowance.
50%UnverifiedOpenAI published documentation on GPTBot and how to allow or block it, and Anthropic provided the same instructions for its ClaudeBot.
50%UnverifiedIn a GEO workflow, pages must be crawlable by AI crawlers (such as GPTBot, ClaudeBot, PerplexityBot), otherwise they cannot be cited.
50%UnverifiedAnthropic explicitly states that blocking ClaudeBot means the website's future content is excluded from the training dataset, i.e., blocking only takes effect for the future
50%UnverifiedAt present there is no public adoption-rate or effectiveness data on whether, and to what extent, mainstream AI crawlers (GPTBot, ClaudeBot, Google-Extended, etc.) read llms.txt.
50%UnverifiedAI crawler user agents can be divided into three categories: training (GPTBot, ClaudeBot, CCBot, Google-Extended, Applebot-Extended), search indexing (OAI-SearchBot, Claude-SearchBot, PerplexityBot), and user-triggered (ChatGPT-User, Claude-User, Perplexity-User).
50%UnverifiedAttackers heavily forge the User-Agents of AI crawlers such as GPTBot, ClaudeBot and PerplexityBot, with a single IP rotating through User-Agents such as KimiBot, Qwenbot and Hunyuan within a single session while requesting the `.env` file.
50%UnverifiedAnthropic's split logic is consistent with OpenAI's: ClaudeBot collects content that may be used for training, Claude-SearchBot is used to improve search result quality, and Claude-User only crawls pages when a user actively asks.
50%UnverifiedMany websites' robots protocols block AI crawlers such as ClaudeBot and GPTBot, preventing these AI engines from accessing content on blocked sites.
50%