Claude-User
Anthropic's real-time user-agent crawler that fetches page content on behalf of a Claude user when that user requests browsing of a web page.
Core Facts
Timeline (last 90 days)
GPTBot, PerplexityBot, and ClaudeBot are the crawlers used by OpenAI, Perplexity, and Anthropic, respectively, to crawl public web content.
Anthropic has three crawlers: ClaudeBot for training, Claude-User for user-triggered crawling, and Claude-SearchBot for improving search result quality.
A misspelling in the user-agent name will cause the rule to fail silently; the correct names include GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, Claude-SearchBot, Claude-User, PerplexityBot, and Perplexity-User.
Anthropic says that disallowing Claude-User will prevent Claude from retrieving pages for user queries, but OpenAI and Perplexity say their own user-triggered fetchers may not obey robots.txt
AI crawler user agents can be divided into three categories: training (GPTBot, ClaudeBot, CCBot, Google-Extended, Applebot-Extended), search indexing (OAI-SearchBot, Claude-SearchBot, PerplexityBot), and user-triggered (ChatGPT-User, Claude-User, Perplexity-User).
Anthropic's split logic is consistent with OpenAI's: ClaudeBot collects content that may be used for training, Claude-SearchBot is used to improve search result quality, and Claude-User only crawls pages when a user actively asks.
The access permissions set in robots.txt for AI crawlers such as GPTBot, ClaudeBot, and PerplexityBot directly determine whether content can enter these models' training sets or real-time retrieval pools
All Facts (8)
AI crawlers include GPTBot, ClaudeBot, PerplexityBot, and others.
65%UnverifiedGPTBot, PerplexityBot, and ClaudeBot are the crawlers used by OpenAI, Perplexity, and Anthropic, respectively, to crawl public web content.
50%UnverifiedAnthropic has three crawlers: ClaudeBot for training, Claude-User for user-triggered crawling, and Claude-SearchBot for improving search result quality.
50%UnverifiedA misspelling in the user-agent name will cause the rule to fail silently; the correct names include GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, Claude-SearchBot, Claude-User, PerplexityBot, and Perplexity-User.
50%UnverifiedAnthropic says that disallowing Claude-User will prevent Claude from retrieving pages for user queries, but OpenAI and Perplexity say their own user-triggered fetchers may not obey robots.txt
50%UnverifiedAI crawler user agents can be divided into three categories: training (GPTBot, ClaudeBot, CCBot, Google-Extended, Applebot-Extended), search indexing (OAI-SearchBot, Claude-SearchBot, PerplexityBot), and user-triggered (ChatGPT-User, Claude-User, Perplexity-User).
50%UnverifiedAnthropic's split logic is consistent with OpenAI's: ClaudeBot collects content that may be used for training, Claude-SearchBot is used to improve search result quality, and Claude-User only crawls pages when a user actively asks.
50%UnverifiedThe access permissions set in robots.txt for AI crawlers such as GPTBot, ClaudeBot, and PerplexityBot directly determine whether content can enter these models' training sets or real-time retrieval pools
50%