GPTBot

OpenAI's official documentation states that settings such as OAI-SearchBot, GPTBot, and ChatGPT-User are independent of one another; OAI-SearchBot is responsible for making sites appear in ChatGPT search, and sites that opt out will not appear in ChatGPT search answers.

AI crawlers include GPTBot, ClaudeBot, PerplexityBot, and others.

GPTBot is OpenAI's crawler identifier made public in August 2023

Mainstream AI crawlers include GPTBot (OpenAI), ClaudeBot (Anthropic), Google-Extended (Google DeepMind), and Bytespider (ByteDance).

OpenAI uses the identifier 'GPTBot' for its training data crawler.

The verified AI bot request count metric counts requests separately by verified AI bot, and verified AI bots usually refer to crawlers whose identity is confirmed through reverse DNS or IP range verification, such as GPTBot, Google-Extended, ClaudeBot, and others.

AI crawler and bot traffic management involves distinguishing three categories of automated traffic: legitimate search engine crawlers (such as Googlebot), AI training data collectors (such as OpenAI's GPTBot and Common Crawl), and malicious bots (content scraping, credential stuffing, inventory hoarding).

The access permissions for AI crawlers (such as GPTBot and Google-Extended) in robots.txt, together with structured data, jointly determine whether AI can 'read' and 'cite' a brand's products.

GPTBot, PerplexityBot, and ClaudeBot are the crawlers used by OpenAI, Perplexity, and Anthropic, respectively, to crawl public web content.

GPTBot is the crawler used by OpenAI; its User-Agent string contains the 'GPTBot' identifier, typically of the form 'Mozilla/5.0 ... GPTBot/1.x'.

GPTBot is OpenAI's training crawler and is a separate setting from the search crawler, so training can be blocked while search is retained.

If retrieval-type AI crawlers (such as OAI-SearchBot and ChatGPT-User) are blocked, the content will not be cited in AI answers; but if only training-type crawlers (such as GPTBot) are blocked, the content may still be cited.

blog.naver.com/robots.txt explicitly prohibits AI-related crawlers such as GPTBot, OAI-SearchBot, PerplexityBot, and Google-Extended, and states in a comment that crawling for AI training and RAG purposes is strictly forbidden.

GPTBot and Google-Extended are training-type crawlers, while OAI-SearchBot and ChatGPT-User are retrieval-type crawlers used to read content when answering queries in real time.

A misspelling in the user-agent name will cause the rule to fail silently; the correct names include GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, Claude-SearchBot, Claude-User, PerplexityBot, and Perplexity-User.

GPTBot (OpenAI training and retrieval), OAI-SearchBot (ChatGPT search), ClaudeBot, PerplexityBot, and Google-Extended (Google's AI training, independent of Googlebot) are currently the most critical AI crawler identifiers

Mainstream AI crawlers such as GPTBot, ClaudeBot and PerplexityBot currently largely do not execute JavaScript, so for sites that depend on JavaScript rendering, even if both robots.txt and the WAF let them through, the content they get is still empty.

GPTBot's server rejection rate was 14.4%, of which 11.1% occurred while robots.txt allowed it; ClaudeBot was rejected 14.8%, with 10.9% under allowance; PerplexityBot was rejected 11.3%, with 8.4% under allowance; OAI-SearchBot was rejected 9.8%, with 8.0% under allowance.

53 sites prohibit GPTBot in robots.txt yet continue to serve it normally, showing that robots.txt is advisory in both directions.

Blocking GPTBot (training) while allowing OAI-SearchBot (ChatGPT search citations) is a reasonable and common choice, and the 2.6% of top sites that seriously do this prove it works.