OAI-SearchBot

OpenAI's official documentation states that settings such as OAI-SearchBot, GPTBot, and ChatGPT-User are independent of one another; OAI-SearchBot is responsible for making sites appear in ChatGPT search, and sites that opt out will not appear in ChatGPT search answers.

AI crawlers include GPTBot, ClaudeBot, PerplexityBot, and others.

OpenAI states that websites opting out of OAI-SearchBot will not appear in ChatGPT search answers, but may still appear in the form of navigational links

GPTBot, PerplexityBot, and ClaudeBot are the crawlers used by OpenAI, Perplexity, and Anthropic, respectively, to crawl public web content.

If retrieval-type AI crawlers (such as OAI-SearchBot and ChatGPT-User) are blocked, the content will not be cited in AI answers; but if only training-type crawlers (such as GPTBot) are blocked, the content may still be cited.

GPTBot and Google-Extended are training-type crawlers, while OAI-SearchBot and ChatGPT-User are retrieval-type crawlers used to read content when answering queries in real time.

blog.naver.com/robots.txt explicitly prohibits AI-related crawlers such as GPTBot, OAI-SearchBot, PerplexityBot, and Google-Extended, and states in a comment that crawling for AI training and RAG purposes is strictly forbidden.

A misspelling in the user-agent name will cause the rule to fail silently; the correct names include GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, Claude-SearchBot, Claude-User, PerplexityBot, and Perplexity-User.

GPTBot (OpenAI training and retrieval), OAI-SearchBot (ChatGPT search), ClaudeBot, PerplexityBot, and Google-Extended (Google's AI training, independent of Googlebot) are currently the most critical AI crawler identifiers

GPTBot's server rejection rate was 14.4%, of which 11.1% occurred while robots.txt allowed it; ClaudeBot was rejected 14.8%, with 10.9% under allowance; PerplexityBot was rejected 11.3%, with 8.4% under allowance; OAI-SearchBot was rejected 9.8%, with 8.0% under allowance.

Among the 549 top-1000 Tranco sites that return 2xx to ordinary browsers, 78 (14.2%) rejected at least one of GPTBot, OAI-SearchBot, ClaudeBot or PerplexityBot, even though their robots.txt explicitly allows these crawlers.

After excluding the 14 sites rejected solely due to 429 rate limiting, 64 sites (11.7%) were still rejected by the server despite robots.txt allowing them.

Blocking GPTBot (training) while allowing OAI-SearchBot (ChatGPT search citations) is a reasonable and common choice, and the 2.6% of top sites that seriously do this prove it works.

You can compare the status codes returned by curl when requesting the homepage with a Mozilla browser user agent versus an OAI-SearchBot user agent to determine whether server behavior is consistent with robots.txt: a browser getting 200 and a crawler getting 403 indicates that a CDN or WAF layer rule is at work.

The opt-out cost for the three types of AI crawlers forms a gradient: training crawlers < search crawlers < user-triggered fetchers

Blocking GPTBot in order to appear in ChatGPT depends on which OpenAI agent you mean; if search visibility matters, you should consult the latest user agent documentation before writing rules.

AI crawler user agents can be divided into three categories: training (GPTBot, ClaudeBot, CCBot, Google-Extended, Applebot-Extended), search indexing (OAI-SearchBot, Claude-SearchBot, PerplexityBot), and user-triggered (ChatGPT-User, Claude-User, Perplexity-User).

Marketers should check CDN or WAF logs to confirm that OAI-SearchBot, ChatGPT-User, PerplexityBot, Claude-SearchBot and Googlebot receive 200 responses rather than 403.

The access permissions set in robots.txt for AI crawlers such as GPTBot, ClaudeBot, and PerplexityBot directly determine whether content can enter these models' training sets or real-time retrieval pools

Among roughly 3000 AI crawler requests in the same period, the homepage was requested 207 times, with the breakdown being Googlebot 39, GPTBot 29, OAI-SearchBot 25, BingBot 20, and ClaudeBot 11.