Google-Extended
A robots.txt user agent provided by Google that controls whether a website's content may be used for the training and improvement of Google AI models such as Gemini.
Core Facts
Timeline (last 90 days)
The verified AI bot request count metric counts requests separately by verified AI bot, and verified AI bots usually refer to crawlers whose identity is confirmed through reverse DNS or IP range verification, such as GPTBot, Google-Extended, ClaudeBot, and others.
The access permissions for AI crawlers (such as GPTBot and Google-Extended) in robots.txt, together with structured data, jointly determine whether AI can 'read' and 'cite' a brand's products.
GPTBot and Google-Extended are training-type crawlers, while OAI-SearchBot and ChatGPT-User are retrieval-type crawlers used to read content when answering queries in real time.
blog.naver.com/robots.txt explicitly prohibits AI-related crawlers such as GPTBot, OAI-SearchBot, PerplexityBot, and Google-Extended, and states in a comment that crawling for AI training and RAG purposes is strictly forbidden.
GPTBot (OpenAI training and retrieval), OAI-SearchBot (ChatGPT search), ClaudeBot, PerplexityBot, and Google-Extended (Google's AI training, independent of Googlebot) are currently the most critical AI crawler identifiers
Google-Extended and Applebot-Extended are not crawlers but control tokens in robots.txt, used to decide whether content crawled by Googlebot and Applebot can be used for Gemini or Apple model training, and they do not appear in server logs
At present there is no public adoption-rate or effectiveness data on whether, and to what extent, mainstream AI crawlers (GPTBot, ClaudeBot, Google-Extended, etc.) read llms.txt.
Googlebot and Google-Extended each serve different systems, and crawling behavior is separate from ranking and AI feature display.
Google-Extended controls whether Google can use content for Gemini model training and grounding; it does not affect Google Search or AI Overviews, which use the regular Googlebot.
AI crawler user agents can be divided into three categories: training (GPTBot, ClaudeBot, CCBot, Google-Extended, Applebot-Extended), search indexing (OAI-SearchBot, Claude-SearchBot, PerplexityBot), and user-triggered (ChatGPT-User, Claude-User, Perplexity-User).
4 more timeline events
All Facts (14)
Blocking Google-Extended does not affect a site's indexing, ranking, or display in Google Search, nor does it require resubmitting the sitemap.
65%UnverifiedMainstream AI crawlers include GPTBot (OpenAI), ClaudeBot (Anthropic), Google-Extended (Google DeepMind), and Bytespider (ByteDance).
60%UnverifiedThe verified AI bot request count metric counts requests separately by verified AI bot, and verified AI bots usually refer to crawlers whose identity is confirmed through reverse DNS or IP range verification, such as GPTBot, Google-Extended, ClaudeBot, and others.
50%UnverifiedThe access permissions for AI crawlers (such as GPTBot and Google-Extended) in robots.txt, together with structured data, jointly determine whether AI can 'read' and 'cite' a brand's products.
50%UnverifiedGPTBot and Google-Extended are training-type crawlers, while OAI-SearchBot and ChatGPT-User are retrieval-type crawlers used to read content when answering queries in real time.
50%Unverifiedblog.naver.com/robots.txt explicitly prohibits AI-related crawlers such as GPTBot, OAI-SearchBot, PerplexityBot, and Google-Extended, and states in a comment that crawling for AI training and RAG purposes is strictly forbidden.
50%UnverifiedGPTBot (OpenAI training and retrieval), OAI-SearchBot (ChatGPT search), ClaudeBot, PerplexityBot, and Google-Extended (Google's AI training, independent of Googlebot) are currently the most critical AI crawler identifiers
50%UnverifiedGoogle-Extended and Applebot-Extended are not crawlers but control tokens in robots.txt, used to decide whether content crawled by Googlebot and Applebot can be used for Gemini or Apple model training, and they do not appear in server logs
50%UnverifiedAt present there is no public adoption-rate or effectiveness data on whether, and to what extent, mainstream AI crawlers (GPTBot, ClaudeBot, Google-Extended, etc.) read llms.txt.
50%UnverifiedGooglebot and Google-Extended each serve different systems, and crawling behavior is separate from ranking and AI feature display.
50%UnverifiedAI crawler user agents can be divided into three categories: training (GPTBot, ClaudeBot, CCBot, Google-Extended, Applebot-Extended), search indexing (OAI-SearchBot, Claude-SearchBot, PerplexityBot), and user-triggered (ChatGPT-User, Claude-User, Perplexity-User).
50%UnverifiedGoogle-Extended controls whether Google can use content for Gemini model training and grounding; it does not affect Google Search or AI Overviews, which use the regular Googlebot.
50%UnverifiedGoogle-Extended is a robots.txt product token introduced by Google in September 2023 to control training and grounding of Google's Gemini models, and it is a completely separate token from Googlebot.
50%UnverifiedOpenAI registered a dedicated crawler identity, GPTBot, and Google registered Google-Extended, allowing website administrators to selectively block them.
50%