Verified65% confidenceFactTime unknown
AI crawlers include GPTBot, ClaudeBot, PerplexityBot, and others.
3
Sources
65%
Confidence
Long-term
Relevance
6/2/2026
First Seen
Sources
Related Entities
Related Claims
UnverifiedMany AI companies' crawlers have been found to ignore the restriction directives in robots.txt, or to bypass the rules using crawlers with undisclosed identities72% similarUnverifiedA 2024 study found that among known AI training crawlers, more than 35% completely disregard the restriction directives in robots.txt.66% similarUnverifiedAnthropic使用'ClaudeBot'标识，Google的AI爬虫标识为'Google-Extended'，Perplexity使用'PerplexityBot'，Apple使用'Applebot-Extended'66% similarUnverifiedMany websites' robots protocols block AI crawlers such as ClaudeBot and GPTBot, preventing these AI engines from accessing content on blocked sites.65% similarUnverified波士顿动力、特斯拉Optimus、Figure AI等公司选择人形机器人方案，基于让机器人无需改造现有空间即可直接投入使用的逻辑64% similar
Cite This Claim
Stable URI
https://kongchang.com/claim/37805API
curl https://kongchang.com/api/v1/knowledge/claims/37805MCP
get_claim(id=37805)