Unverified60% confidenceFactExact time
Many AI companies' crawlers have been found to ignore the restriction directives in robots.txt, or to bypass the rules using crawlers with undisclosed identities
2
Sources
60%
Confidence
Medium-term (~90 days)
Relevance
8/10/2026
First Seen
Valid until: 11/8/2026
Sources
AI时代的公地悲剧：谁在透支互联网的公共资源
Related Claims
UnverifiedA 2024 study found that among known AI training crawlers, more than 35% completely disregard the restriction directives in robots.txt.80% similarUnverifiedAI crawlers can be divided by purpose into three categories: training crawlers, search crawlers, and user-triggered fetchers, each with its own independent robots.txt identifier76% similarUnverifiedSome vendor documentation states that user-triggered AI agents may not follow robots.txt the way traditional crawlers do, because the requests are initiated by a real person.74% similarUnverifiedAI crawlers evade blocking by rotating IPs and forging User-Agent strings, disregarding robots.txt rules and rate limits.74% similarUnverifiedMany websites' robots protocols block AI crawlers such as ClaudeBot and GPTBot, preventing these AI engines from accessing content on blocked sites.73% similar
Cite This Claim
Stable URI
https://kongchang.com/claim/728310API
curl https://kongchang.com/api/v1/knowledge/claims/728310MCP
get_claim(id=728310)