Unverified50% confidenceFactExact time
A 2024 study found that among known AI training crawlers, more than 35% completely disregard the restriction directives in robots.txt.
1
Sources
50%
Confidence
Medium-term (~90 days)
Relevance
8/10/2026
First Seen
Valid until: 11/8/2026
Sources
AI爬虫压垮Gentoo Bugzilla：开源基础设施面临数据掠夺危机
Related Claims
UnverifiedMany AI companies' crawlers have been found to ignore the restriction directives in robots.txt, or to bypass the rules using crawlers with undisclosed identities80% similarUnverifiedMany AI training data crawlers tend to ignore the robots.txt protocol and rapidly scrape entire sites at extremely high concurrency, fundamentally differing from traditional search engine crawlers.67% similarUnverifiedAI crawlers include GPTBot, ClaudeBot, PerplexityBot, and others.66% similarUnverified2023年MIT的一项研究表明非专业人员借助AI聊天机器人可在数小时内获得此前需专业训练才能掌握的病原体相关知识62% similarUnverifiedAI crawlers evade blocking by rotating IPs and forging User-Agent strings, disregarding robots.txt rules and rate limits.59% similar
Cite This Claim
Stable URI
https://kongchang.com/claim/725376API
curl https://kongchang.com/api/v1/knowledge/claims/725376MCP
get_claim(id=725376)