Verified65% confidenceFactExact time
Many mainstream AI training datasets were formed without strictly adhering to the robots.txt protocol.
3
Sources
65%
Confidence
Long-term
Relevance
9/19/2026
First Seen
Sources
Related Entities
Related Claims
VerifiedMany AI training data crawlers tend to ignore the robots.txt protocol and rapidly scrape entire sites at extremely high concurrency, fundamentally differing from traditional search engine crawlers.79% similarUnverifiedThe same robots.txt can finely control allowing search indexing but prohibiting AI training, because different AI vendors use multiple different User-agents corresponding to different purposes.78% similarUnverifiedAI agent execution frameworks (such as LangChain, AutoGPT, or OpenAI's Assistants API) do not by default enforce checking of robots.txt, unless developers actively implement the relevant logic.74% similarUnverifiedAI+机器人自动化实验室的最大技术瓶颈在于表征环节的自动化，因为许多性能测试本质上需要时间无法被软件加速71% similarUnverifiedThe user-agent strings of AI training data crawlers are not uniform, and some even disguise themselves as ordinary browsers.71% similar
Cite This Claim
Stable URI
https://kongchang.com/claim/935920API
curl https://kongchang.com/api/v1/knowledge/claims/935920MCP
get_claim(id=935920)