Unverified50% confidenceFactExact time
robots.txt only constrains well-behaved crawlers and does not constitute access control; actual blocking often comes from bot management rules in a CDN or WAF.
1
Sources
50%
Confidence
Long-term
Relevance
10/17/2026
First Seen
Sources
Related Entities
Related Claims
Unverifiednoindex differs from robots.txt's Disallow: the former allows crawling but refuses indexing, while the latter blocks crawling.69% similarUnverifiedThe robots.txt protocol does not carry legally binding enforcement power.66% similarUnverifiedThe X platform restricts AI crawlers' access in its robots.txt and takes legal action against unauthorized data scraping.65% similarUnverifiedThere is debate over whether the robots.txt protocol has legal binding force on AI training crawlers.62% similar
Cite This Claim
Stable URI
https://kongchang.com/claim/1052273API
curl https://kongchang.com/api/v1/knowledge/claims/1052273MCP
get_claim(id=1052273)