Unverified60% confidenceFactExact time
robots.txt is a 'gentleman's agreement' that relies on the crawler's voluntary compliance and has no enforcement power
2
Sources
60%
Confidence
Long-term
Relevance
8/2/2026
First Seen
Sources
Only 8.9% of Websites Block AI Crawlers, Yet 94.8% Have Never Been Cited in AI Answers
Related Entities
Related Claims
Unverifiedrobots.txt can only express a binary "allow/disallow" stance and cannot distinguish between an agent acting on behalf of a real user, the load cost brought by access frequency, or the potential transactional value.70% similarUnverifiedrobots.txt is publicly readable, and the list of blocked User-Agents published by operators can actually be used by crawlers to disguise themselves as other identities.67% similarUnverifiedIf content must be kept confidential, it should be placed behind a login, because robots.txt cannot block everything.66% similarUnverified53 sites prohibit GPTBot in robots.txt yet continue to serve it normally, showing that robots.txt is advisory in both directions.64% similarUnverifiedWhen scraping web pages, you should comply with the site's robots.txt and terms, and avoid high-frequency requests to small sites.63% similar
Cite This Claim
Stable URI
https://kongchang.com/claim/679738API
curl https://kongchang.com/api/v1/knowledge/claims/679738MCP
get_claim(id=679738)