Unverified50% confidenceOpinionExact time
robots.txt can only express a binary "allow/disallow" stance and cannot distinguish between an agent acting on behalf of a real user, the load cost brought by access frequency, or the potential transactional value.
1
Sources
50%
Confidence
Long-term
Relevance
10/22/2026
First Seen
Sources
Related Entities
Related Claims
UnverifiedEven when a crawler has declared its identity, a high-concurrency request rate can in itself effectively cause a denial-of-service outcome without violating robots.txt rules.73% similarUnverifiedUnrecognized merchants are placed under "other" rather than being forced into the wrong category; when a message contains no amount, the bot asks the user instead of making one up.72% similarUnverifiedrobots.txt is publicly readable, and the list of blocked User-Agents published by operators can actually be used by crawlers to disguise themselves as other identities.72% similarUnverifiedrobots.txt is a 'gentleman's agreement' that relies on the crawler's voluntary compliance and has no enforcement power70% similarUnverifiedThe terms of service of most online platforms explicitly prohibit automated scripts or bots from simulating human actions, placing agents that log into third-party services as real users in a compliance gray area66% similar
Cite This Claim
Stable URI
https://kongchang.com/claim/1085084API
curl https://kongchang.com/api/v1/knowledge/claims/1085084MCP
get_claim(id=1085084)