robots.txt
A plain-text file protocol placed in a website's root directory that controls crawler access to pages through User-Agent fields and Disallow/Allow directives. Introduced in 1994, it operates as a voluntary, honor-based convention without legally binding enforcement mechanisms.
Core Facts
OpenAI's official documentation states that settings such as OAI-SearchBot, GPTBot, and ChatGPT-User are independent of one another; OAI-SearchBot is responsible for making sites appear in ChatGPT search, and sites that opt out will not appear in ChatGPT search answers.
The robots.txt protocol was first proposed by Dutch engineer Martijn Koster in 1994
The robots.txt protocol was proposed by internet pioneer Martijn Koster in 1994 and is a plain text file placed in the root directory of a website
Timeline (last 90 days)
robots.txt is an early conventional crawler protocol that is effective only on those who comply and carries no binding force.
mcp-server-fetch's robots.txt check no longer targets only the initially requested URL but is re-checked after every redirect (GHSA-3jqg-r5pp-rmp2).
robots.txt can only express a binary "allow/disallow" stance and cannot distinguish between an agent acting on behalf of a real user, the load cost brought by access frequency, or the potential transactional value.
Website owners have begun managing AI crawler access through robots.txt, IP allowlists and paid licensing.
For more than two decades, the internet has managed automated traffic through a combination of robots.txt, anti-crawling mechanisms, and CAPTCHAs.
robots.txt is not a technically enforced mechanism but a "gentlemen's agreement," and compliance depends entirely on the crawler's own self-discipline.
Handle robots.txt fetch failures conservatively: 404 means no rules, while a network failure means the rules are unknown, in which case requests should be prohibited.
The access permissions for AI crawlers (such as GPTBot and Google-Extended) in robots.txt, together with structured data, jointly determine whether AI can 'read' and 'cite' a brand's products.
The Python standard library's urllib.robotparser can be used to parse robots.txt and determine whether a URL is allowed to be crawled.
Technical means by which websites block agents mainly include robots.txt extensions, UA identification, behavioral fingerprint detection and CAPTCHA challenges, as well as the emerging paid-crawling protocols.
40 more timeline events
All Facts (20)
OpenAI's official documentation states that settings such as OAI-SearchBot, GPTBot, and ChatGPT-User are independent of one another; OAI-SearchBot is responsible for making sites appear in ChatGPT search, and sites that opt out will not appear in ChatGPT search answers.
80%VerifiedThe robots.txt protocol was first proposed by Dutch engineer Martijn Koster in 1994
80%VerifiedThe robots.txt protocol was proposed by internet pioneer Martijn Koster in 1994 and is a plain text file placed in the root directory of a website
75%VerifiedThe robots.txt protocol relies entirely on voluntary compliance by crawlers and has no technical enforcement mechanism whatsoever.
70%Verifiedrobots.txt is a plain text file placed in a website's root directory that tells web crawlers which pages can be crawled and which should be avoided
70%VerifiedA sitemap is a list of URLs for search engines, usually provided in XML format; it solves page discovery but does not describe the purpose or priority of the links.
65%VerifiedWhen collecting publicly available data, one should respect the website's robots protocol and terms of use; whether the scraping behavior is compliant depends on the target site's terms of service and local laws and regulations.
65%VerifiedMany mainstream AI training datasets were formed without strictly adhering to the robots.txt protocol.
65%VerifiedThe robots.txt protocol does not carry legally binding enforcement power.
65%Verifiedrobots.txt is essentially an honor system — any crawler can bypass all restrictions simply by ignoring the file, and webmasters have no technical means to verify whether crawlers comply.
65%Verifiedrobots.txt is a plain text file placed in the root directory of a website that follows the Robots Exclusion Standard and is used to declare to crawlers which paths are allowed to be accessed.
65%VerifiedGooglebot determines the scope of crawlable pages by following each website's robots.txt protocol; robots.txt is a text file stored in the root directory of a website used to declare crawlable or blocked pages to web crawlers.
65%VerifiedThe robots.txt protocol is a gentleman's agreement and has no technically binding enforcement
65%Verifiedrobots.txt is an industry convention established in 1994 by the Robots Exclusion Protocol (REP) and is not a mandatory legal regulation, but mainstream search engines such as Google and Baidu strictly adhere to it
65%Verifiedllms.txt is an emerging website standard proposal, similar to robots.txt but specifically targeting AI crawlers, written in Markdown format.
65%VerifiedMany AI training data crawlers tend to ignore the robots.txt protocol and rapidly scrape entire sites at extremely high concurrency, fundamentally differing from traditional search engine crawlers.
65%Partially Verifiedllms.txt has a different positioning from robots.txt (which controls crawler access permissions) and sitemap.xml (which lists URLs); its goal is to let models quickly understand a site's structure and core content.
65%UnverifiedA growing number of content platforms have begun charging AI companies data licensing fees and introducing anti-scraping mechanisms.
60%Unverifiedllms.txt draws on the concept of robots.txt and aims to provide large language models with a clear and concise content index
60%Unverifiedrobots.txt is a 'gentleman's agreement' that relies on the crawler's voluntary compliance and has no enforcement power
60%