OpenAI Says Robots.txt May Not Apply to ChatGPT's Fetch Bot
Search Engine Journal reported in its article that ChatGPT's page-fetching bot reached pages on sites that disallowed it in robots.txt, and that it did so on more tracked sites than other AI bots. The publication linked the finding to OpenAI documentation saying robots.txt controls may not apply to bots triggered by a person. OpenAI's documentation separately describes controls for search indexing, model-training crawls, and ad safety checks.
Search Engine Journal reported that ChatGPT's page-fetching bot has accessed pages on websites that disallow the bot through robots.txt. The publication reported that the bot reached disallowed pages on more tracked sites than other AI bots in its dataset, and linked that finding to OpenAI documentation describing limits on robots.txt controls for user-triggered fetching.
OpenAI's crawler documentation states that it uses web crawlers and user agents for actions performed automatically or triggered by a user request. Its published controls distinguish several agents and purposes, rather than treating all OpenAI-originated web requests as a single crawler category.
Separate controls for separate uses
According to OpenAI's documentation, OAI-SearchBot is used to surface sites in ChatGPT search features. Website operators can use robots.txt to opt out of that search bot; OpenAI states that sites opting out will not appear in ChatGPT search answers, although they can still appear as navigational links. The documentation states that changes to robots.txt can take about 24 hours to affect search systems.
The same documentation describes GPTBot controls as a way for webmasters to indicate that crawled content should not be used to train OpenAI's generative AI foundation models. It also documents OAI-AdsBot, which OpenAI states visits landing pages submitted as ChatGPT ads for policy validation and relevance assessment. OpenAI states that data collected by OAI-AdsBot is not used to train its foundation models.
Search Engine Journal reported that OpenAI's documentation treats robots.txt differently for a bot acting at a person's request, meaning a site-level disallow rule may not prevent a user-triggered ChatGPT fetch. The retrieved OpenAI developer documentation does not, in its visible excerpt, provide a detailed operational description of that specific ChatGPT page-fetching agent.
Operational implications
For site operators, the distinction matters because robots.txt is a crawler-access convention, not an access-control mechanism. Teams seeking to restrict retrieval of sensitive or proprietary pages generally use authentication, authorization, network controls, or server-side blocking in addition to crawler directives.
More broadly, AI retrieval systems increasingly combine indexing crawlers with on-demand URL fetching. That architecture can create materially different web-access behavior even where both flows originate from the same product ecosystem, making user-agent logging and explicit policy review important for web infrastructure and data-governance teams.
Key Points
- 1Search Engine Journal reported that ChatGPT's fetch bot reached robots.txt-disallowed pages across more tracked sites than peer AI bots.
- 2OpenAI documents separate bots for search, model-training controls, and ad validation, so a single robots.txt rule does not cover every use.
- 3As AI systems combine indexing with user-triggered retrieval, organizations commonly require access controls beyond voluntary crawler directives.
Scoring Rationale
The reported behavior is relevant to web data governance, content controls, and teams operating sites accessed by AI agents. It does not introduce a new model or API, but it clarifies an important limitation of a widely used crawler-control convention.
Sources
Primary source and supporting public references used for this report.
Practice interview problems based on real data
1,625 SQL & Python problems across 15 industry datasets — the exact type of data you work with.
Try 250 free problems
