GPTBot — OpenAI's ai training crawler

GPTBot is one of the highest-volume AI crawlers on the web. It crawls without executing JavaScript, so client-rendered content is largely invisible to it, and it revisits sites on its own schedule rather than in response to user queries.

OpenAI states that GPTBot respects robots.txt and filters out sources that require paywall access or are known to collect personally identifiable information. In practice, publishers consistently confirm that a robots.txt disallow stops GPTBot's crawling.

GPTBot identifies itself with the following user-agent string:

Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; GPTBot/1.2; +https://openai.com/gptbot

Never trust the user-agent string alone. Scrapers routinely impersonate well-known crawlers to inherit their access. OpenAI publishes GPTBot's IP ranges as a machine-readable JSON file. A request claiming to be GPTBot from an IP outside those ranges is spoofed.

Published IP ranges: https://openai.com/gptbot.json

Note that most AI crawlers and fetchers, GPTBot included, do not execute JavaScript — so this traffic is invisible to GA4 and every script-based analytics tool. Server logs, CDN analytics, or a dedicated bot-analytics layer are the only places you will see it. For the full picture of measuring AI-driven visits, see our guide on how to track AI traffic.

To refuse GPTBot access to your entire site, add this to your robots.txt:

User-agent: GPTBot
Disallow: /

To restrict it from specific sections only (for example, premium content) while leaving the rest open:

User-agent: GPTBot
Disallow: /premium/
Disallow: /members/

The case for blocking: If your content is your product — original reporting, proprietary data, paid research — letting GPTBot ingest it for free hands your value to a model that will answer readers' questions without sending them to you. Blocking GPTBot is the standard first move for publishers who want licensing leverage: you cannot negotiate over something you are already giving away.

The case for allowing or monetizing: Training-data access is exactly the kind of access that can be licensed rather than donated. A growing number of publishers block training crawlers by default and then sell structured access back — per-crawl licensing, RAG access, or sponsored inclusion. That converts GPTBot from a cost (server load, zero referrals) into a revenue line.