Blog

OAI-SearchBot and robots.txt: how to check whether ChatGPT search can crawl your site, and fix it

By Visibility Audit Co. Published November 2, 2026.

Short answer: ChatGPT search finds pages with OpenAI's OAI-SearchBot crawler. If your robots.txt disallows it, or your firewall blocks it, your pages won't appear in ChatGPT search answers, though they can still appear as navigational links. Open yoursite.com/robots.txt, look for OAI-SearchBot or a blanket "User-agent: *" disallow, and add an Allow rule. GPTBot is separate: it controls training, not search.

The short answer

OpenAI says sites that opt out of OAI-SearchBot "will not be shown in ChatGPT search answers, though can still appear as navigational links" (OpenAI crawler docs). To rule out a block:

Open https://yoursite.com/robots.txt in a browser.
Find the rules that apply to OAI-SearchBot: a group named OAI-SearchBot, or the User-agent: * group if there isn't one.
Add an allow block for OAI-SearchBot (copy-paste version below).
Recheck after about 24 hours. That's how long OpenAI says its search systems take to adjust after a robots.txt change.

Which bot does what

The names are similar, but each crawler has its own job. Match on the token in the user-agent string (for example OAI-SearchBot), not the full string, because version numbers change.

Bot	Owner	Purpose	Obeys robots.txt?	Source
OAI-SearchBot	OpenAI	Surfaces sites in ChatGPT search	Yes	OpenAI
GPTBot	OpenAI	Crawls content that may be used to train OpenAI's models	Yes	OpenAI
ChatGPT-User	OpenAI	Visits a page when a ChatGPT user asks it to; not used to decide what appears in search	Rules "may not apply"	OpenAI
PerplexityBot	Perplexity	Surfaces and links sites in Perplexity search; not used for training foundation models	Yes	Perplexity
Perplexity-User	Perplexity	Fetches a page when a user asks	"Generally ignores" robots.txt	Perplexity
Googlebot	Google	Google Search and all its features (AI Overviews are one)	Yes	Google
Google-Extended	Google	Controls use of content for training future Gemini models and for grounding in Gemini Apps and Vertex AI; no effect on Search	Control token in robots.txt	Google

OAI-SearchBot vs GPTBot: you can say yes to one and no to the other

OpenAI says the settings are independent. A site can allow OAI-SearchBot and disallow GPTBot (OpenAI crawler docs). That means you don't have to choose between showing up in ChatGPT search and keeping your content out of training.

A common setup is to allow OAI-SearchBot so you can appear in ChatGPT search, then decide on GPTBot separately based on how you feel about training. If you blocked GPTBot at some point, check that you didn't block OAI-SearchBot along with it.

How to check your robots.txt in 2 minutes

Look for a blanket block. User-agent: * followed by Disallow: / blocks every crawler that doesn't have its own group, including OAI-SearchBot.
Look for named groups. Search the file for OAI-SearchBot and GPTBot and read the Allow and Disallow lines under each.
Know which group wins. A crawler follows the most specific group that matches it. If there's an OAI-SearchBot group, OAI-SearchBot uses that group and ignores the * group.
See what's served. Run curl -A "OAI-SearchBot" https://yoursite.com/robots.txt to see the file your server returns. This only fetches the file with that user-agent. It doesn't simulate the real bot, which comes from OpenAI's IP ranges.

When robots.txt isn't the problem: firewalls and bot protection

A crawler can be allowed in robots.txt and still get blocked before it reaches your pages. Common causes are a Cloudflare or other web application firewall rule, a "block AI bots" toggle, and rate limits.

Allowlist the real IP ranges. OpenAI publishes its crawler IP lists at openai.com/searchbot.json, openai.com/gptbot.json and openai.com/chatgpt-user.json, and recommends allowing them (OpenAI). Perplexity publishes www.perplexity.ai/perplexitybot.json and www.perplexity.ai/perplexity-user.json (Perplexity).
Check Bot Fight Mode. Perplexity's docs note that Cloudflare Bot Fight Mode can't be bypassed with custom rules (Perplexity).
Check your server logs. Search them for the user-agent tokens (OAI-SearchBot, PerplexityBot) and look at the status codes those requests got.

Fixing robots.txt removes one blocker. It doesn't show you whether ChatGPT, Perplexity or Google actually name you. Our $49 AI Visibility Snapshot runs real buyer-style questions on those tools and shows who gets named instead, plus any crawl blocks we find.

Copy-paste robots.txt

This version allows the search crawlers and the training crawlers:

# Allow AI search crawlers (search/answers)
User-agent: OAI-SearchBot
Allow: /

User-agent: PerplexityBot
Allow: /

# Training crawlers: choose one. Allow shown; change to "Disallow: /" to opt out of training.
User-agent: GPTBot
Allow: /

User-agent: Google-Extended
Allow: /

# Everyone else
User-agent: *
Allow: /

Sitemap: https://yoursite.com/sitemap.xml

This version allows search but opts out of training:

# Allow AI search crawlers (search/answers)
User-agent: OAI-SearchBot
Allow: /

User-agent: PerplexityBot
Allow: /

# Training crawlers: opted out
User-agent: GPTBot
Disallow: /

User-agent: Google-Extended
Disallow: /

# Everyone else
User-agent: *
Allow: /

Sitemap: https://yoursite.com/sitemap.xml

Keep your private paths blocked. If you already disallow paths such as /app or /admin for *, repeat those Disallow lines inside each named group. A named group replaces the * group for that bot.
Why ChatGPT-User and Perplexity-User aren't listed. Both companies say robots.txt may not apply to these user-initiated fetchers, so a rule for them isn't a reliable control.

For a working example, see our own robots.txt, which allows GPTBot, OAI-SearchBot, ChatGPT-User, PerplexityBot, Google-Extended and others.

The Google-Extended myth

Blocking Google-Extended doesn't remove you from Google Search. Google says it "does not impact a site's inclusion in Google Search nor is it used as a ranking signal" (Google's common crawlers). It controls whether your content may be used to train future Gemini models and for grounding in Gemini Apps and Vertex AI.

AI Overviews are part of Google Search, and Googlebot rules apply to all Google Search features. So Googlebot, not Google-Extended, decides whether your pages can appear there. Don't block Googlebot to get out of AI Overviews unless you're willing to leave Google Search too.

After the fix: crawlable isn't the same as recommended

Allowing OAI-SearchBot makes your site eligible to appear in ChatGPT search. It doesn't make ChatGPT name you. That depends on your content, mentions on other sites, and more.

To see where you stand, follow how to check if ChatGPT recommends your SaaS. If you're also wondering about the other file AI tools read, see llms.txt: what it is and whether you need it. For the bigger picture, read how to get recommended by ChatGPT.

FAQ

What is OAI-SearchBot?

OAI-SearchBot is OpenAI's crawler that is used to surface websites in search results in ChatGPT's search features, according to OpenAI's crawler documentation.

What's the difference between OAI-SearchBot and GPTBot?

OAI-SearchBot is for ChatGPT search. GPTBot crawls content that may be used to train OpenAI's generative AI foundation models. OpenAI says the two settings are independent, so you can allow one and disallow the other.

If I block GPTBot, will I disappear from ChatGPT search?

No. ChatGPT search uses OAI-SearchBot, and OpenAI says each setting is independent. Blocking GPTBot opts you out of training, not search.

Does ChatGPT-User follow robots.txt?

OpenAI says robots.txt rules may not apply to ChatGPT-User, because a user starts the visit. OpenAI also says ChatGPT-User is not used to determine whether content may appear in search.

Is PerplexityBot used to train AI models?

Perplexity says no. PerplexityBot surfaces and links websites in Perplexity search results and is not used to crawl content for AI foundation models.

Does blocking Google-Extended remove me from Google Search or AI Overviews?

No. Google says Google-Extended does not impact a site's inclusion in Google Search and is not used as a ranking signal. AI Overviews are part of Google Search, so Googlebot rules govern them, not Google-Extended.

How long after I change robots.txt does ChatGPT notice?

OpenAI says it takes about 24 hours for its search systems to adjust after a robots.txt update. Perplexity says changes can take up to 24 hours.

My robots.txt allows OAI-SearchBot. Why am I still not in ChatGPT answers?

First check that a firewall or bot-protection setting isn't blocking the crawler. Beyond that, crawl access only makes you eligible to appear. It doesn't guarantee that ChatGPT will mention you.

See a sample audit or pricing.

Next steps: run the free AI-visibility check on your site, compare audit options and prices, or read llms.txt explained.