How to get your website to show up in ChatGPT: crawlers, robots.txt and firewalls

How to get your website to show up in ChatGPT

Your website shows up in ChatGPT when ChatGPT searches the web and finds pages it is allowed to read. The one permission that matters most is for a crawler called OAI-SearchBot.

A crawler is a program that visits web pages and reads them, the way Googlebot does for Google. OpenAI runs several, each for a different job. Block the wrong one and your site disappears from ChatGPT’s search answers, even if it ranks well in Google.

This guide covers the technical side: which crawlers exist, what to put in robots.txt, the firewall settings that can undo it, and how to check. If you want the content side, read how to get cited by ChatGPT.

Which OpenAI crawlers visit your site

OpenAI names four user agents, and only one of them decides whether you can appear in ChatGPT’s search answers. A user agent is the name a crawler gives when it visits.

OpenAI's own words on the search crawler: "Sites that are opted out of OAI-SearchBot will not be shown in ChatGPT search answers, though can still appear as navigational links." On the separation: "a webmaster can allow OAI-SearchBot in order to appear in search results while disallowing GPTBot" (OpenAI, Overview of OpenAI Crawlers).

On ChatGPT-User, OpenAI says it "is not used for crawling the web in an automatic fashion" and "is not used to determine whether content may appear in Search". So it is not the switch for search. It is the visitor you see when someone asks ChatGPT to open a specific page. Our ChatGPT page lists the same crawlers with OpenAI’s own wording.

What to put in robots.txt

robots.txt is a plain text file at the root of your site, such as example.com/robots.txt, that tells crawlers which pages they may read. To appear in ChatGPT search, it must not block OAI-SearchBot.

Each rule group starts with a "User-agent" line naming a crawler, followed by "Allow" or "Disallow" lines for paths. A group for "User-agent: *" applies to every crawler that has no group of its own. The table shows the lines for three common choices.

Timing matters. OpenAI says "it can take ~24 hours from a site’s robots.txt update for our systems to adjust" for search results (OpenAI). Check again the next day before deciding the change did not work.

Remember what robots.txt is not. Google's guide says "it is not a mechanism for keeping a web page out of Google", and that "it's up to the crawler to obey" the rules in it (Google Search Central). It also warns that different crawlers can read the same rules differently. Keep your file simple, and give each crawler you care about its own clear group.

Should you block GPTBot?

Blocking GPTBot does not remove you from ChatGPT search, so the choice is about training, not about search. It is a business decision with a cost on each side.

OpenAI treats the two crawlers separately, and adds one practical note: "If your site has allowed both bots, we may use the results from just one crawl for both use cases to avoid duplicative crawling" (OpenAI). Allowing both means fewer visits, not more.

• A reason to allow GPTBot: ChatGPT sometimes answers from what its model learned, without searching. A page the model never trained on cannot shape those answers. This is a reasoned guess, because OpenAI does not publish what each model was trained on.
• A reason to block GPTBot: you sell your writing, your data or your research, and you do not want it used to train a model without payment.
• A reason not to worry much: for most business sites, the search crawler decides whether you are cited, and that switch is separate.

Whatever you decide, write it down and make it a deliberate choice. Many sites block GPTBot because a plugin or a template did it, and nobody remembers why. A choice nobody made is the one most likely to be wrong.

The same goes for other assistants. Perplexity, Google and Anthropic each run their own crawlers with their own names and rules. Our pages on Perplexity and Gemini name theirs. A file that welcomes OAI-SearchBot but blocks the others leaves you visible in one assistant and missing from the rest.

The firewall can undo a correct robots.txt

A robots.txt that allows OAI-SearchBot does nothing if a firewall or bot-blocking service turns the crawler away first. This is the most common hidden block, and robots.txt checkers do not see it.

OpenAI recommends "allowing OAI-SearchBot in your site’s robots.txt file and allowing requests from our published IP ranges" (OpenAI). That second half is a firewall setting, not a robots.txt line.

Cloudflare, which sits in front of a large share of websites, changed its defaults twice. In July 2025 it announced it was "changing the default to block AI crawlers unless they pay creators for their content" (Cloudflare blog). Its current documentation sorts AI bots into Search, Agent and Training, and says that from 15 September 2026 new domains block Training and Agent bots on pages that show ads, while "Search will remain allowed" (Cloudflare, Block AI Bots).

Cloudflare also says mixed-purpose crawlers, which crawl for both search and training, are blocked by every setting that blocks AI training. If your site turned on an older "Block AI bots" option, check what it covers today.

Check two places, not one: your robots.txt file, and whatever sits in front of your site deciding which bots get through.

Being readable is not the same as being found

Once the crawler is allowed, your pages still have to turn up in the search ChatGPT runs. That means being indexed by the search providers it uses, and keeping your important text where a crawler can read it.

OpenAI's help page says ChatGPT rewrites a question into search queries and sends them to search providers, and lists Microsoft among them (OpenAI help). Microsoft's guidelines name the ways Bing discovers pages: "IndexNow URL submission", "XML sitemaps", "Crawlable internal links" and "External links from relevant websites" (Bing Webmaster Guidelines).

One outside study shows why Bing matters here. Seer Interactive matched ChatGPT search citations against Google and Bing results for the same questions, and found far more of them in Bing (Seer Interactive).

• Verify your site in Bing Webmaster Tools and submit your sitemap. Many sites only ever did this for Google.
• Turn on IndexNow if your platform supports it, so search engines hear about a changed page straight away.
• Do not hide key facts inside tabs, images or PDFs. Microsoft’s guide warns that "AI systems may not render hidden content".
• Make sure each important page is linked from another page on your site with an ordinary HTML link.

The quote about hidden content comes from Microsoft's guide to content for AI search (Microsoft Advertising). It was written about Bing and Copilot, and OpenAI has not published equivalent advice, but the reasoning applies to any crawler that reads raw pages.

A quick test helps here. Open one of your key pages, turn off JavaScript in your browser settings, and reload it. If the price, the product name or the answer to the main question disappears, a simple crawler may not see it either. Ask your developer to render that text on the server, so it arrives in the page itself.

How to check your site in ten minutes

You can test most of this yourself with a browser and a search. Work through these steps in order and stop at the first one that fails, because it blocks everything after it.

1. Open yoursite.com/robots.txt in a browser. If it does not load, ask your developer why.
2. Look for a group named OAI-SearchBot. If there is one, check it does not say "Disallow: /".
3. If there is no OAI-SearchBot group, read the "User-agent: *" group. A "Disallow: /" there blocks OAI-SearchBot too.
4. Ask whoever manages your hosting, CDN or firewall whether AI bots are blocked, and whether OpenAI’s published IP ranges are allowed.
5. Search for one of your page titles in Bing. If nothing from your site appears, submit your sitemap in Bing Webmaster Tools.
6. Wait a day after any change, then ask ChatGPT a question your page answers, with search on, and open the Sources panel.

Checking it on every scan

A robots.txt that is right today can be changed tomorrow by a plugin, a redesign or a new host. Checking it once is not enough.

Agentcites reads your robots.txt on every scan and reports, for five AI crawlers (GPTBot, OAI-SearchBot, ClaudeBot, PerplexityBot and Google-Extended), whether each is allowed, blocked or unknown. If the file cannot be fetched, the answer is unknown, never allowed. For a blocked crawler it quotes the exact lines responsible. It does not check ChatGPT-User or your firewall, so the firewall step above is still yours. The rest of what it does is on the features page, and how we measure explains every count.