robots.txt for AI crawlers: GPTBot, PerplexityBot, ClaudeBot and the rest

What robots.txt can and cannot do for AI crawlers

robots.txt is a request, not a lock. A well-behaved AI crawler reads it and stays out of what it is told to avoid, but the file cannot stop a crawler that chooses to ignore it.

The file is a plain text list of rules at the root of your site, at /robots.txt. Each group of rules starts with one or more User-agent lines naming a crawler, followed by Allow and Disallow lines for paths. The format is a published standard, RFC 9309, and the standard says plainly that "these rules are not a form of access authorization".

That matters for AI crawlers in two ways. First, a refusal only works on crawlers that honour it, and every vendor below says its named crawlers do. Second, some AI fetchers are started by a person asking a question, and several vendors say those fetchers may not follow robots.txt at all. If you need a page kept from every machine, you need a login or a server rule, not a line in a text file.

The AI crawler names, vendor by vendor

There are about a dozen names worth knowing, and each vendor documents its own. The table below quotes or closely follows each vendor page, read on 10 October 2026. The name in the first column is the exact token you write after "User-agent:".

DeepSeek is missing from the table on purpose. We could not find a crawler page published by DeepSeek naming a user-agent token, so we have nothing checkable to list. Some third-party bot directories list a name such as DeepSeekBot, but none we read linked to a DeepSeek page confirming it. Treat that name as unverified until the company documents it.

Training, search and user fetches are three different switches

The most useful thing to know is that the vendors split their crawlers by job. Refusing training does not refuse search, and refusing search does not refuse training. OpenAI says of its crawlers that "each setting is independent of the others".

• Training crawlers collect pages that may go into a future model. GPTBot, ClaudeBot, Google-Extended and CCBot are in this group. Refusing them affects what future models learn, not what a search engine shows today.
• Search crawlers build the index an AI search feature reads when it answers. OAI-SearchBot, Claude-SearchBot, PerplexityBot and Googlebot are in this group. Refusing them is how a page drops out of those answers and their links.
• User fetchers visit one page because one person asked. ChatGPT-User, Claude-User, Perplexity-User and Google user-triggered fetchers are in this group. OpenAI, Perplexity and Google say these may not follow robots.txt.

So the question to answer before you edit the file is which of the three jobs you want to allow. A business that wants to be named in AI answers but would rather not feed training has a clear answer: allow the search crawlers and refuse the training ones. A publisher that sells its archive may choose the opposite. Both are reasonable. What is not reasonable is a file that does one by accident.

Example robots.txt lines for common choices

Below are three ready-made sets of lines. Each table is one file, one line per row, in order. A blank line separates groups. Copy only the lines, not the explanations.

A crawler that finds a group with its own name follows only that group and ignores the rules under "User-agent: *". Repeat every rule you still want it to obey.

Mistakes that block AI crawlers by accident

Most AI crawler problems are not decisions. They are side effects of a rule someone wrote for another reason. These are the ones to look for first.

1. A rule for every crawler. "User-agent: *" followed by "Disallow: /" refuses every crawler that has no group of its own, including every AI crawler you did not name. It is often left over from a staging site.
2. A named group that forgets the shared rules. RFC 9309 says a crawler uses the groups that match its name, and falls back to the star group only when none match. Adding "User-agent: GPTBot" with "Allow: /" quietly lets GPTBot into folders the star group kept closed.
3. A robots.txt that returns a server error. RFC 9309 says that when the file is unreachable because of a server error, the crawler must assume complete disallow. A file that times out or returns a 500 error blocks everyone, and it looks fine when you open it in a browser later.
4. A firewall or bot-protection setting. A network service can turn AI crawlers away before they ever read robots.txt. The file says yes and the server says no, and only your server logs show it.
5. Blocking Googlebot to stay out of AI Overviews. Google says AI Overviews draw on the Search index, so blocking Googlebot removes the page from Search as well. Google points to nosnippet, data-nosnippet, max-snippet and noindex as the controls for what Search shows.
6. Blocking Google-Extended to stay out of AI Overviews. Google says Google-Extended does not affect inclusion in Google Search, and AI Overviews are part of Search. It controls Gemini training and grounding only.

Two smaller rules from the standard are worth knowing. Paths are matched by the longest rule, so "Allow: /blog/" beats "Disallow: /" for pages under /blog/. And when an Allow rule and a Disallow rule match equally, the standard says Allow wins.

How to check that the change worked

A change to robots.txt is not finished until you have seen it from the outside. Crawlers keep a copy of the file for a while, so the effect is not instant.

1. Open https://your-domain/robots.txt in a private browser window and confirm it shows the new lines. A cached old copy at a CDN is a common surprise.
2. Confirm the file returns a normal success status, not a redirect loop or an error. A server error counts as a full block under the standard.
3. Read each group as a crawler would. Pick a crawler name, find the groups that name it, and ignore the star group if any do.
4. Wait at least a day. RFC 9309 says crawlers should not use a cached copy for more than 24 hours, and OpenAI says its systems take about 24 hours to adjust.
5. Search your server logs for the user-agent names in the table above. A refused crawler should stop requesting refused paths. A user fetcher may keep appearing, as its vendor says.
6. Check that a visitor claiming a name really is that crawler. Common Crawl warns that other crawlers falsely identify themselves as CCBot, and suggests checking the request against its published address ranges.

Allowing AI crawlers is necessary but not enough

Being allowed in only means an assistant can read your pages. Whether it names you depends on what those pages say and on what other sites say about you. A site with a perfect robots.txt and nothing worth quoting still loses the answer. The engines page explains how each assistant we ask uses the web and which crawler reads your site for it.

Agentcites runs a version of this check on every scan. It reads your robots.txt and says, for five AI crawlers, whether each one is allowed, blocked or unknown, and quotes the exact lines responsible. The AI crawler access page lists the five and what the check cannot see, such as a firewall. It does not yet check every name in the table above. For the rest of what a scan measures, see how we measure.