AI crawler access

AI crawler access. Can AI assistants read your site?

Agentcites reads your robots.txt on every scan and says whether five AI crawlers are allowed, blocked or unknown. If one is blocked, you get the lines that let it back in.

Start your free scan See the price

postprism.app/robots.txt

1User-agent: *
2Allow: /
3
4User-agent: Google-Extended
5Disallow: /Allow: /

ChatGPTGPTBotAllowed
ChatGPT searchOAI-SearchBotAllowed
ClaudeClaudeBotAllowed
PerplexityPerplexityBotAllowed
GeminiGoogle-ExtendedBlockedAllowed

One line changed, and Gemini can read the site.

Illustrative.

Why this comes first

A page an assistant cannot read is a page it cannot cite. Every other piece of advice assumes the crawler got in. This check makes sure it did, before you spend time writing.

robots.txt is a plain text file at the root of your site. It lists which crawlers may read which paths. A crawler can be blocked without anyone meaning to, by a rule written for every crawler or by a setting a plugin added.

The five AI crawlers a scan checks, passing through a site's robots.txt to its pages. A crawler the file blocks stops at the gate.

The five crawlers it checks

The crawler names a scan looks for in your robots.txt, and what each one is for.

Crawler	Assistant	What it does
GPTBot	ChatGPT	Collects pages to train OpenAI’s models
OAI-SearchBot	ChatGPT search	Finds pages to show in ChatGPT’s search results
ClaudeBot	Claude	Collects pages to train Anthropic’s models
PerplexityBot	Perplexity	Finds pages to show and link in Perplexity’s search results
Google-Extended	Gemini	Whether Gemini may train on your pages or ground answers in them

Each description is from the company’s own crawler page, listed under Sources. Google-Extended does not affect Google Search, and AI Overviews are part of Search. Google says so on its own crawler page.

How the check works

On every scan, Agentcites fetches your robots.txt.
It works out, for each of the five crawlers, whether the rules let it read your home page.
Each crawler comes back as allowed, blocked or unknown. If the file cannot be fetched, the answer is unknown, never allowed.
For a blocked crawler it quotes the lines that block it, word for word. If the block comes from a rule for every crawler, it quotes that rule.
It gives you the two lines that let each blocked crawler back in, and only for the ones that are blocked.

Each scan fetches your robots.txt and reads its rules once for each of the five crawlers. Each crawler comes back allowed, blocked or unknown.

Each result means one thing.

What the check can find, and what each result means.

You see	What it means
Allowed	The rules let this crawler read your pages.
Blocked	A rule turns this crawler away. The lines are quoted so you can find them.
Unknown	The file could not be read. Treat it as a problem until it can be.

The result is on the To do screen, on its Your site tab, beside the list of work. Its heading says how many of the five crawlers can read your site. When one is blocked, the panel opens out to show the quoted lines and the lines to copy, and it tells you to fix the block before the list, because nothing you publish can be cited by that assistant until you do.

The same scan also checks whether you publish an llms.txt file and which sitemaps your robots.txt lists, and it counts how many of the pages it read carry structured data.

A site, its llms.txt file, and an AI system reading it. The scan notes whether the file is there. It does not claim the file changes what any assistant says.

Letting a blocked crawler back in

The fix is two lines added to your robots.txt for each blocked crawler. Agentcites writes them for you, and only for the crawlers that are blocked.

Open the robots.txt file at the root of your site, or the setting in your website builder that writes it.
Find the lines Agentcites quoted. They are the rule doing the blocking.
Add the two lines Agentcites gave you. For GPTBot they are "User-agent: GPTBot" on one line and "Allow: /" on the next.
Wait for the next scan. It reads the file again and shows the new result.

Blocking a crawler can be a deliberate choice, for example to keep your pages out of model training. The check reports what the file says. Whether to change it is your decision.

Find the quoted lines, add the two lines Agentcites gives you, and the next scan reads the file again.

What it does not do

It reads robots.txt only. A firewall or a bot-protection setting can still turn crawlers away when robots.txt allows them, and this check cannot see that.
It checks five crawlers. DeepSeek is one of the assistants we ask, and it has no crawler entry in the check yet. Anthropic also runs Claude-SearchBot and Claude-User, which the check does not look for by name.
It does not fetch any page your robots.txt forbids. A refusal is recorded as a fact, not worked around.
It has no separate button. It runs as part of every scan.

Answers from the assistants your buyers use

Each question goes to each assistant you choose, with web search on, the way a buyer would ask it.

ChatGPT
Perplexity
Gemini
DeepSeek

Questions

How do I know if ChatGPT can read my website?

Look in your robots.txt for rules naming GPTBot or OAI-SearchBot, or a rule for every crawler. Agentcites does this on every scan and quotes the lines that decide it.

Does blocking Google-Extended remove me from AI Overviews?

No. Google says Google-Extended does not affect Google Search, and AI Overviews are part of Search. It controls whether Gemini may train on your pages or ground answers in them.

If every crawler is allowed, will the assistants cite me?

Not on its own. Being allowed in only means they can read your pages. Whether they name you depends on what those pages say and on what other sites say about you.

Sources

See it for your own site

We ask 20 of your questions to four AI assistants, and you see every answer before you pay anything. You do not need a card.

Start your free scan All features