Free AI crawler checker
Can AI assistants read your site?
Type your website. We read its robots.txt and show which AI crawlers may read your home page. It is free and needs no account.
How robots.txt lets a crawler in
- 1User-agent: *
- 2Allow: /
- 3
- 4User-agent: Google-Extended
- 5Disallow: /Allow: /
- ChatGPTGPTBotAllowed
- ChatGPT searchOAI-SearchBotAllowed
- ClaudeClaudeBotAllowed
- PerplexityPerplexityBotAllowed
- GeminiGoogle-ExtendedBlockedAllowed
One line changed, and Gemini can read the site.
Illustrative.
An AI assistant can only cite a page its crawler was allowed to read. Each assistant's crawlers are explained, from its provider's own documents, on the engines page.
What the check reads
What it checks
- We fetch your robots.txt, the file that tells robots which pages they may read.
- For five AI crawlers, we work out whether that file lets them read your home page. The table quotes the lines that decide it.
- If a crawler may read your home page but is shut out of other paths, it says so.
- We also look for an llms.txt file, a proposed summary of a site written for AI systems.
What it cannot see
- It reads robots.txt, which is a request and not a lock. A firewall or a CDN can block a crawler that robots.txt allows, and we cannot see that from outside.
- It checks your home page. A rule for one folder shows up as "some paths blocked", not as a verdict for every page.
- It does not tell you whether an assistant names you. Being readable is the first step, not the result.
Allowed in. Now, are you named?
Agentcites asks AI assistants the questions your buyers ask and keeps every answer word for word. Your first scan asks 20 questions of four AI assistants. It is free and needs no card.
Questions
Does blocking GPTBot keep me out of ChatGPT?
Not by itself. OpenAI says GPTBot is for training, and ChatGPT search reads sites with OAI-SearchBot. The two are set separately. The ChatGPT page under Engines quotes OpenAI on this.
Does Google-Extended affect Google Search?
Google says it does not. Google-Extended is a control for Google’s AI models, such as Gemini, and Google Search reads sites with Googlebot. The Gemini page under Engines quotes Google on this.
What happens to the address I type?
We fetch two files from that site, robots.txt and llms.txt, as AgentcitesBot. We do not save the address or the result. To stop one visitor running thousands of checks, we hold your IP address in memory for ten minutes.
My site has no robots.txt. Is that a problem?
Not for this. Crawlers that follow the standard read every page of a site with no robots.txt. You only need one if you want to keep something out.