Crawler
AgentcitesBot reads public pages for Agentcites customers.
Agentcites shows a business which questions AI assistants answer by naming a competitor, and which pages those assistants cited. To do that, it reads the customer's own website and the public pages the assistants pointed to.
How it behaves
- It identifies itself as AgentcitesBot in every request, with a link to this page.
- When it reads pages for an audit, or a page an AI assistant cited, it obeys robots.txt. A page robots.txt disallows is not read.
- For those reads it requests one page at a time from any one site, with a pause between requests.
- When a customer sets up Agentcites, it reads up to six pages of the website that customer gave us, a few at once, to learn what the business sells.
- When someone uses the free AI crawler checker on this site, it fetches two files from the website they typed: robots.txt and llms.txt. It reads llms.txt only if robots.txt allows it.
- It reads public pages only. It does not log in, submit forms, or run scripts.
How to block it
Add these two lines to your robots.txt file. It will stop reading your pages for audits and citations.
User-agent: AgentcitesBot Disallow: /
If you have a question about it, or think it misbehaved, email [email protected].