AgentsyCaptureBot - Agentsy Report

If you found this page through your server logs, a request came from Agentsy Report. We measure how AI search engines describe a business, and part of that is reading the public pages of websites included in an Audit, usually the business's own site. This page explains what the crawler does and how to control it.

How it identifies itself

Every request sends this user agent:

AgentsyCaptureBot/1.0 (+https://agentsy.report/bot)

We never send another crawler's user agent. Requests come from our own servers, not from visitors' browsers. Where a site uses Cloudflare, requests may also carry a Web Bot Auth signature (HTTP Message Signatures) that proves they came from us. Our public key directory is at platform.agentsy.report/.well-known/http-message-signatures-directory.

What it fetches

It reads a small, fixed set of public URLs for each site in an Audit: the homepage and a handful of key pages, plus robots.txt, llms.txt, sitemaps and similar files that tell AI systems about a site. It does not follow links across the web, submit forms, sign in, run scripts or download media. Responses are size- and time-limited.

Robots.txt and rate limits

It reads robots.txt before fetching anything else and follows the rules for the AgentsyCaptureBot token. If robots.txt is behind a bot challenge or returns a server error, it fetches nothing else from that site. It sends at most one request per second to any site, honors Crawl-delay, and slows down automatically after a 429, a 503 or a Retry-After header.

To block it, add this to your robots.txt:

User-agent: AgentsyCaptureBot
Disallow: /

Agentsy Reference Bot

A second client, AgentsyReferenceBot/1.0, only calls the public Wikipedia and Wikidata APIs, following Wikimedia's API etiquette and rate limits. It does not visit other websites.

Agentsy Report Bot

The free Scan on agentsy.report identifies itself as AgentsyReportBot/1.0 (+https://agentsy.report). It runs only when someone requests a Scan or Audit of a website, and then fetches a bounded set of that site's public URLs: the homepage, robots.txt, llms.txt, sitemaps and, for an Audit, a few dozen same-site pages. Each request is size- and time-limited. It reads robots.txt to report on it, not to decide what it may fetch; to exclude your site, contact us below.

Contact

If our crawler is causing a problem, or you would like your site excluded, email hello@agentsy.build. Include your domain and, if you can, the time of the requests. Data we collect is handled as described in our privacy policy.