If an AI engine cannot read your page, it cannot mention you. The first thing to check is whether your own robots.txt is quietly turning the AI crawlers away.
It is also one of the cheapest fixes on the list. A single file decides whether ChatGPT, Claude, Gemini, and Perplexity get to see the pages your buyers ask about, or whether they fall back to whatever a directory says about you instead.
What it is
robots.txt is a plain text file at the root of your domain (yourdomain.com/robots.txt). It tells crawlers which paths they are allowed to read. It has been around for decades for search engines. The new part is that AI companies now run their own crawlers, each with its own name (its “user agent”), and you can allow or block them by name.
Two things to know before you touch it:
- It is an honor system. Well-behaved bots from OpenAI, Anthropic, Google, Perplexity, and Common Crawl all say they obey
robots.txt. It is not a lock on the door; it is a sign on it. Reputable AI crawlers follow the sign. - There is no single “AI” switch. Each company uses a different name, and most run more than one crawler for different jobs.
Why AI engines care
AI crawlers usually do one of two jobs, and they matter for different reasons.
- Training crawlers read your pages to help train the model itself. These are
GPTBot(OpenAI),ClaudeBot(Anthropic),Google-Extended(Google’s Gemini and Vertex), andCCBot(Common Crawl, the open dataset most models learn from). - Live retrieval and search crawlers read your pages in the moment, to answer a question or build the search index behind an AI answer. These are
OAI-SearchBotandChatGPT-User(OpenAI),Claude-SearchBotandClaude-User(Anthropic), andPerplexityBot(Perplexity).
Live retrieval crawlers are what put you in today’s answers, with a citation. If you block those, you can be left out of the response a buyer is reading right now, even if your content is excellent. Blocking the training crawlers is a separate decision about whether your content helps shape future models. You can allow one group and not the other; they are controlled by name, separately.
How to set it up
Find your file at yourdomain.com/robots.txt. If it returns a 404, you do not have one yet, and crawlers assume everything is allowed. The risk is usually the opposite: an old Disallow rule, or a broad User-agent: * block, that sweeps the AI crawlers up with everything else.
To make your intent unambiguous, list the AI crawlers and allow them explicitly:
# AI training crawlers
User-agent: GPTBot
Allow: /
User-agent: ClaudeBot
Allow: /
User-agent: Google-Extended
Allow: /
User-agent: CCBot
Allow: /
# AI search and live-retrieval crawlers
User-agent: OAI-SearchBot
Allow: /
User-agent: ChatGPT-User
Allow: /
User-agent: Claude-SearchBot
Allow: /
User-agent: Claude-User
Allow: /
User-agent: PerplexityBot
Allow: /Keep any genuine Disallow rules you need (admin paths, checkout, internal search results). This is not about opening everything. The pages that explain who you are and what you sell are the ones that need to stay readable.
Three things people get wrong
Google-Extendedis notGooglebot. Allowing or blockingGoogle-Extendedchanges whether Google can use your content for Gemini training and grounding. Google has said it does not affect your normal Search inclusion or ranking. They are two different decisions.- User-triggered fetches play by different rules. When a person pastes your link into Perplexity or asks Claude to open it, that real-time fetch (
Perplexity-User,Claude-User) is treated as the user’s action, not a crawl. Perplexity-User in particular does not always honorrobots.txt, because a human asked for that exact page. - Blocking training does not block retrieval. If you disallow
GPTBotbut say nothing aboutOAI-SearchBot, you have opted out of training while staying in ChatGPT’s live search. That may be exactly what you want. Just be deliberate about it.
After you publish the change, reopen your Free Scan report and refresh the technical checks to confirm the crawlers you care about are allowed.
