# AI crawler access (robots.txt)

> How to let the major AI crawlers read the pages your buyers ask about.

By Joey Kudish · Reviewed by Marius C. Paun · Published 2026-07-27 · Updated 2026-07-28

If an AI engine cannot read your page, it cannot mention you. The first thing to check is whether your own `robots.txt` is quietly turning the AI crawlers away.

It is also one of the cheapest fixes on the list. A single file decides whether ChatGPT, Claude, Gemini, and Perplexity get to see the pages your buyers ask about, or whether they fall back to whatever a directory says about you instead.

## What it is

`robots.txt` is a plain text file at the root of your domain (`yourdomain.com/robots.txt`). It tells crawlers which paths they are allowed to read. It has been around for decades for search engines. The new part is that AI companies now run their own crawlers, each with its own name (its "user agent"), and you can allow or block them by name.

Two things to know before you touch it:

- It is an honor system. Well-behaved bots from OpenAI, Anthropic, Google, Perplexity, and Common Crawl all say they obey `robots.txt`. It is not a lock on the door; it is a sign on it. Reputable AI crawlers follow the sign.
- There is no single "AI" switch. Each company uses a different name, and most run more than one crawler for different jobs.

## Why AI engines care

AI crawlers usually do one of two jobs, and they matter for different reasons.

- **Training crawlers** read your pages to help train the model itself. These are `GPTBot` (OpenAI), `ClaudeBot` (Anthropic), `Google-Extended` (Google's Gemini and Vertex), and `CCBot` (Common Crawl, the open dataset most models learn from).
- **Live retrieval and search crawlers** read your pages in the moment, to answer a question or build the search index behind an AI answer. These are `OAI-SearchBot` and `ChatGPT-User` (OpenAI), `Claude-SearchBot` and `Claude-User` (Anthropic), and `PerplexityBot` (Perplexity).

Live retrieval crawlers are what put you in today's answers, with a citation. If you block those, you can be left out of the response a buyer is reading right now, even if your content is excellent. Blocking the training crawlers is a separate decision about whether your content helps shape future models. You can allow one group and not the other; they are controlled by name, separately.

## How to set it up

Find your file at `yourdomain.com/robots.txt`. If it returns a 404, you do not have one yet, and crawlers assume everything is allowed. The risk is usually the opposite: an old `Disallow` rule, or a broad `User-agent: *` block, that sweeps the AI crawlers up with everything else.

To make your intent unambiguous, list the AI crawlers and allow them explicitly:

```
# AI training crawlers
User-agent: GPTBot
Allow: /

User-agent: ClaudeBot
Allow: /

User-agent: Google-Extended
Allow: /

User-agent: CCBot
Allow: /

# AI search and live-retrieval crawlers
User-agent: OAI-SearchBot
Allow: /

User-agent: ChatGPT-User
Allow: /

User-agent: Claude-SearchBot
Allow: /

User-agent: Claude-User
Allow: /

User-agent: PerplexityBot
Allow: /
```

Keep any genuine `Disallow` rules you need (admin paths, checkout, internal search results). This is not about opening everything. The pages that explain who you are and what you sell are the ones that need to stay readable.

## Three things people get wrong

- **`Google-Extended` is not `Googlebot`.** Allowing or blocking `Google-Extended` changes whether Google can use your content for Gemini training and grounding. Google has said it does not affect your normal Search inclusion or ranking. They are two different decisions.
- **User-triggered fetches play by different rules.** When a person pastes your link into Perplexity or asks Claude to open it, that real-time fetch (`Perplexity-User`, `Claude-User`) is treated as the user's action, not a crawl. Perplexity-User in particular does not always honor `robots.txt`, because a human asked for that exact page.
- **Blocking training does not block retrieval.** If you disallow `GPTBot` but say nothing about `OAI-SearchBot`, you have opted out of training while staying in ChatGPT's live search. That may be exactly what you want. Just be deliberate about it.

After you publish the change, reopen your [Free Scan report](/#hero-scan-form) and refresh the technical checks to confirm the crawlers you care about are allowed.