Content Signals (robots.txt)

Allowing a crawler in and agreeing to how it uses what it reads are two different things. Content Signals is a small line you add to robots.txt that states the second part: once an AI system has your page, what is it allowed to do with it.

What it is

Content Signals is a policy that Cloudflare published on September 24, 2025, and released into the public domain. It adds one machine-readable line to your existing robots.txt, the Content-Signal line, that separates whether a bot may crawl you from how your content may be used afterward.

There are three signals, and each is set to yes or no:

• search: building a search index that returns links and short snippets to people. This deliberately does not cover AI-generated summaries.
• ai-input: feeding your content into an AI model to generate a live answer, the retrieval step behind most AI answers today.
• ai-train: training or fine-tuning an AI model on your content.

Setting a signal to yes or no states a preference. Like robots.txt itself, it is a clearly worded sign, not a technical lock. Its value is that it makes your intent unambiguous and machine-readable, and it gives you a documented position if a crawler ignores it.

Why AI engines care

Until now, your only lever was all-or-nothing: let a crawler in, or shut it out. That forces a bad trade. Block the crawler and you vanish from AI answers. Allow it and you have implicitly agreed to every use, including training.

Content Signals lets you keep the door open for the use you want and say no to the one you do not. A common stance for a business that wants to be cited but not used as training data: yes to search, yes to AI input, no to AI training.

How to set it up

The line goes inside a User-agent group in your robots.txt, alongside your Allow and Disallow rules. List only the signals you have an opinion about:

User-agent: *
Content-Signal: search=yes, ai-input=yes, ai-train=no
Allow: /

Read that as: anyone may crawl me; you may use my content for search and for live AI answers; please do not use it to train models.

If your site is behind Cloudflare, you do not have to hand-edit the file. Cloudflare’s managed robots.txt feature can add the Content-Signal line for you, and on free plans without an existing robots.txt it will serve the full policy text, with the actual signal values left for you to choose. The generator and the canonical wording live at contentsignals.org.

Three things to keep straight

• It is a preference, not enforcement. Content Signals tells well-behaved systems your terms. It does not stop a bad actor. Cloudflare frames it as the machine-readable half of your terms of use.
• search=yes does not mean AI summaries. The search signal is scoped to classic index-and-link search. If you want to allow AI answer generation, that is the separate ai-input signal.
• Keep it consistent with your crawler rules. Saying ai-train=no while leaving GPTBot and CCBot fully allowed is a valid combination, but make sure it reflects what you actually intend. Pair this guide with your AI crawler access setup.