Hello, Robots

The broad question is close to a sure thing for WordCamp Canada: it has “WordPress events” and “Canada” in it, and so does the event’s name. Change who is asking and the shortlist changes. A freelance developer in Vancouver looking for other WordPress people mostly got the local meetup, WPYVR, which came up in every answer. Google AI Mode named WordCamp Vancouver, a different event, in all three of its answers, and never WordCamp Canada.

That’s why one answer to one question tells you very little, and why “we rank #1 on ChatGPT” isn’t a thing. What you can measure is how often you’re on the shortlist, per question and per engine, across many asks.

To see how AI treats a business, check six areas: reach, performance, identity, answers, corroboration and source pool.

This is the checklist I use when I audit a business. Reach and performance are technical, and they’re deterministic: the same check gives the same result every time, and the fix is on your own site. Can AI get to your pages at all, and once it does, do they load quickly with the content easy to reach?

The other four you have to measure, by asking AI the same questions again and again, and you only partly control them. Identity and answers depend on your own site: does AI know who you are, and can it find the answer to a buyer’s question there? Corroboration and source pool depend on what everyone else publishes about you.

The practice sections on this page follow that order: check your access, ask the AI, write your llms.txt, map the sites it cites, then pick your fixes.

Getting from your server into an AI recommendation is a chain: allowed, fetched, indexed, retrieved, cited, mentioned, recommended.

The first three steps are your site, and you can inspect them directly. That’s the access check. The last four happen inside the answer, and you can only measure them by asking the AI. Each step needs its own evidence; passing one doesn’t prove the next.

Keep three outcomes apart when you write down what you see. Mentioned: your name appears somewhere in the answer. Recommended: you’re one of the main picks. Cited: one of your pages is a source. You can be cited without being recommended, and recommended without being cited.

• Blocking the wrong bot. Each AI company runs separate bots for search, training and live fetches, and each one obeys its own robots.txt rule. Block a training bot and future models won’t learn about you from your site. Block a search bot and you drop out of that engine’s answers. A single “block AI” rule usually does both (OpenAI; Anthropic; Perplexity).
• robots.txt says yes, but the CDN, host or firewall says no. Bot protection sits in front of WordPress and can refuse a request that robots.txt invites in. Since Sep 15, 2026, Cloudflare’s “Block” setting also stops Googlebot, Bingbot and Applebot; to refuse training but stay in search, choose “Disallow AI Training” instead (Cloudflare, Sep 2026). Managed hosts and security plugins can also block bots at the server, before WordPress sees the request.
• A 200 that isn’t your page. The server answers “OK”, but what it sends is a bot challenge, a cookie wall, an empty page-builder shell or a maintenance-mode screen. The bot files away a page with nothing useful on it, and nothing in your dashboards looks wrong.
• Too slow or too heavy. Crawlers don’t wait as long as people do, and they stop reading at a fixed size. Googlebot reads the first 2 MB of a page’s HTML, uncompressed, and indexes it as if that were the whole page; anything after the cutoff is never seen (Google). Page builders can push a page past that with inline styles, scripts and embedded images, and your facts are often near the bottom. Other AI crawlers set their own limits and don’t publish them. Keep the HTML lean, put heavy CSS and JavaScript in separate files, and make the page answer quickly.
• Crawlable, but not indexed. In WordPress, the classic culprit is Settings → Reading → “Discourage search engines from indexing this site”, left on after launch. Since WordPress 5.3 it adds a noindex, nofollow robots tag to every page and turns off the core sitemap; it no longer touches robots.txt, so bots can still fetch pages they’re told not to index (WordPress, 2019; WordPress). SEO plugins can also noindex whole post types or archives.

Let every search bot in. Unless you have a reason not to, let the training bots in too.

AI companies run three kinds of bots, and robots.txt treats each one separately.

Search bots build the index an engine searches when someone asks it a question: OAI-SearchBot for ChatGPT, Claude-SearchBot, PerplexityBot, and Googlebot and Bingbot for Google’s AI answers and Copilot. Block one and that engine can’t find your pages, so it can’t cite or recommend you from them. If you want to be found at all, allow every one.

Training bots collect pages to train future models: GPTBot, ClaudeBot, Google-Extended and Applebot-Extended. A model that learned about you in training can name you without running a search. If you sell something, that’s exposure you get for free. I recommend allowing them unless you have a legal or contractual reason not to, or content you don’t want a future model to know.

User fetchers open a page live, when someone asks about it or pastes your link: ChatGPT-User, Claude-User and Perplexity-User. They act for one person, so some don’t follow robots.txt at all.

Blocking GPTBot doesn’t take you out of ChatGPT search; OAI-SearchBot decides that (OpenAI). Blocking Google-Extended doesn’t take you out of AI Overviews or AI Mode, which follow Googlebot (Google). Applebot-Extended doesn’t crawl at all; disallowing it opts you out of Apple’s training without affecting search (Apple).

Where WordPress decides what bots see

• The robots.txt WordPress generates. With no physical file, WordPress serves a virtual one: User-agent: *, Disallow: /wp-admin/, Allow: /wp-admin/admin-ajax.php, plus a Sitemap: line since 5.5. SEO and security plugins can add to it. A physical robots.txt file replaces all of it, including what plugins add (WordPress).
• WordPress.com. Settings → Reading → Site visibility → “Prevent third-party sharing” keeps your content out of Automattic’s AI licensing deals and adds known AI bots to your robots.txt disallow list. It switches on automatically if you discourage search engines (WordPress.com).
• Cloudflare in front of WordPress. Check Security → Settings: Search, Agent and Training bots each have their own setting. Its managed robots.txt disallows training bots such as GPTBot, ClaudeBot and Google-Extended but not the search bots (Cloudflare; Cloudflare, Sep 2026).
• Your host. Managed hosts can block or rate-limit AI crawlers at the server, before WordPress sees the request. Ask, or check your host’s knowledge base for “AI crawlers”.

Google’s and Apple’s crawlers run JavaScript; most other AI crawlers don’t.

When Vercel studied AI crawler traffic, OpenAI’s, Anthropic’s and Perplexity’s crawlers fetched JavaScript files but never ran them: they read only the HTML your server sends (Vercel). The page itself usually comes through. What goes missing are the parts that load by script: some page-builder sections, booking, menu and hours widgets, ticket embeds and review carousels.

The quick test: open your page, choose View source (Ctrl/Cmd+U) and search for a fact a buyer would ask about: your price, your hours, your address. If it isn’t in the source, most AI crawlers don’t see it. Facts in images, PDFs, tabs, pop-ups or behind a login have the same problem.

llms.txt is a map of your site written for AI: a short markdown file at your root that says what you do and where to find the details.

Google says it doesn’t use it for Search (Google), but other AI engines, like ChatGPT, do read it. In our clients’ server logs, GPTBot and ClaudeBot request it regularly. It takes a few minutes to write if you know your site well, and it has no downside, so I recommend it to every client.

Write it by hand. SEO plugins can generate one for you, but the automatic file is usually a list of your latest posts with their titles, which tells AI very little about what you do or where to find it. I don’t recommend leaving it at that.

You’ll write yours after asking the AI, once you know which questions it should answer and which facts it should state plainly.

A good one starts with your name and a plain description of the business: what you offer, who it’s for, where you work, and what you can and can’t do. Then short sections linking to the pages that answer buyer questions: services, pricing, locations and hours, FAQs, comparisons. Give each link a line saying what that page answers, so an engine knows which page to open for which question. Keep it a map rather than a dump of every URL, and keep it current; a stale llms.txt is worse than none.

How to publish it on WordPress

• Yoast SEO (free, since 25.3): Yoast SEO → Settings → Site features → AI tools → llms.txt. It writes the file and regenerates it weekly; you can pick the pages yourself instead of the automatic selection (Yoast).
• Rank Math (free, since 1.0.250): turn on the LLMS Txt module, then edit it under General Settings. It stays empty until you choose what goes in (Rank Math).
• All in One SEO (since 4.8.4): All in One SEO → Sitemaps → LLMs.txt. llms-full.txt and markdown copies of posts need Pro (AIOSEO).
• By hand. Upload a file named llms.txt to the folder that holds wp-config.php, by SFTP or your host’s file manager. That’s what the generator further down gives you.

Markdown copies of your pages

An agent reading your page has to dig the content out of menus, scripts, cookie banners and page-builder markup. A markdown copy is the same page with that stripped away: headings, text, lists, tables and links. Agents can ask for it by sending Accept: text/markdown, or by opening a .md address next to the page. It costs them far fewer tokens, so they read more of it and are less likely to get it wrong.

It must say the same thing as the HTML page. Same facts, same prices, same wording. Don’t stuff it with keywords or add claims that aren’t on the page; a copy that disagrees with your page is the kind of contradiction that makes AI less sure about you.

You can let a tool generate it. Cloudflare’s Markdown for Agents converts each page on the fly when an agent asks for markdown (Cloudflare), and All in One SEO Pro writes markdown copies of posts (AIOSEO). Writing your key pages by hand gives you more control over how things get read. A conversion drops what it can’t turn into text, so an image, chart or diagram becomes a file name or nothing. By hand, you can describe what the chart shows, write out the numbers behind it, and turn a pricing graphic into a table.

Each AI engine searches its own index, and Google Search Console only shows you Google’s.

Google’s AI Overviews, AI Mode and Gemini draw on Google’s index. Copilot draws on Bing’s: Microsoft describes Bing search, Copilot and its grounding API as one crawling and indexing pipeline (Bing). ChatGPT draws on Bing too: OpenAI says ChatGPT search sends queries to Bing for web results (OpenAI), alongside other providers and its own crawler, OAI-SearchBot. A page Google ranks well can still be hard for Bing to find.

We asked five engines “Is WordCamp US worth it, or is there a good WordPress conference in Canada?” three times each. ChatGPT, Gemini, AI Overviews and AI Mode named WordCamp Canada every time, and almost always cited canada.wordcamp.org. Copilot never cited it. Every page it found was about WordCamp US, so all three answers sent Canadians to “regional WordCamps” instead. The one answer that mentioned WordCamp Canada said it doesn’t exist: “Canada does not have a single national “WordCamp Canada,” but it has strong regional WordCamps that run most years.”

Check Bing too

• Verify your site in Bing Webmaster Tools. Its AI Performance report shows which of your pages Copilot and Bing’s AI summaries cite, how that changes over time, and the grounding queries behind the citations (Bing). Google has no equivalent report.
• Turn on IndexNow. It tells Bing the moment a page is added, changed or removed; Bing says this reduces outdated URLs in Copilot answers (Bing). Yoast SEO Premium does it automatically (Yoast, 2022), and Rank Math has an IndexNow module (Bing, 2022). Google doesn’t use IndexNow, so keep your sitemap for Google.

Measure AI visibility with a fixed set of questions, built from real buyer language.

Start from the questions your customers actually ask, in their own words: sales calls, support tickets, reviews, community threads, Search Console queries. Cover the whole decision: the problem, the category, comparisons, price and fit, and trust.

Ask unaided questions first, because branded questions flatter you. Keep roughly 70% unaided, with no names, to test whether you’re in the picture at all, and 30% branded, to check that AI describes you accurately. Measure intents, not exact strings: different wordings of the same need are one question. Then freeze the set, so next month’s numbers compare with this month’s.

A full audit uses 50 or more questions. To start, ten is enough to see trends: two from each part of the decision, each asked on more than one engine.

Branded questions are where AI is usually right, and where a wrong answer hurts most. AI rarely invents facts anymore. It repeats what it finds, and once a wrong fact shows up in a few places, like an old price, a former address or a service you dropped, it tends to stick.

Mark each answer: right, partly right, wrong, or about someone else with a similar name. For every wrong one, open the sources it cites and find where the bad fact comes from.

Then fix it at the source. If it’s your own page, update it. If it’s a directory listing or a profile you control, edit it. If it’s someone else’s site, ask them to correct it. When you can’t, publish the right fact on your own site with a clear publication date, so there’s a current page that says otherwise.

This wrong answer came from the event’s own website: its page from a prior year.

All 3 Copilot answers built their pricing from the 2024 and 2025 ticket pages. None mentioned that this year's event is in Vancouver, Nov 5–6, where a regular ticket is $100. The 2024 tickets page is still live and still says $75. It has a “WordCamp Canada 2024 is over” banner at the top, and Copilot used it anyway. Open the page and see.

WordPress keeps everything: last year’s event, an old price list, a service you stopped offering. Update old pages so their facts are plainly dated (“In 2024, tickets were $75. 2026 tickets are $100”) and link to the current version, redirect or unpublish what nobody needs, and let Bing know with IndexNow when you do.

Not every wrong answer has a page behind it. Several engines said WordCamp tickets are “typically $20–$50”: a general figure for WordCamps, not this event’s price. When your own page doesn’t state a fact plainly, AI fills the gap with the category default.

Related things need their relationship spelled out, too. WordCamp Canada and WordCamp Vancouver are separate events, and the engines kept them apart. But several still sent a Vancouver freelancer to WordCamp Vancouver as if one were coming up; Copilot said “Vancouver typically hosts one.” Gemini got it right: “Rather than a standalone city-focused WordCamp Vancouver this year, the city was selected to host the national conference.” One plain sentence on your own site gives AI the right answer to quote.

Before you count anything, decide three things: how many times you’ll ask each question, what you’re recording, and which names count as you.

• Too few samples. AI answers change every time you ask, so one answer tells you very little. Two or three runs across your whole question set show a trend. To know where you stand on one question, ask it 10 to 20 times. To prove that a small change moved one question, you need around 100. Then sort each question and engine into always, sometimes and never, and start with the sometimes: you’re already close on those.
• Too much blending. Being the main pick, being mentioned in passing and being cited as a source are three different results, and each engine behaves differently. One blended score can hold steady while ChatGPT stops recommending you and Gemini starts citing you. Record each result separately, per engine. That’s why the tracker below has three boxes.
• One business, many names. Answers refer to you in more ways than your official name: a short form, a nickname, last year’s name, your social handle or your domain. Count each of them as you, and keep the same list for each competitor. Keep separate businesses apart too: WordCamp Vancouver is a different event from WordCamp Canada, even in a year when both are in Vancouver. Without that list, a change in wording looks like a drop. As many as 36% of apparent drop-offs are the same company under another name (Detailed, Sep 2026).

Your own site may be the single most cited source, but most of what AI cites lives somewhere else. Map which other sites it cites for your questions.

WordCamp Canada’s own site led the chart above, and still accounted for only 4 in 10 of the 736 citations. 162 of the 205 answers cited at least one other site, and only 40 cited the event’s site alone. Those other sites shape the answer as much as yours does.

For each question in your set, list the domains that get cited. That’s your source pool. In one audit, the competitors in the client’s brief appeared in 2–3% of answers, while two others appeared in 22–30%.

Get described accurately on the sites engines already cite, with a link back to you: directories, review sites, event listings, roundups, community threads, and for WordPress businesses, your wordpress.org plugin or theme page and your Make WordPress profile. Brands that get mentioned more across the web tend to show up more in AI answers (Ahrefs, Dec 2025).

Then turn citation gaps into content gaps: what do the cited pages answer that yours don’t? Publish what others want to cite: comparisons, pricing, your own data.

So far, AI has read your site and told a person about you. Next, an agent visits for that person and does the task: books the call, buys the product, asks for the quote.

Picture someone asking their assistant to book a consultation with a WordPress agency next week. Today the agent reads Cedar & Co.’s pages, finds the booking form and fills it in, guessing at fields and buttons as it goes. That works some of the time. If the site tells the agent plainly that there’s a booking action and it needs a date and a name, the agent can call it directly, and WordPress checks that it’s allowed before anything happens.

WordPress is building exactly that:

• Abilities. Since WordPress 6.9, plugins can list the actions they offer, each with its own permission check (WordPress, Nov 2025). Yoast SEO already ships some (Yoast, Oct 2026), and WooCommerce has a preview (WooCommerce, Jun 2026).
• The MCP Adapter. An official plugin that lets AI assistants such as Claude and ChatGPT use those abilities as tools (WordPress).
• WebMCP. The same idea in the browser: a page offers “book a consultation” to the browser agent, which calls it instead of clicking through the form. It’s still a trial in Chrome and Edge (Chrome), and the browser in the ChatGPT desktop app already uses it (OpenAI).

The agent still has to reach your site, understand it and trust what it finds. Everything earlier in this workshop is the groundwork for this too.

• Everyone. Keep your booking, pricing and contact pages plain and current, and keep your plugins updated. Only connect agents you trust: they act with the permissions of the account you connect them to.
• Stores and booking sites. Check whether your shop or booking plugin offers abilities yet, and watch its changelog.
• Plugin developers. Register abilities for your plugin’s main actions, so agents can use them without guessing.

• Burying the answer. Engines quote claims they can lift from the text. If your answer is only implied, they take it from someone else’s page. That was 48% of the 3,244 WordPress sites we scanned: SEO-complete, answer-light.
• Schema. Yoast, Rank Math and AIOSEO already output a schema graph for your site. It labels facts for machines; it clarifies what’s there and doesn’t boost you by itself (Google; Ahrefs, May 2026). Fill in the organization or local business details in your SEO plugin, and keep them identical to what the page says.
• Old pages. WordPress never forgets: last year’s event page, the 2019 pricing post, the service you stopped offering. Update them, redirect them or unpublish them, and put dates on the facts you keep.

Checking a few questions by hand is the right way to start. Doing it every month, on five engines, several times each, is where it stops being practical. I built Librarium for that part.

• One question, every engine. Ask ChatGPT, the Gemini app, Google AI Mode, AI Overviews and Copilot at once, and add search and deep-research APIs such as Perplexity, Exa or Brave when you want a wider read. You pick the sources by name.
• Keep the evidence. Each answer is stored with the pages it cited. When an engine gets something wrong about you, you can open the page it took it from. That’s how the 2024 tickets page behind Copilot’s wrong price turned up.
• See the cost before it runs. A free preview shows which sources would be called and what each would cost, before anything is sent. You use your own provider keys, so the spend is yours to set.
• Use it where you work. Run it from a terminal, hand it to Claude Code, Codex or Cursor as a tool, or add it to a JavaScript or Laravel app. It’s free and open source (MIT).

Agentsy Report collects with it. The WordCamp Canada numbers in this workshop are 205 answers from five engines, each one stored with its citations.