Start here if your Scan flags access problems. Crawlers need to reach a working page before they can read its facts. These checks describe the pages and responses we sampled. They do not prove every AI crawler gets the same response.
Work on a staging copy when changing routing or indexing settings. Keep intentional private-page exclusions. After each change, check the response headers and the HTML without running JavaScript, then refresh your technical checks.
AI crawler access in robots.txt
robots.txt sets crawler access preferences. A block aimed at training can be intentional; a block aimed at search crawlers can keep useful pages out of the sources they collect. Check which bot and role your Scan names before removing a rule.
Edit only the relevant user-agent group and paths. Keep admin and private paths protected by authentication, not just robots.txt. Test the rules against a public page and an intentionally excluded path. Our crawler access guide goes through the bot names and examples.
Can crawlers read your pages?
A crawler may see a challenge screen, a refusal or very little text before scripts run. Our unsigned request is one observed client, not proof that a named AI crawler is blocked.
Check your firewall and bot rules before opening access broadly. Put the business name, offer and useful proof in the initial HTML. Test a normal unsigned request and inspect what actually comes back. See crawler reachability for the distinction between response failures and script-only content.
Homepage response time
This check measures how soon the homepage HTML arrives. It does not measure Largest Contentful Paint or how quickly the browser draws the page. A slow response makes every later step wait.
Check server logs for slow database work, redirects and uncached template rendering. Cache suitable public responses and move unnecessary work out of the response path. Compare several requests before and after. Keep personalized pages private; a faster response is not worth leaking customer data.
Pages allowed in search
noindex in page metadata or an HTTP response header asks a search engine to leave a page out of its index. That is appropriate for some pages, but not a service or proof page you want buyers to find.
Remove accidental exclusions in the template or server configuration. Check both the HTML robots tag and X-Robots-Tag. Do not remove deliberate exclusions from account pages. Google’s noindex documentation explains why a crawler must be allowed to read the rule.
Canonical URLs
A canonical declaration points to the preferred URL for equivalent content. Missing, conflicting or wrong targets make it harder for search engines to consolidate a page’s signals.
Use one absolute HTTPS URL that serves the intended page. Align HTML canonical tags and HTTP Link headers. Do not point every page at the homepage. Check translated and paginated pages separately. Follow Google’s canonical guidance, then verify the preferred page and its alternatives.
Duplicate URL variants
HTTP, HTTPS, host and path variations can expose the same page at several addresses. The Scan checks a bounded set of variants, not every duplicate on your site.
Pick a preferred host and path format. Redirect equivalent versions to it and update internal links. Preserve query parameters that genuinely change content. Test both a homepage variant and an inner-page variant; make sure the destination is the matching page, not a generic landing page.
Conflicting indexing signals
Your sitemap, robots rules, canonical tags and indexing directives can disagree. A sitemap can list a URL that is blocked, excluded from search or canonicalized elsewhere. This gives discovery tools an inconsistent map.
Decide which URL should be public first. Then make the signals agree: list the preferred indexable page, remove accidental exclusions and leave intentionally private pages out. Recheck all affected signals together rather than fixing one warning in isolation. See our sitemap guide.
Missing pages with a success response
A missing page that returns 200 OK looks like a real page to a basic crawler. Search engines can still recognize some error screens, but the HTTP response should tell the truth.
Return 404 Not Found or 410 Gone for a nonexistent URL and keep a helpful error screen. For client-rendered apps, fix the server route as well as the screen. Test an invented path and a real path. The real page must still return a successful status. See MDN’s 404 reference.
HTTPS redirects
HTTP requests should reach the matching HTTPS page without loops. A bad redirect can stop a crawler or strip the path the visitor wanted.
Configure the preferred HTTPS host once, preserve the path and avoid repeated hops between your CDN and app. Test the whole chain for an inner page as well as the root. Confirm HTTPS works before adding long-lived HSTS rules. MDN’s redirect guide explains permanent and temporary redirects.
