You can allow every AI crawler in the world, and they will still miss your best content if that content only appears after JavaScript runs. This is the quiet failure mode behind a lot of low scores: the page looks perfect in a browser, and nearly empty to the crawler.
What it is
Crawler reachability is whether your important content, who you are, what you sell, your proof, is present in the raw HTML your server sends back, before any JavaScript executes.
A browser downloads your HTML, then runs your JavaScript, which often fetches and renders the real content. A person sees the finished page. Many AI crawlers do not. They read the first HTML response and stop.
Why AI engines care
A 2024 study by Vercel and MERJ that looked at more than 500 million crawler requests found that the major AI crawlers do not execute JavaScript at all. That includes OpenAI’s crawlers (GPTBot, OAI-SearchBot, ChatGPT-User), Anthropic’s ClaudeBot, and PerplexityBot. They sometimes download JavaScript files, but they do not run them.
The consequence is direct. If your homepage ships an almost-empty HTML shell and builds the content client-side, which is the default for many React, Vue, and Angular single-page apps, then the crawler sees the empty shell. Your strongest page can be effectively invisible to the engines you most want to reach.
The second half of reachability is speed. A crawler that times out or gets a slow response may fetch less, or skip the page. Reliable, reasonably fast responses keep more of your content within reach.
How to set it up
The fix is to make sure the content exists in the initial HTML, not just in the rendered browser view.
- Render your key content on the server. Use server-side rendering (SSR), static site generation (SSG), or incremental static regeneration so the main text, headings, and links are in the HTML response. Most modern frameworks (Next.js, Astro, Nuxt, SvelteKit) support this directly.
- Put identity, offer, and proof in the markup. The pages that describe who you are and what you do should not depend on JavaScript to show their main content.
- Keep responses quick and reliable. Slow time-to-first-byte and frequent timeouts cost you crawl coverage. See the page speed guide for the levers.
How to check it
Look at the page the way a crawler does, without JavaScript:
# Fetch the raw HTML your server returns, before any JS runs.
curl -sL https://yourdomain.com/ | lessSearch that output for your headline, your main paragraph, and your key links. If they are missing, the content is being added by JavaScript, and crawlers that do not run JavaScript will miss it. You can also disable JavaScript in your browser’s dev tools and reload: if the page goes blank or loses its main content, so does the crawler’s view.
One nuance: Google’s own systems and Apple’s crawler can render JavaScript, so a JS-heavy page is not invisible everywhere. But the crawlers behind ChatGPT, Claude, and Perplexity cannot. If you want to be read across all of them, put the content in the HTML.