Most AI crawlers read only the HTML your server sends; they don't run JavaScript. If your site builds its content in the browser, as many single-page apps do, those crawlers may see an empty page. Googlebot is the main exception because it renders JavaScript, which is why a site can rank in Google and still be invisible to ChatGPT, Claude and Perplexity. This guide explains how to test your site and the options for fixing it.
Why this matters more for AI than for Google
| Crawler | Renders JavaScript? |
|---|---|
| Googlebot | Yes, with a modern rendering engine |
| Bingbot | Partially |
| OAI-SearchBot, GPTBot, ChatGPT-User | Generally read raw HTML |
| Claude-SearchBot, ClaudeBot, Claude-User | Generally read raw HTML |
| PerplexityBot, Perplexity-User | Generally read raw HTML |
Rendering is expensive at web scale, so most AI crawlers skip it. Behavior can change, but the safe assumption is: if it isn't in the initial HTML, AI may not see it.
What goes missing
On a client-rendered site, the raw HTML often contains little more than:
<body>
<div id="root"></div>
<script src="/assets/app.js"></script>
</body>
Everything a person sees, from headings and product descriptions to prices, FAQs and article text, is added later by JavaScript. Common casualties:
- Entire pages on single-page apps (React, Vue, Angular without server rendering)
- Prices and plan details loaded from an API
- Reviews and ratings loaded by a widget
- Tabs and accordions that fetch content on click
- Help center articles rendered by an embedded widget
- Meta tags, canonical tags and structured data injected by scripts
How to test
1. View source, not the inspector
In your browser, use "View page source" (not the developer tools Elements panel, which shows the rendered page). Search for a sentence from the page. If it's missing, it's rendered client-side.
2. Fetch with curl
curl -s https://www.example.com/pricing | grep -i "per month"
Try it with an AI crawler user agent too:
curl -s -A "Mozilla/5.0 (compatible; OAI-SearchBot/1.0; +https://openai.com/searchbot)" https://www.example.com/pricing | grep -i "per month"
3. Disable JavaScript
Turn off JavaScript in your browser and load the page. What you see is roughly what a non-rendering crawler gets.
4. Ask an assistant
Ask ChatGPT, Claude or Perplexity to summarize a specific URL. If the summary is empty or generic, rendering may be the problem.
How to fix it
Option 1: Server-side rendering (SSR)
The server generates full HTML for each request, and JavaScript then makes it interactive. Frameworks such as Next.js, Nuxt, Remix and SvelteKit support SSR. Best for dynamic pages whose content changes per request.
Option 2: Static site generation (SSG)
HTML is generated at build time for each page. Excellent for marketing pages, blogs and documentation. Fast and simple to host.
Option 3: Prerendering
For existing single-page apps, prerender the important routes to static HTML at build time, or serve prerendered snapshots to crawlers. This is often the quickest fix when a full migration isn't practical. Make sure crawlers and users get the same content.
Option 4: Hybrid
Render the content-critical parts on the server, and keep highly interactive parts (dashboards, configurators) client-side. Most sites only need marketing, product, pricing, blog and docs pages to be server-rendered.
Don't forget the head
Even with server-rendered body content, check that these are in the initial HTML:
<title>and meta description- Canonical tag
- Open Graph tags
- Structured data (JSON-LD). See schema markup for AEO.
hreflangtags for international sites
If a script injects them after load, many crawlers won't see them.
Prioritize the pages that matter
You don't need to server-render everything. Start with:
- Pricing and plans
- Product and feature pages
- Comparison and use-case pages
- Blog posts and guides
- Documentation and help articles
- About page and other entity pages
These are the pages AI assistants retrieve for buying and how-to questions. See the technical AEO audit checklist.
Common pitfalls
- Lazy-loaded text that only appears on scroll
- Content behind client-side routing with no server response for deep URLs (they return the same empty shell)
- Hydration errors that break the page after load
- Different content for crawlers and users, which can be treated as cloaking
- Blocking JavaScript and CSS files in robots.txt, which hurts Googlebot's rendering
The bottom line
If your important content only exists after JavaScript runs, most AI crawlers can't read it. Test with view-source and curl, then fix the pages that matter with server-side rendering, static generation or prerendering, including the metadata and structured data in the head. It's often the single biggest technical AEO win on modern websites.
For the rest of the crawler picture, see our AI crawlers and robots.txt guide.
