Your product might be invisible to the agents developers now ask
I pointed the cav CLI at a widely-used developer-tools SaaS — the kind whose marketing
site and API docs are a single-page app that renders entirely in the browser. Here’s what an
AI assistant gets when it visits.
A no-JS crawler — which is what GPTBot, ClaudeBot, and PerplexityBot effectively are — gets
an empty shell: a few hundred bytes of <div id="root"> and a script tag, no prose, no
endpoints, no code samples. Content Recovery drops toward zero. The verdict isn’t a
nuanced score — the page is effectively empty to a machine.
So when a developer asks Claude or ChatGPT “what’s a good library for this” or “show me how to authenticate against this API,” the product cannot be the source. The assistant reads, cites, and writes code from competitors whose docs it can actually parse — or worse, it hallucinates your API from a stale training snapshot. And you won’t see it in analytics: there’s no request, no bounce — the agent simply never gets the content.
This is worse than a bad score
A page that scores poorly is readable but degraded — you can fix it. A page that renders only client-side is unreadable to an agent, period; the quality of the docs behind it is irrelevant because the agent never executes your JavaScript. The SPA that felt fast and modern to a human is a blank page to the machine — and AI assistants are fast becoming how developers discover tools and learn APIs (Copilot, Cursor, ChatGPT, Claude, Perplexity).
The wider pattern
Across a few hundred real pages, content recovery turned out to be bimodal: pages cluster near fully-readable or near-invisible, with little in between. A page is either legible to an agent or it isn’t. For software, the invisible end is usually a rendering choice — client-side rendering that ships a shell and hydrates later — not malice. But the agent doesn’t wait for hydration, and it doesn’t run your bundle.
What to do
- Server-render the content that matters — landing copy, docs prose, code samples, endpoint tables — so it’s in the HTML, not assembled in the browser.
- Add structured data (
SoftwareApplication,TechArticle, and API reference markup where it fits) so the agent gets typed facts, not just text. - Allowlist legitimate answer-engine crawlers (GPTBot, ClaudeBot, PerplexityBot, OAI-SearchBot, Google-Extended) — serve them content, not a challenge.
- Re-measure. The goal is to move from empty to a real, scorable page.
You can check your own site in two minutes — and if it comes back near-empty, that’s not a metric to argue about, it’s your product missing from the place developers are already asking.