www.sofi.com

Business impact If robots.txt blocks AI crawlers you are invisible to ChatGPT, Claude and Perplexity — they skip you and recommend a competitor instead.

What we measured We read /robots.txt and test it against 16 AI user-agents (GPTBot, ClaudeBot, PerplexityBot, …) for a Disallow that blocks them.

How to fix Allow major AI bots to public content; restrict only private paths (/admin, /api).

Learn how to implement →

User-agent: GPTBot
Allow: /
Disallow: /admin/

Spec: https://platform.openai.com/docs/gptbot

Business impact llms.txt is the robots.txt for AI: it tells agents what your site is, what matters, and where to find it. Without it AI guesses — and guessing means inaccurate recommendations and lost visibility.

What we measured We fetch /llms.txt and /.well-known/llms.txt and validate the spec (H1 title + a one-line blockquote summary). We also note /llms-full.txt (your full content as Markdown).

How to fix Create /llms.txt with a short summary + key pages; optionally /llms-full.txt with full content in Markdown.

Learn how to implement →

# Your Site
> One-line description for AI agents.

## Key pages
- /products — catalog
- /pricing — plans
- /docs — documentation

Spec: https://llmstxt.org

Business impact Public content behind authentication is unreachable to agents (they have no account). Gate only genuinely private material.

What we measured We follow redirects and detect 401/403 and login-form signals on content pages.

How to fix Separate public marketing/docs from authenticated areas; keep public pages reachable without login.

Spec: https://llmstxt.org

Business impact Schema.org JSON-LD tells agents what a page IS (product, article, business) with typed fields (price, rating, hours). Without it agents extract less reliably.

What we measured We parse <script type=application/ld+json>, validate it, and check for populated @type fields.

How to fix Add JSON-LD: Organization/LocalBusiness on the homepage, Product on product pages, Article on posts.

Learn how to implement →

<script type="application/ld+json">{"@context":"https://schema.org","@type":"Organization","name":"Your Co","url":"https://example.com"}</script>

Spec: https://schema.org/

Business impact A sitemap is your table of contents for AI crawlers. Without it agents follow homepage links and miss deep pages (products, docs, pricing) — shrinking what they can recommend.

What we measured We fetch /sitemap.xml (and /sitemap_index.xml), confirm valid XML with <loc> entries, and check <lastmod> freshness.

How to fix Generate an XML sitemap of all public pages with current lastmod dates and reference it in robots.txt.

Learn how to implement →

# robots.txt
Sitemap: https://example.com/sitemap.xml

Spec: https://www.sitemaps.org/

Business impact agents.json describes what your site can DO for agents (services, endpoints, capabilities) — an emerging discovery standard. Early adopters get native agent integration.

What we measured We check /agents.json and /.well-known/agents.json for a valid configuration.

How to fix Publish /agents.json describing your site's capabilities and actions.

Learn how to implement →

Spec: https://github.com/wild-card-ai/agents-json

Business impact WebMCP lets agents call actions on your site directly (book, buy, query) instead of scraping the DOM. Early adopters get native AI-agent interoperability.

What we measured We check /.well-known/webmcp and /webmcp.json for a valid actions array.

How to fix Add a WebMCP endpoint exposing your key actions to agents.

Learn how to implement →

Spec: https://webmcp.org

Business impact Programmatic agents prefer a typed API. An OpenAPI/Swagger spec lets them integrate without scraping.

What we measured We probe /openapi.json, /swagger.json, /api-docs and /.well-known/openapi.json.

How to fix Publish an OpenAPI spec at a well-known path.

Learn how to implement →

Spec: https://www.openapis.org/