Crawler access
Are the AI search bots, the on-demand fetchers and the training bots allowed or blocked, and was that your choice?
Product·Overview//Everything AI needs
Everything AI needs to read your site.
LLMReady watches the crawlers, fixes what they cannot read, serves them a clean version of every page and shows what they send back.
01·analytics
AI traffic analytics
Connect your CDN or send server logs. See every AI crawler, the pages it fetched, what it got back and which AI answers sent visitors. Most of this never shows up in Google Analytics, because crawlers do not run scripts.
- Every request classified by bot and purpose: search, user fetch or training
- Served, refused and failed requests per bot and per page
- Visits from ChatGPT, Perplexity, Gemini and Claude answers
0AI requests
0Refused or failed
0Visits from AI
| Bot | Requests | Served |
|---|---|---|
| GPTBot | 18,402 | 92% |
| ClaudeBot | 9,115 | 90% |
| PerplexityBot | 6,220 | 38% |
| OAI-SearchBot | 5,480 | 98% |
PerplexityBot was refused 62 percent of the time by a firewall rule. Fixing it is the largest single gain on this site.
02·agent pages
Agent pages
When a crawler asks for a page, it gets a clean, structured, current version served from the edge. Human visitors see your normal site. The machine version rebuilds on every deploy, so it never goes stale.
- Markdown and structured data for every key page
- llms.txt written and refreshed from your sitemap and top pages
- Works with any platform, with no changes to your pages
curl -A GPTBot https://acme-supply.example/pricingHTTP/2 200 content-type: text/markdown# Pricing> Acme Supply sells reusable packaging to small food businesses. ## Plans- Starter, $19 a month, one location- Growth, $59 a month, up to five locations llms.txt found Markdown found JSON-LD found
03·policy and proof
Bot policy and proof
Allow, block or limit each bot, with identity checks so a fake user agent cannot walk through. Keep a log you can show a client, a manager or a lawyer.
- Per-bot rules for search, user and training bots
- Verification against each vendor's published IP ranges
- A dated log of every decision
policy
OAI-SearchBotSearch. Finds pages for ChatGPT search.
ChatGPT-UserUser fetch. Opens a page when a person asks.
GPTBotTraining. Collects content that may train models.
Unknown botFails identity check against vendor IP ranges.
Log: 4 decisions today, 0 overrides.
04·readiness scan
Readiness scan
The scan fetches your pages as each major AI crawler would and records what it receives, what it is refused and what is missing. Every finding comes with the exact fix.
- Eight checks and a score out of 100
- Fixes ordered by effort and effect
- Re-scan after every change
0/ 100
Three problems to fix first. Two take under ten minutes.
Fix
llms.txt
No file found at /llms.txt.
LLMReady writes it from your sitemap and top pages.
Review
Crawler access
GPTBot is allowed. ClaudeBot and PerplexityBot are blocked by a firewall rule.
Pass
Sitemap health
412 URLs listed, all return 200.
05·alerts
Alerts
A firewall rule that starts refusing PerplexityBot is the kind of change nobody notices for weeks. LLMReady tells you the day it happens.
- Alerts when a crawler starts getting 403 or 429
- Alerts when llms.txt or the sitemap breaks
- Email and webhook delivery
Now
PerplexityBot started getting 403
Requests to /products and /pricing were refused from 09:12 UTC after a firewall rule changed.
Today
llms.txt links to 2 pages that redirect
/shipping and /returns now return 301.
Fixed
Sitemap lists 412 URLs again
The sitemap returned 500 for 40 minutes yesterday.
06·attribution
Referral and sales attribution
See which AI answers send visitors, which pages they land on and which of those visits turn into orders or sign-ups.
- Referral sources from the four main assistants
- Landing pages and conversions per source
- Weekly report for clients or managers
visits from AI answers, 30 days
Checks·The readiness scan//Eight checks
Eight checks, each with the fix.
Every finding names the file, the status code or the bot, then gives the exact change.
llms.txt file
A short guide to your best pages, written for language models. LLMReady checks that it exists, loads and links to live pages.
Sitemap health
Is every important page listed, current and reachable, with no dead or redirected URLs.
Structured data
Organization, Product, Article and FAQ markup that tells machines what a page is about without guessing.
Content without scripts
Many crawlers do not run JavaScript. LLMReady compares what loads with and without it and flags text that goes missing.
Headings and structure
Clear headings, one topic per page and answers placed near the question, so a model can lift them cleanly.
Markdown versions
A plain markdown copy of each key page is the easiest thing for a model to read. LLMReady tests whether yours exist.
Firewall and bot rules
Security tools often block AI crawlers by accident. LLMReady requests your pages as each bot and reports what comes back.
Developers·CLI and API
Stop a release from quietly breaking AI visibility.
Run the readiness check in your build. If the score drops below your threshold, the release fails. The same data is available over the API.
- A CLI for local runs and CI, with a fail-under threshold
- An API for scans, bot decisions and crawl data
- Log export and single sign-on on the Scale plan
$ llmready check https://yoursite.com --fail-under 80 PASS sitemap 412 urlsFAIL llms.txt not foundFAIL markdown pages 0 of 12WARN crawler access 2 bots refused (403) Score 64, needs 80
$ curl -X POST https://api.llmready.site/v1/scan \ -H "Authorization: Bearer $LLMREADY_KEY" \ -d '{"url": "https://yoursite.com"}' { "id": "scn_8f3a", "status": "queued" }
Beta·Early access
Join the beta and see what AI does with your site.
Leave your email and you will hear when your spot is ready.