Crawler access
Are the AI search bots, the on-demand fetchers and the training bots allowed or blocked, and was that your choice?
Use case·Publishers//LLMReady
Decide who may read and train on your content, and prove it.
Publishers want search and answer engines to find them but may not want models trained on their work. A robots.txt file is a request, and some fetchers ignore it.
GPTBot 200 (training)ClaudeBot 200 (training)Unknown-bot 200No log of decisions
AI readiness
34
Bot policy
Search may read. Training may not.
Decisions today
- GPTBot 403, policy: block
- OAI-SearchBot 200, policy: allow
- Unknown-bot 403, identity failed
llms.txtmarkdownjson-ld
AI readiness
88
What LLMReady does
What changes for this job.
- Sets a different policy for search, user-fetch and training bots
- Checks bot identity against vendor IP ranges so a fake user agent is refused
- Keeps a dated log of every bot decision
Checks that matter most
Start with these three.
robots.txt
sitemap.xml
Sitemap health
Is every important page listed, current and reachable, with no dead or redirected URLs.
403 / 429
Firewall and bot rules
Security tools often block AI crawlers by accident. LLMReady requests your pages as each bot and reports what comes back.
FAQ
Questions on this use case.
01Can LLMReady stop a bot that ignores robots.txt?
It can refuse requests at the edge by policy and identity. Robots.txt alone cannot enforce anything.
02Does blocking training bots remove me from AI search?
No. Search and training are separate bots with separate rules.
Beta·Early access
Join the beta and see what AI does with your site.
Leave your email and you will hear when your spot is ready.