Implementation · OWNER’S GUIDE
Can you allow AI search without allowing training?
Some providers separate search and training controls; the separation is not identical everywhere. Decide which uses you want to permit, check the provider’s current instructions and document the tradeoff before editing robots.txt.
Use a provider-specific decision table
| Provider | Search or retrieval | Training and tradeoff |
|---|---|---|
| OpenAI | OAI-SearchBot handles search; ChatGPT-User handles certain user actions. | GPTBot is separately controlled. User-initiated requests may not follow robots rules in the same way. |
| Anthropic | Claude-SearchBot supports search; Claude-User performs user-directed retrieval. | ClaudeBot covers potential training collection. |
| Googlebot, page-level indexing/preview rules and Search Console’s generative setting govern Search participation. | Google-Extended covers specified training and Gemini grounding; blocking it can affect both. |
These are summaries of OpenAI: Crawler documentation, Anthropic: Crawler controls and Google: Common crawler controls. Recheck them at the time of an owner-approved policy change. Google-Extended is a control token, not a separate HTTP user-agent string. Search Console’s generative participation setting is a separate owner choice. Google: Search generative AI control.
Write the decision before writing the rule
Complete this sentence for each provider: “We want our public pages available for ___; we do not want ___; we accept the documented effect on ___.” Blocking your site’s crawler access does not remove brand mentions on other websites, and it can prevent direct checking of your corrected facts. If two goals conflict, give the owner a choice. Do not silently trade away Gemini grounding while describing the change as a training-only block.
Our owner record includes domain, affected paths, current rule, proposed rule, reason, source checked, approver and rollback note. Keep marketing pages distinct from private workflows. Robots instructions do not protect confidential data; access control is a separate responsibility.
There is no universal robots snippet in this guide. Copying a new specific bot group can accidentally replace wildcard restrictions for that crawler. Review the effective policy, including private paths, rather than merely checking that the desired bot name appears.
Verify the policy and the production response
- Save the old rules and the intended behavior for public and private paths.
- Check syntax and effective matching locally.
- Verify the actual robots file served by the production hostname.
- Inspect CDN and hosting restrictions that may still block retrieval.
- Allow for crawler caching and propagation, then use authenticated platform evidence or verified crawler logs where available. Google-Extended does not appear as its own HTTP user agent.
A request that imitates a bot’s name is only a synthetic check; anyone can set that name. Likewise, a training opt-out is not a promise about removal of material already collected. Your deliverable should state the current rule and its supported purpose, followed by the checks performed and the evidence still unavailable.
Sources and review
Primary references checked November 11, 2026. Platform rules can change; check the linked instructions before acting.
- OpenAI: Crawler documentation
- Anthropic: Crawler controls
- Google: Common crawler controls
- Google: Search generative AI control
How we research and maintain this content · Send a correction