Guide
ChatGPT Search and Website Publishers
What should a website owner do about OpenAI's crawlers?
ChatGPT Search and Website Publishers
What website owners should know about OpenAI's crawlers, the separate controls for search and training access, and how to decide.
- 01ServicesFive connected disciplines: SEO, AI optimisation, answer engine optimisation, generative engine optimisation and search experience optimisation.
- 02InsightsRead practical insights on SEO, AI optimisation, answer engines, generative search and website experience.
- 03PricingExplore individual SEO, AIO, AEO, GEO and SXO services, plus combined packages. Confirm scope and billing before purchase.
- 04ContactTalk to Make My Brand Found about SEO, AIO, AEO, GEO and SXO. Share your website and goals to start a focused conversation.
Summary
OpenAI documents different crawlers for different purposes, and the controls for search access and model-training access are separate. The right setting depends on your business goals, and you should read the current documentation before changing anything.
Two separate decisions
- Whether you want your pages available for search-style features
- Whether you permit your content to be used for model training
- These can be set independently through robots rules, according to OpenAI's documentation
How to decide
- If being found through AI search matters to revenue, allowing the search crawler is usually sensible
- If you object to training use, disallow the training crawler
- Document the decision and who made it
Check, then verify
Crawler names and behaviour change, so confirm the names in the provider's documentation and test that your robots file does what you intend. A robots file is a request that compliant crawlers follow, not access control.
Example
Illustrative scenario: a publisher blocks all unknown bots with a broad rule and later wonders why it is absent from AI search results. A deliberate, named policy avoids blocking by accident.
Sources
Examples are illustrations, not client results. No ranking, citation or revenue outcome is guaranteed.
Next step
Ready to improve how customers discover and experience your brand?
Book a strategy call and tell us what growth needs to look like for your business.