Blocking AI training crawlers is a business decision about your content, not an SEO setting. Training crawlers such as GPTBot, ClaudeBot and CCBot collect content that may train future models. Search crawlers such as OAI-SearchBot, Claude-SearchBot and PerplexityBot power today's AI search answers. You can block one and allow the other. For most brands that sell products or services, allowing training helps future models know who they are. For businesses whose content is the product, blocking often makes sense.
First: understand what you're blocking
| Crawler or token | Operator | Purpose | Affects AI search answers? |
|---|---|---|---|
| GPTBot | OpenAI | Training | No (search uses OAI-SearchBot) |
| ClaudeBot | Anthropic | Training | No (search uses Claude-SearchBot) |
| CCBot | Common Crawl | Open dataset used by many model trainers | Indirectly, via training |
| Google-Extended | Gemini training and some grounding outside Search | Not Google Search or AI Overviews | |
| Applebot-Extended | Apple | Apple foundation model training | Not Siri or Spotlight search |
| Meta-ExternalAgent | Meta | AI training and related uses | Indirectly |
Details and recommended configurations are in our AI crawlers and robots.txt guide.
The case for allowing training
- Future models know you. When an assistant answers without searching, it relies on what it learned. If your site was excluded, the model's picture of you comes only from third parties, which may be outdated or less favorable.
- Accurate positioning. Your own pages are the best source for what you do, who you serve and how you differ.
- Category association. Consistent coverage across training data helps models connect your brand with your category and use cases.
- Low cost for most businesses. If your marketing pages are designed to be read widely, letting models read them carries little downside.
See how AI assistants generate answers for why model knowledge matters.
The case for blocking training
- Your content is the product. News, research, courses, premium analysis and proprietary datasets lose value if models reproduce them freely.
- Licensing leverage. Some publishers block training while negotiating licensing deals with AI companies.
- Server load. Heavy crawling can strain small servers, though rate limits or crawl-delay may solve that without a full block.
- Policy or legal reasons. Regulated industries or contractual obligations may require restrictions.
What blocking doesn't do
- It doesn't remove content already collected. Past crawls may already be in datasets.
- It doesn't stop third parties from writing about you, which models will still learn from.
- It doesn't affect AI search if you keep search crawlers allowed.
- It doesn't stop non-compliant scrapers. Some bots ignore robots.txt; that needs server-level controls.
A decision framework
Answer these four questions:
Is your content your product? Yes → lean toward blocking training on that content. No → lean toward allowing.
Do you want future models to describe you accurately without searching? Yes → allow at least your marketing, product and docs pages.
Are you negotiating, or likely to negotiate, content licensing? Yes → blocking may preserve leverage.
Is crawl load a real problem? Yes → try rate limiting or crawl-delay before blocking.
Common outcomes
| Business type | Typical choice |
|---|---|
| SaaS, ecommerce, local services, B2B | Allow training on public marketing and docs pages |
| News and media publishers | Block training, allow search crawlers, consider licensing |
| Education and course creators | Allow marketing pages; block course content |
| Research and data businesses | Allow summaries; block full reports and datasets |
| Community platforms | Depends on user agreements and policy |
Partial blocking
You don't have to choose all or nothing. Block training on specific paths:
User-agent: GPTBot
User-agent: ClaudeBot
User-agent: CCBot
Allow: /
Disallow: /courses/
Disallow: /reports/full/
Disallow: /members/
Remember that a crawler follows only the most specific group that matches it, so include every rule you need in that group.
Keep AI search open
Whatever you decide about training, if you want to appear in AI answers, allow:
- OAI-SearchBot and ChatGPT-User
- Claude-SearchBot and Claude-User
- PerplexityBot and Perplexity-User
- Googlebot and Bingbot
And check that your CDN or firewall isn't blocking them. See AI crawler log analysis.
The bottom line
Separate the two decisions. Keep AI search crawlers open if you want visibility. For training crawlers, allow them if your public content is marketing for your business, and block them, fully or by path, if your content is what you sell. Revisit the decision as licensing markets and AI products evolve.
News and media sites face a sharper version of this decision; see AEO for news publishers.
