If you want your brand to appear in AI answers, the crawlers behind those answers have to be able to read your site. Most AI companies now operate three kinds of bots: one that gathers training data, one that builds a search index, and one that fetches pages when a user asks. You can allow or block each separately in robots.txt. This guide lists the main ones, explains what blocking each actually does, and gives a recommended configuration.
The three kinds of AI bots
| Type | What it does | Effect of blocking |
|---|---|---|
| Training crawlers | Collect public content that may be used to train future models | Your future content is excluded from that company's training data. No effect on today's search answers. |
| Search crawlers | Build the index an assistant searches before answering | You stop appearing (or appear less) in that assistant's search-based answers and citations. |
| User-triggered fetchers | Load a specific page because a user asked about it or pasted the link | The assistant can't open your page for users, even when they ask for it directly. |
The important point: training and search are separate decisions. You can opt out of training and still be fully visible in AI search, or the reverse.
The major AI crawlers
OpenAI (ChatGPT)
| User agent | Type | Notes |
|---|---|---|
OAI-SearchBot |
Search | Powers ChatGPT search results and citations. Allow it if you want to appear in ChatGPT search. |
ChatGPT-User |
User-triggered | Fetches pages during a user's conversation. OpenAI notes that because these requests are user-initiated, robots.txt rules may not apply to them the way they do to automated crawlers. |
GPTBot |
Training | Collects content that may be used to train OpenAI's models. |
ChatGPT search also uses third-party search providers, so Bing visibility matters too. See how to get your brand mentioned in ChatGPT.
Anthropic (Claude)
| User agent | Type | Notes |
|---|---|---|
Claude-SearchBot |
Search | Indexes content to improve Claude's search results. Blocking it may reduce your visibility in them. |
Claude-User |
User-triggered | Fetches pages when a Claude user's question needs them. |
ClaudeBot |
Training | Collects content that may be used to train Anthropic's models. |
Anthropic says its bots honor robots.txt, supports the non-standard Crawl-delay directive, and publishes its crawler IP addresses for verification.
Perplexity
| User agent | Type | Notes |
|---|---|---|
PerplexityBot |
Search | Builds Perplexity's index for search answers and citations. Perplexity says it isn't used to train foundation models. |
Perplexity-User |
User-triggered | Fetches pages in response to a user's question. Perplexity documents that this fetcher generally ignores robots.txt because the user requested the page. |
Google (Search, AI Overviews, AI Mode, Gemini)
| Token | Type | Notes |
|---|---|---|
Googlebot |
Search | Crawls for Google Search, including AI Overviews and AI Mode. Blocking it removes you from Google Search entirely. |
Google-Extended |
Control token | Not a separate crawler. Disallowing it tells Google not to use your content for training Gemini models and for some grounding uses outside Search. It does not affect AI Overviews or rankings. |
More detail in our Google AI Overviews guide.
Microsoft (Bing, Copilot)
| User agent | Type | Notes |
|---|---|---|
Bingbot |
Search | Crawls for Bing, which also grounds Microsoft Copilot answers and is used by other AI products. |
Apple
| Token | Type | Notes |
|---|---|---|
Applebot |
Search | Powers search features in Siri, Spotlight and Safari. |
Applebot-Extended |
Control token | Disallowing it opts your content out of training Apple's foundation models. It doesn't stop Applebot from crawling for search. |
Others worth knowing
| User agent | Operator | Notes |
|---|---|---|
CCBot |
Common Crawl | Builds the open Common Crawl dataset, used to train many AI models. |
Meta-ExternalAgent |
Meta | Crawls for AI training and related uses. |
Amazonbot |
Amazon | Crawls for Amazon services, including Alexa answers. |
Bytespider |
ByteDance | Crawler associated with ByteDance's AI products. |
DuckAssistBot |
DuckDuckGo | Fetches content for DuckDuckGo's AI-assisted answers. |
User agent names and policies change. Check each company's official crawler documentation before making changes, and re-check a few times a year.
A recommended robots.txt for AI visibility
For most brands that want maximum visibility in AI search, while keeping private areas private:
# Search engines and AI search crawlers
User-agent: Googlebot
User-agent: Bingbot
User-agent: OAI-SearchBot
User-agent: Claude-SearchBot
User-agent: PerplexityBot
User-agent: Applebot
Allow: /
Disallow: /admin/
Disallow: /account/
# User-triggered fetchers
User-agent: ChatGPT-User
User-agent: Claude-User
User-agent: Perplexity-User
Allow: /
# Training crawlers: allow or block based on your policy
User-agent: GPTBot
User-agent: ClaudeBot
User-agent: CCBot
Allow: /
# Everyone else
User-agent: *
Allow: /
Disallow: /admin/
Disallow: /account/
Sitemap: https://www.example.com/sitemap.xml
Two notes on syntax:
- Grouping user agents (several
User-agentlines before one set of rules) is supported by major crawlers and keeps the file readable. - A crawler follows only the most specific group that matches it. If you add a
User-agent: GPTBotgroup, GPTBot ignores yourUser-agent: *rules, so repeat anyDisallowlines you need.
If you want to opt out of training
Replace the training section with:
User-agent: GPTBot
User-agent: ClaudeBot
User-agent: CCBot
User-agent: Google-Extended
User-agent: Applebot-Extended
User-agent: Meta-ExternalAgent
Disallow: /
This keeps you visible in AI search while signaling that your content shouldn't be used to train future models.
Should you block training crawlers?
There's a real trade-off:
- Allowing training means future models are more likely to know your brand, products and positioning without needing to search. For most companies selling products, that's an advantage.
- Blocking training protects content you license or sell, such as journalism, research or courseware, from being absorbed into models without compensation.
Brands usually allow; publishers often block. Either way, keep the search crawlers allowed.
robots.txt isn't the only gate
Many sites block AI crawlers without realizing it. Check:
- CDN and firewall bot rules. Services such as Cloudflare offer settings to block AI crawlers, and Cloudflare began blocking known AI crawlers by default on new domains in 2025. A firewall block overrides whatever your robots.txt says.
- Rate limiting and bot challenges. CAPTCHAs and JavaScript challenges stop crawlers that can't solve them.
- Login walls and cookie banners that hide content until interaction.
- Client-side rendering. Most AI crawlers don't run JavaScript. If your content only appears after JavaScript runs, they see an empty page. Serve important content in the initial HTML through server-side rendering, static generation or prerendering.
noindexandnosnippet. These affect search-based AI features, including Google's.
How to verify crawler access
Read your live robots.txt at
yourdomain.com/robots.txt. Make sure it's the file you think it is; some platforms generate it automatically.Test with curl using each user agent and check you get a
200and real content:curl -A "OAI-SearchBot" -I https://www.example.com/pricing curl -A "PerplexityBot" https://www.example.com/pricing | grep -i "price"A
403, a challenge page or HTML without your content means something is blocking or hiding it.Check your logs for visits from these user agents. Regular visits from search crawlers are a good sign. Verify against the published IP ranges, since user agents can be spoofed.
Ask the assistant. Ask ChatGPT, Claude or Perplexity to summarize a specific URL on your site. If it can't access the page, investigate.
The bottom line
AI visibility starts with access. Allow the search crawlers and user-triggered fetchers of the assistants you care about, make a deliberate choice on training crawlers, and then check that your CDN, firewall and rendering setup aren't quietly undoing your robots.txt. It's one of the few AEO tasks that can be done in an afternoon and affects every answer that follows.
Once crawlers can read your pages, make sure the pages are worth quoting: see how to write content AI assistants cite.
For a ChatGPT-specific walkthrough, see how to submit your website to ChatGPT and the ChatGPT ranking factors.
