Technical AEO

    AI Crawlers and robots.txt: GPTBot, ClaudeBot, PerplexityBot and More

    What GPTBot, OAI-SearchBot, ClaudeBot, PerplexityBot, Google-Extended and other AI crawlers do, and a robots.txt that keeps you visible in AI search.

    Saarthak Srivastav

    Saarthak Srivastav

    · 6 min read

    Key takeaways

    • Most AI companies now run separate bots for training, for search indexing and for fetching pages on a user's request. You can treat each differently.
    • To appear in AI search answers, allow the search crawlers: OAI-SearchBot, Claude-SearchBot, PerplexityBot, Googlebot and Bingbot.
    • Blocking training crawlers like GPTBot, ClaudeBot or CCBot doesn't remove you from AI search results. It's a separate decision.
    • Google-Extended and Applebot-Extended are control tokens, not crawlers. They govern AI training use, not search visibility.
    • robots.txt isn't the only gate. CDN and firewall bot rules frequently block AI crawlers without anyone noticing.

    If you want your brand to appear in AI answers, the crawlers behind those answers have to be able to read your site. Most AI companies now operate three kinds of bots: one that gathers training data, one that builds a search index, and one that fetches pages when a user asks. You can allow or block each separately in robots.txt. This guide lists the main ones, explains what blocking each actually does, and gives a recommended configuration.

    The three kinds of AI bots

    Type What it does Effect of blocking
    Training crawlers Collect public content that may be used to train future models Your future content is excluded from that company's training data. No effect on today's search answers.
    Search crawlers Build the index an assistant searches before answering You stop appearing (or appear less) in that assistant's search-based answers and citations.
    User-triggered fetchers Load a specific page because a user asked about it or pasted the link The assistant can't open your page for users, even when they ask for it directly.

    The important point: training and search are separate decisions. You can opt out of training and still be fully visible in AI search, or the reverse.

    The major AI crawlers

    OpenAI (ChatGPT)

    User agent Type Notes
    OAI-SearchBot Search Powers ChatGPT search results and citations. Allow it if you want to appear in ChatGPT search.
    ChatGPT-User User-triggered Fetches pages during a user's conversation. OpenAI notes that because these requests are user-initiated, robots.txt rules may not apply to them the way they do to automated crawlers.
    GPTBot Training Collects content that may be used to train OpenAI's models.

    ChatGPT search also uses third-party search providers, so Bing visibility matters too. See how to get your brand mentioned in ChatGPT.

    Anthropic (Claude)

    User agent Type Notes
    Claude-SearchBot Search Indexes content to improve Claude's search results. Blocking it may reduce your visibility in them.
    Claude-User User-triggered Fetches pages when a Claude user's question needs them.
    ClaudeBot Training Collects content that may be used to train Anthropic's models.

    Anthropic says its bots honor robots.txt, supports the non-standard Crawl-delay directive, and publishes its crawler IP addresses for verification.

    Perplexity

    User agent Type Notes
    PerplexityBot Search Builds Perplexity's index for search answers and citations. Perplexity says it isn't used to train foundation models.
    Perplexity-User User-triggered Fetches pages in response to a user's question. Perplexity documents that this fetcher generally ignores robots.txt because the user requested the page.

    Google (Search, AI Overviews, AI Mode, Gemini)

    Token Type Notes
    Googlebot Search Crawls for Google Search, including AI Overviews and AI Mode. Blocking it removes you from Google Search entirely.
    Google-Extended Control token Not a separate crawler. Disallowing it tells Google not to use your content for training Gemini models and for some grounding uses outside Search. It does not affect AI Overviews or rankings.

    More detail in our Google AI Overviews guide.

    Microsoft (Bing, Copilot)

    User agent Type Notes
    Bingbot Search Crawls for Bing, which also grounds Microsoft Copilot answers and is used by other AI products.

    Apple

    Token Type Notes
    Applebot Search Powers search features in Siri, Spotlight and Safari.
    Applebot-Extended Control token Disallowing it opts your content out of training Apple's foundation models. It doesn't stop Applebot from crawling for search.

    Others worth knowing

    User agent Operator Notes
    CCBot Common Crawl Builds the open Common Crawl dataset, used to train many AI models.
    Meta-ExternalAgent Meta Crawls for AI training and related uses.
    Amazonbot Amazon Crawls for Amazon services, including Alexa answers.
    Bytespider ByteDance Crawler associated with ByteDance's AI products.
    DuckAssistBot DuckDuckGo Fetches content for DuckDuckGo's AI-assisted answers.

    User agent names and policies change. Check each company's official crawler documentation before making changes, and re-check a few times a year.

    For most brands that want maximum visibility in AI search, while keeping private areas private:

    # Search engines and AI search crawlers
    User-agent: Googlebot
    User-agent: Bingbot
    User-agent: OAI-SearchBot
    User-agent: Claude-SearchBot
    User-agent: PerplexityBot
    User-agent: Applebot
    Allow: /
    Disallow: /admin/
    Disallow: /account/
    
    # User-triggered fetchers
    User-agent: ChatGPT-User
    User-agent: Claude-User
    User-agent: Perplexity-User
    Allow: /
    
    # Training crawlers: allow or block based on your policy
    User-agent: GPTBot
    User-agent: ClaudeBot
    User-agent: CCBot
    Allow: /
    
    # Everyone else
    User-agent: *
    Allow: /
    Disallow: /admin/
    Disallow: /account/
    
    Sitemap: https://www.example.com/sitemap.xml
    

    Two notes on syntax:

    • Grouping user agents (several User-agent lines before one set of rules) is supported by major crawlers and keeps the file readable.
    • A crawler follows only the most specific group that matches it. If you add a User-agent: GPTBot group, GPTBot ignores your User-agent: * rules, so repeat any Disallow lines you need.

    If you want to opt out of training

    Replace the training section with:

    User-agent: GPTBot
    User-agent: ClaudeBot
    User-agent: CCBot
    User-agent: Google-Extended
    User-agent: Applebot-Extended
    User-agent: Meta-ExternalAgent
    Disallow: /
    

    This keeps you visible in AI search while signaling that your content shouldn't be used to train future models.

    Should you block training crawlers?

    There's a real trade-off:

    • Allowing training means future models are more likely to know your brand, products and positioning without needing to search. For most companies selling products, that's an advantage.
    • Blocking training protects content you license or sell, such as journalism, research or courseware, from being absorbed into models without compensation.

    Brands usually allow; publishers often block. Either way, keep the search crawlers allowed.

    robots.txt isn't the only gate

    Many sites block AI crawlers without realizing it. Check:

    1. CDN and firewall bot rules. Services such as Cloudflare offer settings to block AI crawlers, and Cloudflare began blocking known AI crawlers by default on new domains in 2025. A firewall block overrides whatever your robots.txt says.
    2. Rate limiting and bot challenges. CAPTCHAs and JavaScript challenges stop crawlers that can't solve them.
    3. Login walls and cookie banners that hide content until interaction.
    4. Client-side rendering. Most AI crawlers don't run JavaScript. If your content only appears after JavaScript runs, they see an empty page. Serve important content in the initial HTML through server-side rendering, static generation or prerendering.
    5. noindex and nosnippet. These affect search-based AI features, including Google's.

    How to verify crawler access

    1. Read your live robots.txt at yourdomain.com/robots.txt. Make sure it's the file you think it is; some platforms generate it automatically.

    2. Test with curl using each user agent and check you get a 200 and real content:

      curl -A "OAI-SearchBot" -I https://www.example.com/pricing
      curl -A "PerplexityBot" https://www.example.com/pricing | grep -i "price"
      

      A 403, a challenge page or HTML without your content means something is blocking or hiding it.

    3. Check your logs for visits from these user agents. Regular visits from search crawlers are a good sign. Verify against the published IP ranges, since user agents can be spoofed.

    4. Ask the assistant. Ask ChatGPT, Claude or Perplexity to summarize a specific URL on your site. If it can't access the page, investigate.

    The bottom line

    AI visibility starts with access. Allow the search crawlers and user-triggered fetchers of the assistants you care about, make a deliberate choice on training crawlers, and then check that your CDN, firewall and rendering setup aren't quietly undoing your robots.txt. It's one of the few AEO tasks that can be done in an afternoon and affects every answer that follows.

    Once crawlers can read your pages, make sure the pages are worth quoting: see how to write content AI assistants cite.

    For a ChatGPT-specific walkthrough, see how to submit your website to ChatGPT and the ChatGPT ranking factors.

    FAQ

    Frequently asked questions

    Should I block GPTBot?

    It depends on whether you want your content used to train future OpenAI models. Blocking GPTBot doesn't affect ChatGPT search, which uses OAI-SearchBot. Many brands allow GPTBot because they want future models to know them; publishers with licensing concerns often block it.

    Do AI crawlers respect robots.txt?

    The major AI companies say their automated crawlers follow robots.txt. Fetchers that act on a specific user request, such as ChatGPT-User or Perplexity-User, may be treated differently, as those companies document. Some lesser-known scrapers ignore robots.txt entirely, which is why server-level controls exist.

    How do I know if AI crawlers are visiting my site?

    Check your server or CDN logs for their user agent strings, such as GPTBot, OAI-SearchBot, ClaudeBot, Claude-SearchBot and PerplexityBot. Because user agents can be faked, verify important traffic against the IP ranges each company publishes.

    Does blocking Google-Extended remove me from AI Overviews?

    No. AI Overviews and AI Mode are part of Google Search and use Googlebot. Google-Extended controls whether your content can be used for Gemini model training and some grounding uses outside Search.

    Can AI crawlers read JavaScript-rendered content?

    Most AI crawlers fetch the raw HTML and don't execute JavaScript, so content rendered only on the client may be invisible to them. Googlebot is the main exception, as it renders pages. Server-side rendering or prerendering is the safest option.

    • robots.txt
    • AI crawlers
    • GPTBot
    • ClaudeBot
    • PerplexityBot
    • technical SEO
    Saarthak Srivastav

    Written by

    Saarthak Srivastav

    AEO & GEO, AEO-Rank

    Saarthak Srivastav writes and reviews articles on the AEO-Rank blog, drawing on what the AEO-Rank team sees tracking AI answers across ChatGPT, Gemini, Claude, Perplexity and Grok.