AI Platforms

    How Claude Finds and Cites Web Sources

    How Anthropic's Claude decides to search, which crawlers it uses, how it cites pages, and what to do so Claude can find and accurately describe your brand.

    Saarthak Srivastav

    Saarthak Srivastav

    · 4 min read

    Key takeaways

    • Claude searches the web when a question needs current or specific information, and cites the pages it uses inline.
    • Anthropic runs three bots: Claude-SearchBot for search indexing, Claude-User for user-requested fetches and ClaudeBot for training data.
    • Allow Claude-SearchBot and Claude-User to be eligible for Claude's search answers and for pages users ask it to read.
    • Claude is used heavily for work tasks, so documentation, technical content and B2B comparisons are especially relevant.
    • When Claude answers without searching, model knowledge shapes the answer, so consistent brand facts across the web matter.

    Claude, Anthropic's AI assistant, searches the web when a question needs current or specific information, and cites the pages it draws on inline. Anthropic runs separate bots for search indexing, user-requested fetches and training, so you can choose how each interacts with your site. For brands, the job is the same as on other assistants: be accessible to Claude's search and fetch bots, publish clear answers, and keep your story consistent so the model describes you accurately even when it doesn't search.

    Claude doesn't search for every message. Questions about recent events, specific products, prices, comparisons and anything it isn't confident about are more likely to trigger a search. Users can also ask it to look things up, or paste a link for it to read.

    When Claude answers from its own knowledge, what it says reflects its training data. For the difference between the two, see how AI assistants generate answers.

    Anthropic's three bots

    User agent What it does If you block it
    Claude-SearchBot Indexes content to improve Claude's search results Your pages may appear less in Claude's search answers
    Claude-User Fetches a page when a Claude user's request needs it Claude can't read your pages for users, even when they paste the link
    ClaudeBot Collects content that may be used to train Anthropic's models Your future content is excluded from training; search isn't affected

    Anthropic states that its bots honor robots.txt, supports the non-standard Crawl-delay directive and publishes its crawler IP addresses so you can verify traffic. Our AI crawlers and robots.txt guide has a full configuration.

    A common setup for brands:

    User-agent: Claude-SearchBot
    User-agent: Claude-User
    Allow: /
    
    # Choose based on your training policy
    User-agent: ClaudeBot
    Allow: /
    

    How Claude cites sources

    When Claude searches, it reads the most relevant passages from the results and writes its answer with inline citations pointing to the pages that support specific claims. As with other assistants, you don't need to be the source of the whole answer; being the clearest source for one claim earns a citation.

    Why Claude matters for B2B and technical brands

    Claude is widely used for work: research, analysis, writing and coding. People ask it to compare vendors, explain technical concepts, summarize documentation and draft recommendations for their teams. That makes certain content particularly valuable:

    • Documentation and API references for developer products
    • Technical explainers with precise, verifiable detail
    • B2B comparison pages that lay out trade-offs honestly
    • Security, compliance and integration pages that procurement teams ask about
    • Pricing pages that explain how pricing works for teams

    How to improve your visibility in Claude

    1. Allow Claude-SearchBot and Claude-User in robots.txt and in your CDN or firewall.
    2. Serve content in HTML. Like most AI crawlers, Anthropic's bots read raw HTML.
    3. Write answer-first sections with question-style headings. See how to write content AI assistants cite.
    4. Keep documentation current and well structured, with one task per page or section.
    5. Be consistent about your brand across your site, profiles and listings, so the model's background knowledge matches reality.
    6. Earn third-party coverage in the publications, review sites and communities your buyers read. See how AI assistants decide which brands to recommend.

    Testing and measuring

    • Ask Claude to read a URL. If it can't, check access and rendering.
    • Run your prompt set on Claude regularly, with and without search, and record mentions, framing and cited URLs. See AI visibility metrics.
    • Check logs for Claude-SearchBot and Claude-User visits, verified against Anthropic's published IP list.
    • Track referrals from claude.ai in analytics.

    Worked example: a developer tool

    A company selling an API for sending transactional email wants Claude to recommend it when developers ask, "What's the best email API for a Node.js app sending password resets?"

    What helps:

    • A quickstart page for Node.js that shows working code to send a password-reset email, with the package name and install command in text.
    • A pricing page that states the free tier and the cost per thousand emails plainly.
    • A deliverability page that explains what the service does for inbox placement, with specifics.
    • An honest comparison page against the two best-known alternatives, with a table covering pricing, SDKs, templates and limits.
    • Consistent descriptions on GitHub, package registries and developer communities.
    • Third-party mentions in tutorials and developer forums.

    When Claude searches, those pages give it exact, quotable answers to every part of the question. When it doesn't search, years of consistent descriptions shape what the model already believes about the product.

    Content Claude users ask about most

    Because Claude is used heavily for work, these questions come up often in B2B categories:

    • "Compare [A] and [B] for [team or use case]"
    • "Summarize [product]'s security and compliance"
    • "How do I set up [integration]?"
    • "Draft a recommendation for my manager on which [tool] to choose"
    • "What are the limits of [product]'s free plan?"

    Make sure each has a clear, current answer on your site.

    The bottom line

    Claude finds sources through its own search index and user-requested fetches, and cites them inline. Let Claude-SearchBot and Claude-User in, make a deliberate choice about ClaudeBot, publish precise answer-first content, especially documentation and B2B comparisons, and keep your brand facts consistent so Claude gets you right whether or not it searches.

    For how Claude compares with ChatGPT, Perplexity and Gemini, see how each assistant finds sources.

    FAQ

    Frequently asked questions

    Does Claude search the internet?

    Yes. Claude can search the web when a question needs up-to-date or specific information, and it cites the sources it uses. Whether it searches depends on the question and the user's settings.

    Which crawler does Claude use?

    Anthropic operates Claude-SearchBot to index content for search, Claude-User to fetch pages when a user's request needs them, and ClaudeBot to collect training data. Anthropic says all three respect robots.txt.

    Can I block Claude from training on my site but still appear in its answers?

    Yes. Disallow ClaudeBot to opt out of training while allowing Claude-SearchBot and Claude-User, which handle search and user-requested fetches.

    How do I check whether Claude can read my pages?

    Ask Claude to summarize a specific URL on your site. If it can't access the page, check robots.txt, firewall rules and whether the content is in the raw HTML. Server logs will also show Claude-User and Claude-SearchBot visits.

    • Claude
    • Anthropic
    • citations
    • Claude-SearchBot
    • AI search
    Saarthak Srivastav

    Written by

    Saarthak Srivastav

    AEO & GEO, AEO-Rank

    Saarthak Srivastav writes and reviews articles on the AEO-Rank blog, drawing on what the AEO-Rank team sees tracking AI answers across ChatGPT, Gemini, Claude, Perplexity and Grok.