Technical AEO

    CDNs and Bot Protection: Are You Blocking AI Crawlers by Accident?

    How CDN and WAF bot settings, including Cloudflare's AI crawler defaults, can block AI search crawlers your robots.txt allows, and how to check and fix it.

    Saarthak Srivastav

    Saarthak Srivastav

    · 6 min read

    Key takeaways

    • robots.txt is only a request; your CDN or firewall decides who actually gets through, and it can block AI crawlers your robots.txt allows.
    • Cloudflare has blocked AI crawlers by default on new domains since July 2025, and since July 2026 splits AI traffic into Search, Agent and Training categories.
    • Cloudflare says Block settings for training also apply to mixed-use crawlers like Googlebot and Bingbot, so a careless choice can hurt regular search.
    • Bot challenges, rate limits, country blocks and security plugins are common accidental blockers; crawlers can't solve challenges.
    • Check CDN security events and server logs for blocked AI crawlers rather than trusting a test with a spoofed user agent.

    Yes, it's common: a robots.txt that welcomes AI crawlers means nothing if your CDN, firewall or security plugin blocks or challenges them first. robots.txt is a request that good crawlers honor; the network edge is where access is actually decided. This post covers where accidental blocks come from, what Cloudflare's AI crawler controls do today, how AWS WAF treats AI bots, how to test properly, and a safe configuration.

    Where accidental blocks happen

    Most AI crawler blocks are defaults and old rules nobody revisited, not deliberate decisions:

    Layer How it blocks AI crawlers
    CDN AI bot settings One-click or default "block AI bots" options that cover more crawlers than intended
    Bot fight or bot management modes Challenge anything that looks automated and isn't on an allowlist
    WAF custom rules Old user-agent blocklists ("block anything containing 'bot'"), country blocks, ASN blocks on cloud providers
    Rate limiting Crawlers that fetch many pages quickly get throttled into 429s
    JavaScript or CAPTCHA challenges A challenge is a block for a crawler that can't solve it
    Hosting platform defaults Managed hosts and site builders with their own bot rules
    Security plugins CMS firewall plugins with bot blocklists

    In every case the crawler gets a 403, a 429 or a challenge page instead of your content, and you quietly drop out of answers. For which crawlers to allow in the first place, start with AI crawlers and robots.txt.

    What Cloudflare's AI crawler controls do now

    Cloudflare's AI crawler settings have changed several times since 2025.

    July 2025: Cloudflare announced it would block AI crawlers by default on newly added domains, alongside a pay-per-crawl option, as covered by MIT Technology Review and Cloudflare's own posts.

    July 1, 2026: Cloudflare's changelog introduced separate controls for three kinds of AI traffic, available to all plans including Free:

    • Search: crawlers that build search indexes
    • Agent: AI agents fetching pages on behalf of a user (the kind of request an assistant makes when someone asks it about your page)
    • Training: crawlers that collect content for model training

    For each, you can allow, block on all pages, or block only on pages that display ads.

    September 15, 2026: Cloudflare changed the defaults for new domains so that Training and Agent bots are blocked on pages that display ads, while Search stays allowed. Cloudflare also said that Block and "Block on pages with ads" now apply to mixed-use crawlers, including Googlebot, Bingbot and Applebot, so choosing either for Training can stop those crawlers for search too. Its "Disallow AI Training" option is meant for that case: it keeps those crawlers' search access while signaling that content shouldn't be used for training.

    Bot Preference Sync (Cloudflare's feature for writing your AI bot choices into robots.txt) can update your robots.txt to match these settings. If it's on, check that the resulting file still says what you intend.

    What to check in your Cloudflare dashboard:

    • The AI traffic setting for Search is Allow
    • Agent is Allow, unless you've decided assistants shouldn't fetch pages for users
    • Training reflects a deliberate decision; if you restrict it, use Disallow AI Training rather than Block, so Googlebot and Bingbot aren't caught
    • Bot Fight Mode: Cloudflare's docs say it can't be bypassed with WAF skip rules, so on plans where it's on, check its effect in security events
    • Custom rules and rate limits don't match verified crawlers

    Settings and names change, so confirm against Cloudflare's current documentation before you edit. For the training decision itself, see should you block AI training crawlers?.

    AWS WAF and other platforms

    Other CDNs and WAFs have their own AI bot categories, and their defaults deserve the same review. In AWS WAF, the Bot Control managed rule group's category rules normally match only unverified bots, but AWS documents that its AI category rule (CategoryAI) also matches verified bots. If you've enabled Bot Control, check what action that rule takes and whether you need to override it to Count and allow specific AI crawlers with a custom rule that uses Bot Control's labels.

    For any other provider, ask: does it have an AI bot category, what is its default action, and does it separate search from training crawlers?

    How to test properly

    The reliable test is your logs, not a spoofed request. A curl with a crawler's user agent is useful for catching user-agent-based rules, but bot protection systems verify crawlers by IP and other signals, so your fake "OAI-SearchBot" from an office IP will be treated as an impostor.

    Quick check for user-agent rules:

    for ua in "OAI-SearchBot/1.0" "ChatGPT-User/1.0" "Claude-SearchBot/1.0" "PerplexityBot/1.0" "Googlebot/2.1"; do
      printf "%-22s " "$ua"
      curl -s -o /dev/null -w "%{http_code}\n" -A "Mozilla/5.0 (compatible; $ua)" https://www.example.com/pricing/
    done
    

    A 403 for every bot name but a 200 in your browser points to a user-agent rule. A 200 here doesn't prove real crawlers get through.

    The real evidence:

    1. CDN security events: filter by user agent or verified bot name and look for blocks, challenges and rate limits on AI crawlers.
    2. Origin server logs: confirm AI crawlers reach your server and get 200s. If a bot is allowed in robots.txt but never appears in origin logs, something upstream is stopping it. See finding AI crawler activity in your server logs.
    3. A live test: ask ChatGPT, Claude or Perplexity to open and summarize a specific new URL. If it can't, check the logs for that moment.

    Many AI companies publish the IP ranges of their crawlers, which is how you or your CDN can tell real crawlers from impostors.

    A safe configuration

    A good setup allows verified search crawlers and user-triggered fetchers, makes the training decision deliberately, and keeps protection on for everything else. On Cloudflare, a first-position custom rule can let verified bots past your other custom rules:

    Rule name: Skip custom rules for verified bots
    Expression: (cf.client.bot)
    Action: Skip
      - All remaining custom rules
      - Rate limiting rules
    

    cf.client.bot matches Cloudflare's verified bots list. This rule only skips the components you select; your AI traffic settings are configured separately, so the Search, Agent and Training choices above remain the place to control AI crawlers. For finer control, Cloudflare offers a verified bot category field; its category values were revised in July 2026, so copy them from the current docs rather than an old blog post.

    Elsewhere:

    • Replace broad user-agent blocklists with verified-bot logic
    • Exclude verified crawlers from aggressive rate limits, or raise limits for them
    • Don't challenge crawlers on pages you want cited: pricing, docs, comparisons, blog
    • Keep login, checkout and account areas protected as usual

    User-triggered fetchers matter more as assistants act for users; see AI agents and agentic browsing.

    Example: finding a silent block

    This example is illustrative, with made-up details.

    A B2B company notices that Perplexity hasn't cited its new comparison pages in two months, though older pages still appear. robots.txt allows PerplexityBot. Origin logs show PerplexityBot requests stopped in midsummer.

    In the CDN's security events, they find PerplexityBot requests hitting a rate-limiting rule added during a traffic spike, returning 429s on any burst of more than a handful of requests. Older pages were already indexed, so only new content suffered. They exclude verified bots from that rule, request recrawls where possible, and confirm 200s for PerplexityBot in origin logs within days.

    Ongoing monitoring

    Bot settings drift as security teams respond to incidents and vendors change defaults. Add to your routine:

    • A monthly review of CDN security events filtered to AI crawlers
    • An alert when a verified AI crawler's 403 or 429 rate spikes in logs
    • A check of AI bot settings after any CDN vendor announcement
    • A line in your change process: security rule changes get reviewed for crawler impact

    Add the access checks to your technical AEO audit so they're never skipped.

    The bottom line

    Your robots.txt states your policy, but your CDN and firewall enforce it, and they often enforce something stricter than you intended. Review your CDN's AI traffic settings (on Cloudflare: Search and Agent allowed, Training decided deliberately without catching Googlebot), remove broad bot blocklists and crawler-hostile rate limits, verify with security events and origin logs rather than spoofed tests, and recheck whenever settings or vendor defaults change.

    FAQ

    Frequently asked questions

    Does Cloudflare block ChatGPT and Perplexity by default?

    Since July 2025, Cloudflare has blocked AI crawlers by default on newly added domains. Since September 15, 2026, new domains block Training and Agent bots on pages that show ads while Search is allowed. Existing domains keep whatever settings they have, so check your dashboard.

    If robots.txt allows a bot, can it still be blocked?

    Yes. robots.txt is read by well-behaved crawlers, but your CDN, WAF, hosting platform or security plugin can still return a 403 or a challenge page. Those layers are where most accidental blocks happen.

    How can I tell if my CDN is blocking AI crawlers?

    Filter your CDN's security events or firewall logs for AI crawler user agents and look for blocks and challenges. Then check origin server logs for those bots; if they never appear, something upstream is stopping them.

    Can I block AI training but still appear in AI search?

    Yes. Allow search crawlers like OAI-SearchBot, Claude-SearchBot and PerplexityBot while blocking training crawlers like GPTBot or ClaudeBot. On Cloudflare, be careful with mixed-use crawlers such as Googlebot, where Cloudflare offers a Disallow AI Training option instead of a hard block.

    Can AI crawlers pass CAPTCHA or JavaScript challenges?

    Generally no. Automated crawlers don't solve interactive challenges, so a challenge works like a block for them, even if humans pass it easily.

    • Cloudflare
    • WAF
    • bot protection
    • AI crawlers
    • CDN
    Saarthak Srivastav

    Written by

    Saarthak Srivastav

    AEO & GEO, AEO-Rank

    Saarthak Srivastav writes and reviews articles on the AEO-Rank blog, drawing on what the AEO-Rank team sees tracking AI answers across ChatGPT, Gemini, Claude, Perplexity and Grok.