Yes, it's common: a robots.txt that welcomes AI crawlers means nothing if your CDN, firewall or security plugin blocks or challenges them first. robots.txt is a request that good crawlers honor; the network edge is where access is actually decided. This post covers where accidental blocks come from, what Cloudflare's AI crawler controls do today, how AWS WAF treats AI bots, how to test properly, and a safe configuration.
Where accidental blocks happen
Most AI crawler blocks are defaults and old rules nobody revisited, not deliberate decisions:
| Layer | How it blocks AI crawlers |
|---|---|
| CDN AI bot settings | One-click or default "block AI bots" options that cover more crawlers than intended |
| Bot fight or bot management modes | Challenge anything that looks automated and isn't on an allowlist |
| WAF custom rules | Old user-agent blocklists ("block anything containing 'bot'"), country blocks, ASN blocks on cloud providers |
| Rate limiting | Crawlers that fetch many pages quickly get throttled into 429s |
| JavaScript or CAPTCHA challenges | A challenge is a block for a crawler that can't solve it |
| Hosting platform defaults | Managed hosts and site builders with their own bot rules |
| Security plugins | CMS firewall plugins with bot blocklists |
In every case the crawler gets a 403, a 429 or a challenge page instead of your content, and you quietly drop out of answers. For which crawlers to allow in the first place, start with AI crawlers and robots.txt.
What Cloudflare's AI crawler controls do now
Cloudflare's AI crawler settings have changed several times since 2025.
July 2025: Cloudflare announced it would block AI crawlers by default on newly added domains, alongside a pay-per-crawl option, as covered by MIT Technology Review and Cloudflare's own posts.
July 1, 2026: Cloudflare's changelog introduced separate controls for three kinds of AI traffic, available to all plans including Free:
- Search: crawlers that build search indexes
- Agent: AI agents fetching pages on behalf of a user (the kind of request an assistant makes when someone asks it about your page)
- Training: crawlers that collect content for model training
For each, you can allow, block on all pages, or block only on pages that display ads.
September 15, 2026: Cloudflare changed the defaults for new domains so that Training and Agent bots are blocked on pages that display ads, while Search stays allowed. Cloudflare also said that Block and "Block on pages with ads" now apply to mixed-use crawlers, including Googlebot, Bingbot and Applebot, so choosing either for Training can stop those crawlers for search too. Its "Disallow AI Training" option is meant for that case: it keeps those crawlers' search access while signaling that content shouldn't be used for training.
Bot Preference Sync (Cloudflare's feature for writing your AI bot choices into robots.txt) can update your robots.txt to match these settings. If it's on, check that the resulting file still says what you intend.
What to check in your Cloudflare dashboard:
- The AI traffic setting for Search is Allow
- Agent is Allow, unless you've decided assistants shouldn't fetch pages for users
- Training reflects a deliberate decision; if you restrict it, use Disallow AI Training rather than Block, so Googlebot and Bingbot aren't caught
- Bot Fight Mode: Cloudflare's docs say it can't be bypassed with WAF skip rules, so on plans where it's on, check its effect in security events
- Custom rules and rate limits don't match verified crawlers
Settings and names change, so confirm against Cloudflare's current documentation before you edit. For the training decision itself, see should you block AI training crawlers?.
AWS WAF and other platforms
Other CDNs and WAFs have their own AI bot categories, and their defaults deserve the same review. In AWS WAF, the Bot Control managed rule group's category rules normally match only unverified bots, but AWS documents that its AI category rule (CategoryAI) also matches verified bots. If you've enabled Bot Control, check what action that rule takes and whether you need to override it to Count and allow specific AI crawlers with a custom rule that uses Bot Control's labels.
For any other provider, ask: does it have an AI bot category, what is its default action, and does it separate search from training crawlers?
How to test properly
The reliable test is your logs, not a spoofed request. A curl with a crawler's user agent is useful for catching user-agent-based rules, but bot protection systems verify crawlers by IP and other signals, so your fake "OAI-SearchBot" from an office IP will be treated as an impostor.
Quick check for user-agent rules:
for ua in "OAI-SearchBot/1.0" "ChatGPT-User/1.0" "Claude-SearchBot/1.0" "PerplexityBot/1.0" "Googlebot/2.1"; do
printf "%-22s " "$ua"
curl -s -o /dev/null -w "%{http_code}\n" -A "Mozilla/5.0 (compatible; $ua)" https://www.example.com/pricing/
done
A 403 for every bot name but a 200 in your browser points to a user-agent rule. A 200 here doesn't prove real crawlers get through.
The real evidence:
- CDN security events: filter by user agent or verified bot name and look for blocks, challenges and rate limits on AI crawlers.
- Origin server logs: confirm AI crawlers reach your server and get 200s. If a bot is allowed in robots.txt but never appears in origin logs, something upstream is stopping it. See finding AI crawler activity in your server logs.
- A live test: ask ChatGPT, Claude or Perplexity to open and summarize a specific new URL. If it can't, check the logs for that moment.
Many AI companies publish the IP ranges of their crawlers, which is how you or your CDN can tell real crawlers from impostors.
A safe configuration
A good setup allows verified search crawlers and user-triggered fetchers, makes the training decision deliberately, and keeps protection on for everything else. On Cloudflare, a first-position custom rule can let verified bots past your other custom rules:
Rule name: Skip custom rules for verified bots
Expression: (cf.client.bot)
Action: Skip
- All remaining custom rules
- Rate limiting rules
cf.client.bot matches Cloudflare's verified bots list. This rule only skips the components you select; your AI traffic settings are configured separately, so the Search, Agent and Training choices above remain the place to control AI crawlers. For finer control, Cloudflare offers a verified bot category field; its category values were revised in July 2026, so copy them from the current docs rather than an old blog post.
Elsewhere:
- Replace broad user-agent blocklists with verified-bot logic
- Exclude verified crawlers from aggressive rate limits, or raise limits for them
- Don't challenge crawlers on pages you want cited: pricing, docs, comparisons, blog
- Keep login, checkout and account areas protected as usual
User-triggered fetchers matter more as assistants act for users; see AI agents and agentic browsing.
Example: finding a silent block
This example is illustrative, with made-up details.
A B2B company notices that Perplexity hasn't cited its new comparison pages in two months, though older pages still appear. robots.txt allows PerplexityBot. Origin logs show PerplexityBot requests stopped in midsummer.
In the CDN's security events, they find PerplexityBot requests hitting a rate-limiting rule added during a traffic spike, returning 429s on any burst of more than a handful of requests. Older pages were already indexed, so only new content suffered. They exclude verified bots from that rule, request recrawls where possible, and confirm 200s for PerplexityBot in origin logs within days.
Ongoing monitoring
Bot settings drift as security teams respond to incidents and vendors change defaults. Add to your routine:
- A monthly review of CDN security events filtered to AI crawlers
- An alert when a verified AI crawler's 403 or 429 rate spikes in logs
- A check of AI bot settings after any CDN vendor announcement
- A line in your change process: security rule changes get reviewed for crawler impact
Add the access checks to your technical AEO audit so they're never skipped.
The bottom line
Your robots.txt states your policy, but your CDN and firewall enforce it, and they often enforce something stricter than you intended. Review your CDN's AI traffic settings (on Cloudflare: Search and Agent allowed, Training decided deliberately without catching Googlebot), remove broad bot blocklists and crawler-hostile rate limits, verify with security events and origin logs rather than spoofed tests, and recheck whenever settings or vendor defaults change.
