Your server and CDN logs are the only direct record of which AI crawlers visit your site, what they fetch and whether they succeed. Analytics tools usually filter bots out, so the logs are where you'll see GPTBot, OAI-SearchBot, ClaudeBot, PerplexityBot and the rest. This guide shows how to find them, verify them, and turn what you see into fixes.
What logs tell you that nothing else does
- Whether AI crawlers can reach you: successful responses vs blocks and errors
- Which pages they fetch, and which they never touch
- How often they return, which hints at how fresh their copy is
- User-triggered fetches, which mean real people asked an assistant about your pages
- Whether your robots.txt and firewall rules work the way you intended
Step 1: Get the logs
Depending on your setup:
- Web server logs (Nginx, Apache) on your hosting
- CDN logs (Cloudflare, Fastly, Akamai, CloudFront), often more complete because they see traffic before your origin
- Platform logs on managed hosts, sometimes via an add-on
You need at least: timestamp, IP address, requested URL, status code and user agent.
Step 2: Filter for AI user agents
Search for these strings:
| User agent contains | Operator | Purpose |
|---|---|---|
GPTBot |
OpenAI | Training |
OAI-SearchBot |
OpenAI | ChatGPT search index |
ChatGPT-User |
OpenAI | User-triggered fetch |
ClaudeBot |
Anthropic | Training |
Claude-SearchBot |
Anthropic | Search index |
Claude-User |
Anthropic | User-triggered fetch |
PerplexityBot |
Perplexity | Search index |
Perplexity-User |
Perplexity | User-triggered fetch |
Googlebot |
Search, including AI Overviews and AI Mode | |
bingbot |
Microsoft | Bing and Copilot |
Applebot |
Apple | Siri and Spotlight search |
CCBot |
Common Crawl | Open dataset used for training |
meta-externalagent |
Meta | AI crawling |
Amazonbot |
Amazon | Amazon services |
A quick command-line example:
grep -Ei "GPTBot|OAI-SearchBot|ChatGPT-User|ClaudeBot|Claude-SearchBot|Claude-User|PerplexityBot|Perplexity-User" access.log | awk '{print $1, $7, $9}' | head
See our AI crawlers and robots.txt guide for what each bot does.
Step 3: Verify the traffic
User agents can be spoofed. For important decisions, verify:
- OpenAI, Anthropic and Perplexity publish IP ranges for their crawlers. Check requests against those lists.
- Googlebot and Bingbot can be verified with a reverse DNS lookup followed by a forward lookup, as Google and Microsoft document.
Unverified requests claiming to be AI bots may be scrapers.
Step 4: Analyze
Status codes
Group AI bot requests by status code:
- 200: success
- 301/302: redirects; fine in moderation, but chains waste crawl effort
- 403/401: blocked, often by a firewall or bot protection
- 404: broken links or removed pages
- 429: rate-limited
- 5xx: server errors
A high share of 403s or 429s for search crawlers you want means your security setup is undoing your robots.txt.
Pages crawled
List the URLs each AI search bot fetched. Compare with your priority pages: pricing, product, comparison, docs. Are the important pages being crawled? If not, check internal linking, sitemaps and access. See site architecture for AI retrieval.
Frequency
How often do search bots return to key pages? Infrequent visits to pages that change often (pricing, docs) can mean AI answers use stale copies. Improve internal links and sitemaps, and use IndexNow for Bing. See Bing Webmaster Tools and IndexNow.
User-triggered fetches
ChatGPT-User, Claude-User and Perplexity-User requests usually mean a user asked about your page, or the assistant decided to read it to answer them. The URLs they fetch show what people are asking assistants about.
Response size
If an AI bot receives a 200 but a tiny response, it may be getting an empty client-rendered shell. See JavaScript rendering and AI crawlers.
Step 5: Act on it
| Finding | Action |
|---|---|
| Search bots getting 403s | Allow them in CDN or firewall rules; verify IPs |
| Key pages never crawled | Add internal links; include in sitemaps |
| Tiny responses | Server-render or prerender the pages |
| Many 404s | Fix links; redirect removed pages |
| Heavy load from training crawlers | Use Crawl-delay where supported, rate limits or robots.txt rules |
| Unverified "bots" scraping | Block by behavior and IP, not by user agent alone |
Make it a monthly report
Track, for each AI search crawler:
- Requests and unique URLs fetched
- Share of successful responses
- Coverage of priority pages
- User-triggered fetch counts and top URLs
Put it next to your AI visibility metrics. If crawl access improves and citations follow, you'll see the connection.
The bottom line
Logs turn AI crawling from guesswork into data. Filter for the main AI user agents, verify the traffic, check status codes, coverage and frequency, and fix what's blocking or confusing the crawlers you want. It's the most reliable way to confirm that the assistants you care about can actually read your site.
