Skip to content
Cloudflare Docs

Analyze AI traffic

AI Crawl Control metrics provide you with insight on how AI crawlers are interacting with your website (Cloudflare zone).

To view AI Crawl Control metrics:

  1. Log in to the Cloudflare dashboard ↗, and select your account and domain.

  2. Go to AI Crawl Control.

    Go to AI Crawl Control

You can find meaningful information across both Crawlers and Metrics tabs.

View AI crawler activity

The Crawlers tab provides you with the following information:

MetricDescription
Total requestsTotal number of requests to crawl your website from common AI crawlers.
Requests by crawlerNumber of requests made by each AI crawler.

View AI Crawl Control metrics

The Metrics tab provides you with the following metrics to help you understand how AI crawlers are interacting with your website.

Analyze referrer data

Identify traffic sources with referrer analytics to understand discovery patterns and content popularity from AI operators.

  • View top referrers driving traffic to your site.
  • Understand discovery patterns and content popularity from AI operators.

Track successful crawler requests over time

Visualize successful crawler activity patterns over time using the Successful requests over time chart, which shows requests with status codes 200-300. You can group data by different dimensions to get more specific insights:

DimensionDescription
CrawlerTrack activity from individual AI crawlers (like GPTBot, ClaudeBot, and Bytespider).
CategoryAnalyze crawlers by their purpose or type.
OperatorDiscover which companies (such as OpenAI, Anthropic, and ByteDance) are crawling your site.
HostBreak down activity across multiple subdomains.

Monitor status code distribution

The Status code distribution chart shows HTTP response codes (2xx, 3xx, 4xx, 5xx) for AI crawler requests over time. This provides detailed visibility beyond the sparkline charts shown in the Crawlers tab.

The chart displays:

  • Total request counts for each status code category (2xx, 3xx, 4xx, 5xx)
  • Time series visualization showing status code trends over your selected time period
  • Filtering by crawler, category, operator, and time range

Use this chart to identify patterns in how your site responds to AI crawlers in areas such as the distribution of error codes or the volume of redirects.

Understand what content is crawled

The Most popular paths table shows you which pages on your site are most frequently requested by AI crawlers. This can help you understand what content is most popular with different AI models.

ColumnDescription
PathThe path of the page on your website that was requested.
HostnameThe hostname of the requested page.
CrawlerThe name of the AI crawler that made the request.
OperatorThe company that operates the AI crawler.
Allowed requestsThe number of times the path was successfully requested by the crawler.

You can also filter the results by path or content type to narrow down your analysis.

Filter and export data

You can use the date filter to choose the period of time you wish to analyze. To export your data, select Download CSV. The downloaded file will include all applied filters and groupings.

Filter options:

  • Past 24 hours

The values of the AI Crawl Control metrics will update according to your filter.

Per-crawler drilldowns

The Crawlers tab includes an actions menu for each crawler with the following options:

ActionDescription
View MetricsFilter the Metrics tab to the selected crawler.
View on Cloudflare RadarLearn more about each verified crawler on Cloudflare Radar.
View in Security AnalyticsFilter Security Analytics by detection IDs (Bot Management customers).
Copy User AgentCopy user agent strings for use in WAF custom rules, Redirect Rules, or robots.txt files.
Copy Detection IDCopy detection IDs for use in WAF custom rules (Bot Management customers).