Skip to content
Cloudflare Docs

Changelog

New updates and improvements at Cloudflare.

Subscribe to RSS
View all RSS feeds

hero image
← Back to all posts

Enhanced crawler insights and custom 402 responses

AI Crawl Control

We improved AI crawler management with detailed analytics and introduced custom HTTP 402 responses for blocked crawlers. AI Audit has been renamed to AI Crawl Control and is now generally available.

Enhanced Crawlers tab:

  • View total allowed and blocked requests for each AI crawler
  • Trend charts show crawler activity over your selected time range per crawler
Updated AI Crawl Control table showing request counts and trend charts

Custom block responses (paid plans): You can now return HTTP 402 "Payment Required" responses when blocking AI crawlers, enabling direct communication with crawler operators about licensing terms.

For users on paid plans, when blocking AI crawlers you can configure:

  • Response code: Choose between 403 Forbidden or 402 Payment Required
  • Response body: Add a custom message with your licensing contact information
AI Crawl Control block response configuration interface

Example 402 response:

HTTP 402 Payment Required
Date: Mon, 24 Aug 2025 12:56:49 GMT
Content-type: application/json
Server: cloudflare
Cf-Ray: 967e8da599d0c3fa-EWR
Cf-Team: 2902f6db750000c3fa1e2ef400000001


{
  "message": "Please contact the site owner for access."
}