Pay per crawl
Pay per crawl is a Cloudflare feature that lets website owners charge AI crawlers a fee each time they fetch a page. A crawler that agrees to pay gets the page. One that doesn't gets a "402 Payment Required" reply with the price. The owner sets one price and chooses, crawler by crawler, to allow, charge or block.
Source: Cloudflare Docs, AI Crawl Control: What is pay per crawl?, checked October 29, 2026
What Pay per crawl means for your business
Cloudflare lists pay per crawl as a closed beta. Sites join through a sign-up page, or through their account team if they're Enterprise customers. It only covers sites that run through Cloudflare. Crawlers that want to pay have to sign up with Cloudflare and identify themselves with Web Bot Auth, a way of signing their requests.
A crawler pays again every time it comes back to the same page, Cloudflare's FAQ says, while robots.txt and sitemap files stay free and error responses aren't charged. If Cloudflare's firewall or bot tools block a crawler, the block wins and the crawler can't buy its way in.
A crawler that won't pay doesn't get the page, so it can't use what's on it. We can see that trade making sense for a publisher whose writing is the product. For a plumber or a hotel, the pages are there to be found, so we'd leave the crawlers behind AI search free.
If you're weighing what to let AI bots see and what to hold back, setting your crawler rules with search in mind is work we can help with.
Last reviewed October 29, 2026. All terms
How AI answers it today
On October 29, 2026, Google showed no AI Overview, and the top results were a Reddit thread, Cloudflare's docs and videos. ChatGPT cited only Cloudflare's pages but described pay per crawl as a general business model first, with Cloudflare's feature as one example.
Our run through ChatGPT and Google's AI Overview, October 29, 2026. A sample, not what every buyer sees.
Related terms in crawlers and technical access
Find out where your brand stands.
Tell us what you're trying to figure out. We'll get back to you within one business day with a time to talk.