Cloudflare Introduces Pay per Crawl to Make AI Bots Pay for Content
amsterdam, woensdag, 2 juli 2025.
Cloudflare has introduced a new system called Pay per Crawl, which allows content creators to charge AI bots for accessing their content. This provides website owners with more control over how their content is used and helps protect intellectual property. The service is currently in private beta and leverages existing web infrastructure, such as HTTP status codes and authentication mechanisms, to create a payment mechanism. Website owners can choose to grant crawlers free access, block them, or make them pay, creating a new economic model for the digital world.
New System Offers More Control
With Cloudflare’s new Pay per Crawl system, website owners can decide whether to grant AI crawlers free access, block them, or charge them for crawling their content [1]. This gives website owners more control over their digital assets and helps protect intellectual property. Matthew Prince, co-founder and CEO of Cloudflare, states: “If the internet is to survive the AI age, we need to give publishers the control they deserve and build a new economic model that works for everyone – creators, consumers, future AI founders, and the future of the web itself.” [3]
Technical Functioning of Pay per Crawl
The Pay per Crawl system uses HTTP status codes and existing authentication mechanisms to create a payment mechanism for content access [1]. Domain owners have full control over their monetisation strategy and can choose to grant crawlers free access, make them pay, or block them entirely [1]. If a crawler does not have a billing relationship with Cloudflare, the publisher can still ‘make them pay,’ which functionally equates to a network-level block (HTTP 403 Forbidden response) [1].
Beta Phase and Implementation
Pay per Crawl is currently in private beta and integrates with existing web infrastructure [1]. Crawler operators must generate an Ed25519 key pair, register with Cloudflare, and use HTTP Message Signatures for each request [1]. Crawler requests can be reactive or proactive: in a reactive request, the crawler receives an HTTP 402 Payment Required if payment is required, while in a proactive request, the crawler can specify a maximum price in advance [1]. Financial settlements occur through the Cloudflare account, where the crawler is billed, and the proceeds are distributed to the publisher [1].
Support from Publishers and AI Companies
Top global publishers, media, and technology companies, including ADWEEK, The Associated Press, Condé Nast, Gannett Media, and Reddit, support the permission-based model for AI crawling [3]. Roger Lynch, CEO of Condé Nast, notes: “Cloudflare’s innovative approach to blocking AI crawlers is a game changer for publishers and sets a new standard for how content is respected online.” [3] Nicholas Thompson, CEO of The Atlantic, emphasises: “This could dramatically change the power dynamics. Until now, AI companies didn’t have to pay to license content because they knew they could just take it without consequences.” [4]
Impact on Content Distribution and Monetisation
The tradition of the internet, where search engines index content and redirect users to original websites, is being undermined by AI crawlers that collect content without directing visitors to the original sources [3]. This new policy from Cloudflare aims to preserve a sustainable future for both content creators and AI innovators [3]. The new default setting from Cloudflare is the first step towards a more sustainable future for content creators and AI innovators [3].
Expansion of Cloudflare Radar
Cloudflare Radar now offers new insights into the behaviour of AI bots, including the crawl-to-refer ratio, which shows the ratio between the number of HTML page requests from crawlers and the number of HTML page requests with referring headers [2]. This ratio helps website owners decide which AI bots to allow and which to block [2]. Time series data is available via the Data Explorer and an API endpoint, assisting website owners in understanding the impact of AI bots on their internet traffic [2].
Control and Transparency
Cloudflare is also introducing an extensive verified bots directory with detailed information about each bot, including name, owner, category, and rank [2]. The new directory enables users to search and filter bots based on various criteria [2]. The bot-specific pages provide metadata, user-agent information, and traffic trends for each bot, increasing transparency and control for content creators [2].