Cloudflare Implements AI Blocker on Millions of Websites
amsterdam, donderdag, 3 juli 2025.
Cloudflare introduces a built-in AI blocker on millions of websites, preventing tech companies from accessing content without permission for training AI models. This measure provides publishers with more control and is seen as a ‘gamechanger’, with support from media and forum giants such as The Atlantic and Stack Overflow. However, OpenAI refuses to cooperate with the new system.
Cloudflare’s AI Blocker: A New Era for Online Content
The internet company Cloudflare introduces a built-in AI blocker on millions of websites, preventing tech companies from accessing content without permission for training AI models. This measure provides publishers with more control and is seen as a ‘gamechanger’, with support from media and forum giants such as The Atlantic and Stack Overflow. However, OpenAI refuses to cooperate with the new system [1][2][3].
How Does the AI Blocker Work?
Cloudflare’s AI blocker is designed to block AI bots that visit websites to scrape content for training AI models. Website owners can choose which bots to allow or block, and they can even set rates for access to their content, known as ‘Pay Per Crawl’. This gives publishers the ability to decide who can use their content and at what price [2][4].
Impact on Publishers and Content Providers
This measure provides publishers with more control over their content and helps them retain the value of their work. According to Matthew Prince, co-founder and CEO of Cloudflare, the goal is to give power back to content creators while still allowing room for innovation in AI companies. Media companies such as The Associated Press, Time, Quora, and Stack Overflow have already supported the measure [2][3][4].
Reactions from the Tech Industry
While many publishers are enthusiastic about the new measures, there are also critical reactions. OpenAI, one of the largest AI companies, refuses to cooperate with Cloudflare’s new system. They argue that adding an intermediary complicates the process and that they consider publishers’ preferences through robots.txt files [2][3][5].
Ethical Considerations and Possible Drawbacks
The implementation of the AI blocker also raises ethical considerations and possible drawbacks. Some experts point out that the default blocking of AI bots could hinder non-commercial uses, such as research. Shayne Longpre, a PhD candidate at the MIT Media Lab, argues that not all AI systems are commercial and that personal use and open research should not be sacrificed [2][5].
Technical Details and Implementation
Cloudflare has experience in blocking harmful bots, such as during DDoS attacks. The new AI blocker uses similar techniques to identify and block unreliable crawlers. Website owners can now gain insight into how often an AI model sends traffic to a site compared to how often it scrapes the site, thanks to the new metric on the AI Insights page of Cloudflare Radar [3][4][6].
Future Perspectives
The implementation of the AI blocker by Cloudflare is seen as a significant step in protecting online content. Matthew Prince emphasizes that the goal is to secure the future of a free and vibrant internet with a new model that works for everyone. It will be interesting to see how these measures develop and what impact they ultimately have on the AI industry and the digital content market [2][3][4][5].