Cloudflare to block AI bots from scraping website content

Cloudflare has begun blocking AI crawlers from scraping content from websites by default and has introduced a new model that allows publishers to charge bots for access.

The company announced that it will no longer allow artificial intelligence (AI) companies to freely access content hosted on its platform unless explicitly permitted by website owners. This takes effect immediately for all new websites signing up on Cloudflare.

Matthew Prince, chief executive officer of Cloudflare, stated that the company’s decision to block AI crawlers by default is aimed at rebalancing control in favour of content creators, while still supporting innovation in AI development.

Read also: Group unveils Artificial Intelligence to boost business data collection

“AI crawlers have been scraping content without limits. Our goal is to put the power back in the hands of creators while still helping AI companies innovate. This is about safeguarding the future of a free and vibrant Internet with a new model that works for everyone,” Prince stated.

This latest move builds on a feature Cloudflare introduced in September 2023, which allowed publishers to block AI crawlers with a single click. Going forward, blocking these bots will be the default setting for all websites under Cloudflare’s network.

As part of the new framework, Cloudflare is also introducing a ‘pay-per-crawl’ option that allows publishers to charge AI companies that want to access their content. Cloudflare’s content delivery network (CDN) handles a significant share of global internet traffic, making this decision impactful for AI companies that rely heavily on public data for model training.