Cloudflare Gives Websites Control Over AI Web Crawlers

Website owners are finally getting the tools they need to decide how AI companies use their content, thanks to a major policy shift from internet giant Cloudflare.

The hosting platform announced it will automatically block mixed-use web crawlers that both index websites for search engines and train AI models at the same time. Starting September 15, 2026, new Cloudflare customers will have their sites protected by default, blocking AI training and AI agent use on pages with advertisements while still allowing traditional search indexing.

"Now that the majority of traffic on the Internet is non-human, we must go further and act faster so that a sustainable ecosystem can emerge," said Matthew Prince, Cloudflare's CEO and co-founder. The company hosts millions of websites worldwide, making this change significant for the entire internet.

The decision addresses a growing problem: AI bots visiting websites don't generate revenue through ads or subscriptions like human visitors do. These crawlers extract information to train AI models or answer chatbot questions, essentially taking content without compensation.

Cloudflare is also upgrading its payment system for AI companies. The new "Pay Per Use" feature means website owners get paid when their content appears in AI chatbot answers, not just when pages get crawled. Partners like Ceramic.AI and You.com have already signed on.

The Ripple Effect

The move puts pressure on Google, whose main crawler Googlebot serves double duty for both traditional search and AI training. Publishers currently face an all-or-nothing choice: allow Google to use their content for everything, or risk disappearing from search results entirely.

Cloudflare's approach forces companies to be transparent about what they're doing with website content. Mixed-use crawlers that don't let site owners choose will be automatically blocked on pages with ads.

The change represents a fundamental shift in how the internet works. Website creators invest time and money producing content, and they deserve control over whether AI companies can use that work for free.

Free Cloudflare account holders will also switch to the new protective defaults unless they opt out before the September deadline. This ensures even small website owners without technical expertise get the same protection as major publishers.

The internet is becoming a fairer place, one website at a time.