Cloudflare enforces AI crawler separation policy by September 15
Original: Cloudflare’s new policy pushes AI companies to pay for publishers’ content
Why This Matters
Addresses growing tension between AI companies and publishers over content licensing by creating infrastructure forcing commercial negotiations.
Cloudflare announced on July 1, 2026, that starting September 15, 2026, it will block mixed-use AI crawlers from ad-supported pages by default, pushing AI companies to separate search crawlers from training and agent-use crawlers or negotiate content licensing with publishers.
Cloudflare issued a new policy requiring AI companies to separate web crawlers used for traditional search from those used for AI training and agent services. Beginning September 15, 2026, Cloudflare's default settings will block combined-use crawlers from pages hosting advertisements unless site owners manually adjust their settings. The policy applies to new customers, new sites from existing customers, and all free-tier users. Cloudflare CEO Matthew Prince stated: "Now that the majority of traffic on the Internet is non-human, we must go further and act faster so that a sustainable ecosystem can emerge." The announcement references a recent milestone where bot traffic surpassed human traffic online for the first time. Cloudflare specifically highlighted that major search engines like Google access approximately twice as much information as other AI companies due to their integrated search-to-AI infrastructure. The company noted that while Google offers the Google Extended bot allowing opt-out for training use without affecting Search inclusion, its main Googlebot crawls for both Search and AI features like AI Overviews. Cloudflare has previously launched tools including a Pay Per Crawl marketplace allowing publishers to charge AI companies for content scraping, now evolving into a Pay Per Use model for content licensing agreements.