Cloudflare Bot Preference Sync Ends robots.txt Confusion
Quick Reads
- Cloudflare launched Bot Preference Sync, a new feature available to all customers from Free to Enterprise.
- It automatically updates your robots.txt to match the AI bot policies you’ve set for Search, Agent, and Training.
- For new publisher and ad-supported sites, Training is set to Disallow by default.
- Bots are tracked via Cloudflare’s BotBase and listed in the public bots directory.
- The feature rolls out to all plans within the coming week.
Cloudflare has introduced Cloudflare Bot Preference Sync, a new tool that automatically keeps your robots.txt file in line with your AI bot configuration. The announcement was made on August 21, 2026, via the Cloudflare Blog.
Until now, site owners faced a frustrating gap. Their robots.txt might tell a crawler to stay away, yet their enforcement rules did not actually block it. That contradiction gave some bots an opening, either to disregard the stated preference or to attempt bypassing the enforcement layer. Bot Preference Sync closes that gap entirely.
The feature works at the zone level. When enabled, Cloudflare reads your AI bot settings for the three main categories, Search, Agent, and Training, and prepends the corresponding directives directly to your robots.txt. So what you tell the world matches what you enforce at the edge.
For Training specifically, Cloudflare is introducing a refined Disallow option. A “no training” preference is written to robots.txt, meaning cooperating mixed-use crawlers that meet Cloudflare’s transparency requirements can still access your content for search indexing. Your search visibility is unaffected, but your content stays out of model training. Cloudflare tracks which bots qualify using BotBase, and the full list of verified Search, Agent, and Training bots is publicly visible in the Cloudflare Radar bots directory.
Transparency is central to how this works. Cloudflare now requires bots that blend both Search and Training activity to meet specific criteria before they receive any benefit of the doubt. Those bots must respect “no training” signals in robots.txt, give site owners a way to opt out of AI summaries, provide URL-level visibility into which pages were made available for training, and publicly demonstrate that disallowing training does not hurt traditional search rankings. Bots that fail to meet those standards are blocked when Training is set to Disallow, no exceptions. Cloudflare tracks compliance in the AI bot transparency section on Cloudflare Radar.
For publishers and ad-supported websites, Cloudflare is introducing a smarter default. During onboarding, those site owners can select “I monetize from pages with ads on this domain.” That choice sets Training to Disallow automatically. They remain in search results while keeping their content away from model training. The setting can be changed at any time.
For non-publishers, no blocks or disallows are added by default. The choice remains entirely with the customer.
Site owners who have more granular policies, such as a private arrangement with a specific bot operator, can simply turn off the sync and manage their robots.txt manually. Bot Preference Sync is designed for category-wide decisions, not one-off custom rules.
Cloudflare Bot Preference Sync is rolling out to every plan, including Free, in the coming week. Existing customers using the legacy managed robots.txt feature will receive a prompt to review and confirm their preferences before transitioning. The feature can be turned on or off at any time from the dashboard.
For web publishers and developers across Africa and Nigeria increasingly navigating AI-driven traffic, this kind of granular, automated control represents a meaningful step toward fairer access and data sovereignty in an AI-first internet era.
Watch your Cloudflare dashboard and check the Cloudflare changelog for the exact availability date.
