Is your CDN or firewall blocking AI crawlers? How to check

It might be. Your robots.txt file is only one of the places a crawler can be turned away. A content delivery network (CDN) serves your site from servers close to the visitor. A web application firewall (WAF) decides which requests get through. Both sit in front of your website and can block a bot before it reads a single page, whatever your robots.txt says.

This deserves a fresh look because Cloudflare changed its controls twice in 2026. It replaced its single "Block AI bots" switch with three separate policies on July 1, 2026, then changed what its Block option does on September 15, 2026. One of the current options stops Google and Bing along with AI training crawlers, so check what is switched on today.

Why a clean robots.txt is not enough

A robots.txt rule is a request a crawler chooses to follow. A firewall rule is enforced before the crawler gets a response. Our guide to AI crawlers and robots.txt covers the first layer and lists every crawler token; this article covers the second.

The AI vendors point at this layer themselves. OpenAI's help center tells site owners to allow OAI-SearchBot and to "confirm that the website host or content delivery network allows traffic from OpenAI's published searchbot IP addresses." Google's best practices for AI features include making sure crawling is allowed "in robots.txt, and by any CDN or hosting infrastructure." Perplexity's bot documentation says that behind a WAF "you may need to explicitly whitelist Perplexity's bots to ensure they can access your content."

Cloudflare's AI bot controls as of October 2026

Cloudflare now sorts bots by what they do on your site. Its changelog for July 1, 2026 says all customers can now manage AI crawlers by behavior "instead of a single Block AI bots toggle." Cloudflare says the controls are available on all plans, under Security Settings, in a section its documentation calls "Configure AI bot policies."

Each policy takes one of the options below. The wording is from Cloudflare's September 15, 2026 announcement.

Two caveats. First, we read these names in Cloudflare's documentation and blog, not in a live dashboard, and Cloudflare's pages disagree. The September 15 post says the legacy "Block AI Bots" setting and Managed Robots.txt "will be deprecated" in favor of the three policies and Bot Preference Sync. The "Block AI bots" documentation page, last updated July 1, 2026, lists three options, treats September 15 as a future date and says the legacy option will also block mixed-purpose crawlers; the post says legacy settings migrate to Disallow AI Training. The robots.txt setting page, updated October 8, 2026, calls the Training option "Disallow." Confirm labels and values in your own dashboard. Second, Cloudflare says Disallow AI Training does not yet convey a no-training preference to Bing through robots.txt.

The option that also blocks Google and Bing

Cloudflare defines a mixed-use crawler as "a single crawler doing both Search and Training," and names Applebot, Bingbot and Googlebot as examples. Before September 15, 2026, the old block excluded those crawlers. Now, in Cloudflare's words, "Block and Block on pages with ads now apply to mixed-use crawlers, including Applebot, Bingbot, and Googlebot, so either setting impacts search as well as training. To stop training and keep search, use Disallow AI Training."

Selecting Block, Cloudflare writes, "will stop Applebot, Bingbot, and Googlebot from reaching your site," and it adds: "search included." For a business that depends on search traffic, Block is the wrong way to opt out of training.

What happened to sites that used the old toggle

Cloudflare says existing preferences were migrated. Under the heading "What you need to do," its answer is: "Nothing, in almost every case." For domains that had only ever used the legacy "Block AI bots" setting, it published this mapping.

Domains that had already set the three policies kept their Search and Agent values. For Training, Cloudflare says: "Previous Training selections of Block or Block on pages with ads will migrate to Disallow AI Training."

Look at the Agent column. Cloudflare's July 1, 2026 post gives ChatGPT-User as an example of an Agent, and OpenAI says ChatGPT may use that agent to visit a page when a user asks a question. What follows is our reading of those statements, not something Cloudflare documents: if your site shows ads and had the old toggle on, the migrated setting may turn that visit away on pages Cloudflare detects as serving an ad.

Other Cloudflare features that can stop a bot

• Bot Fight Mode. This free feature issues "computationally expensive challenges" to traffic it identifies as bots. Cloudflare states: "You cannot bypass or skip Bot Fight Mode using WAF custom rules or Page Rules." The one documented exception is an IP Access rule that matches the request first. Super Bot Fight Mode, on the Pro, Business and Enterprise plans, has a separate setting for verified bots and can be skipped with a custom rule. Cloudflare does not say whether Bot Fight Mode challenges any given AI crawler.
• AI Crawl Control. This lists AI crawlers in a "Crawlers" tab with a Block action for each. Allow there is not an override: "allowed bots may still be affected by other security rules."
• Managed robots.txt. With Bot Preference Sync on, Cloudflare writes directives for you. If you already have a file, it "will prepend our managed robots.txt before your existing robots.txt, combining both into a single response." The file crawlers receive may not be the file you uploaded.

Other hosts and firewalls

What each platform's documentation says as of October 2026:

Squarespace explains its default this way: "we don't want to potentially impact your site's traffic by excluding it from chat answers and sources." Shopify adds that blocking crawlers "affects only open-web discoverability" and does not stop product data from reaching the agentic storefronts a store has activated, as our Shopify GEO guide explains.

This table is not a complete list. If your host or CDN is not in it (Akamai, Fastly, Netlify and WP Engine are common examples), read its bot management documentation and send its support team the questions in the last FAQ below.

How to test your own site

1. Read the robots.txt that is served. Open yourdomain.com/robots.txt in a browser and look for rules you did not write. If the file is new to you, start with our robots.txt guide.
2. Compare a browser request with a bot request. Run the command below with your own address in place of www.example.com. It prints the status code a request with that crawler's user-agent string receives. Run it again with a browser's user-agent string. If the browser string returns 200 and the crawler string returns 403, something is probably reacting to the user agent.
3. Read the platform's logs. Cloudflare's AI Crawl Control "Crawlers" tab shows requests per crawler. For Google, use the URL Inspection tool in Search Console.
4. Confirm who the visitor was. A line in a log that says GPTBot is not proof it came from OpenAI. Our guide to verifying AI bots in server logs explains how to check.

curl -s -o /dev/null -w "%{http_code}\n" -A "Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; GPTBot/1.4; +https://openai.com/gptbot" https://www.example.com/

What the command-line test cannot tell you

• You are not the real bot. Cloudflare, Vercel, AWS and Perplexity all describe verification by IP address. A request from your laptop that claims to be GPTBot can be blocked while the real crawler is allowed, or the reverse.
• A challenge is not a block. Bot Fight Mode issues a challenge, and an AI Crawl Control block returns "403 Forbidden" or "402 Payment Required." The status code alone does not tell you which feature acted.
• A 200 says nothing about robots.txt. The page may load while the served robots.txt still tells the crawler to stay out.

How to let the search crawlers in

1. In Cloudflare, confirm the Search policy is set to Allow. If you want to opt out of training, choose Disallow AI Training, not Block.
2. Review WAF custom rules, IP rules and rate limits for anything that matches a crawler you want.
3. If a wanted crawler is still blocked, add an allow rule the way Perplexity documents it for Cloudflare: match the user agent and the vendor's published IP ranges together, with the action Allow.
4. Read the logs again a day later. A setting is confirmed when the real crawler receives a 200.

Repeat the check after any change of host, CDN or security plugin, and put it on the launch list for every redesign or migration. Whether a crawler can read what it fetches is covered in our guide to JavaScript rendering and AI crawlers.

Conclusion

Whether AI search crawlers can reach your site is decided in at least two places: the robots.txt file and whatever sits in front of your server. Cloudflare's Block option for Training now reaches Googlebot and Bingbot, so a setting chosen to stop AI training can cost you search visibility. Read your dashboard, the file your site serves and your logs. Letting a crawler in makes your pages eligible to be read; it does not earn a citation. This check is one line in a full technical audit for GEO, and for a second pair of eyes on your setup, request a free GEO audit. More guides are in our GEO and AEO Guide.

Frequently asked questions