--- title: "How do I unblock AI crawlers on Cloudflare?" description: "Cloudflare's 'Block AI bots' toggle and Super Bot Fight Mode hide sites from ChatGPT, Claude and Perplexity. The settings to change, and how to verify." canonical: https://see-geo.com/unblock/cloudflare language: en published: 2026-09-02 updated: 2026-09-02 publisher: SeeGeo --- # How do I unblock AI crawlers on Cloudflare? On Cloudflare, AI crawlers are usually blocked by one of three things: the one-click "Block AI bots" toggle, Super Bot Fight Mode challenging "definitely automated" traffic, or a WAF custom rule — and the fix is to allow verified search and AI crawlers in AI Crawl Control while leaving training crawlers to your own policy. Ten minutes in the dashboard, then verify with a crawler-user-agent request. ## Why is Cloudflare blocking AI crawlers on my site? Cloudflare made blocking AI crawlers a single switch, available on every plan including free, and many owners flipped it to stop model training without realizing it also blocks the crawlers that answer customer questions. Separately, Bot Fight Mode and Super Bot Fight Mode challenge any client Cloudflare classifies as automated — AI crawlers included — with a JavaScript challenge that no crawler can solve. The challenge page is what an AI assistant receives instead of your homepage, and it is also what SeeGeo's audit received when it graded the site as walled. Since 2025 Cloudflare has also shipped a managed robots.txt with Content Signals (search / ai-input / ai-train directives) and an AI Crawl Control panel with per-crawler allow and block decisions. Those are the right tools for a deliberate policy; the problem is defaults that were never a decision. ## Where does the block live in the Cloudflare dashboard? Check these in order; the first one you find is usually the whole story. Menu names are current as of mid-2026 and shift occasionally. - Security → Bots → "Block AI bots" — the one-click toggle. On means every known AI crawler, search ones included, is blocked at the edge. - Security → Bots → Super Bot Fight Mode (or Bot Fight Mode on free plans) → "Definitely automated" set to Block or Managed Challenge, and "Allow verified bots" turned off. - Security → AI Crawl Control — the per-crawler table (OAI-SearchBot, GPTBot, ClaudeBot, PerplexityBot, Google-Extended…) with Allow / Block per row. - Security → WAF → Custom rules — a rule matching user-agents ("bot", "crawler", "GPT") or ASNs, or a rate limit tight enough to catch a polite crawl. - Your robots.txt — Cloudflare's managed robots.txt can add Content Signals and a block on AI training crawlers; check what it actually says at yoursite.com/robots.txt. ## How do I allow AI crawlers on Cloudflare, step by step? Do these top to bottom; each is reversible and takes effect within a minute. - 1. Security → Bots: turn "Block AI bots" OFF. If you want to keep blocking training crawlers, do that per-crawler in the next step instead of with this blanket switch. - 2. Security → AI Crawl Control: set OAI-SearchBot, ChatGPT-User, ClaudeBot, Claude-User, PerplexityBot and Googlebot to Allow. Decide GPTBot and Google-Extended (training) on their own merits — allowing them is not required to be cited. - 3. Security → Bots → Super Bot Fight Mode: turn "Allow verified bots" ON. OpenAI's, Anthropic's, Perplexity's and Google's crawlers are on Cloudflare's verified list, so this admits them while still challenging unverified automation. - 4. If "Definitely automated" is set to Block, either change it to Allow or add a WAF custom rule that Skips bot protection for verified bots — expression: (cf.verified_bot_category in {"Search Engine Crawler" "AI Crawler"}) with action Skip → Super Bot Fight Mode. - 5. Security → WAF → Custom rules: read every rule that mentions user-agent, "bot" or "crawler" and add an exception for the crawlers above, or delete the rule if it was a default you never chose. - 6. Open yoursite.com/robots.txt. If it contains Content-Signal lines or Disallow rules for search crawlers you want, edit them under Security → Bots → Managed robots.txt (or your origin's robots.txt if managed mode is off). Keep ai-train=no if that's your policy; make sure search=yes. - 7. Verify with the crawler-user-agent requests below, then re-run the audit. ## Which AI crawlers should I allow, and which can I keep blocking? The distinction that matters is search versus training. Search crawlers (OAI-SearchBot, ChatGPT-User, PerplexityBot, ClaudeBot, Googlebot) are what makes an assistant able to find and cite you; blocking them makes you invisible in AI answers. Training crawlers (GPTBot, Google-Extended, CCBot) feed model training and blocking them costs you nothing in visibility today. Most "Cloudflare blocks AI bots" settings treat both groups as one, which is exactly why owners who only meant to opt out of training end up invisible. | Crawler | What it feeds | Allow? | |---|---|---| | OAI-SearchBot | ChatGPT search answers and citations | Yes — this is the one that recommends you | | ChatGPT-User | Live fetches when a user asks ChatGPT about a page | Yes | | GPTBot | OpenAI model training | Your call — no effect on being cited today | | ClaudeBot / Claude-User | Anthropic's index and live fetches for Claude | Yes | | PerplexityBot / Perplexity-User | Perplexity answers and citations | Yes | | Googlebot | Google Search AND AI Overviews / AI Mode | Yes — blocking it removes you from Google entirely | | Google-Extended | Gemini training (not Search) | Your call | | Bytespider, CCBot | Third-party scrapers and training sets | Block if you like — no visibility cost | [Every crawler, one page each: what it is and how to control it](https://see-geo.com/bots) · [The platform toggles that block the wrong crawlers](https://see-geo.com/blog/website-platform-ai-visibility-defaults) ## How do I verify the wall is actually open? Test from outside, as a crawler would — not from your browser, which is exactly the client the wall was built to admit. Run these from any terminal (or an online HTTP tester) and compare the responses: A healthy answer is a 200 status with your real HTML. A 403, a challenge page, or a response with the vendor's mitigation header means the crawler is still blocked. If your wall verifies bots by IP range rather than user-agent, a spoofed user-agent from your laptop may still be challenged even though the real crawler gets through — in that case the definitive test is the vendor's own bot analytics, or simply re-running the audit and checking the crawler table. ``` # As ChatGPT's search crawler: curl -sI -A "OAI-SearchBot/1.0" https://yoursite.com/ | head -5 # As Claude's crawler: curl -sI -A "ClaudeBot/1.0" https://yoursite.com/ | head -5 # As a plain browser, for comparison: curl -sI -A "Mozilla/5.0" https://yoursite.com/ | head -5 # SeeGeo's own crawler, if you want the audit itself to get through: curl -sI -A "SeeGeoAudit/1.0" https://yoursite.com/ | head -5 ``` [Re-run the free audit — the crawler table is the receipt](https://see-geo.com/) ## Frequently asked questions ### Does Cloudflare's 'Block AI bots' toggle block ChatGPT search too? Yes. The one-click toggle blocks the known AI crawlers as a group, and that group includes OAI-SearchBot and ChatGPT-User — the crawlers that make ChatGPT able to cite you — not just GPTBot, the training crawler. To block training while staying citable, turn the toggle off and use AI Crawl Control to decide per crawler. ### What does the cf-mitigated: challenge header mean? It's Cloudflare telling the client it was served a challenge page instead of your site. When a crawler's request comes back with that header, the crawler saw a JavaScript puzzle, not your content — which for an AI assistant is the same as the page being blank. It's also the signal SeeGeo's audit uses to report a Cloudflare wall. ### Will allowing verified bots let scrapers in? No. Cloudflare's verified-bot list is a vetted registry of crawlers that prove their identity through published IP ranges or reverse DNS; scrapers spoofing a Googlebot user-agent don't qualify and stay challenged. Allowing verified bots is the narrowest change that admits real search and AI crawlers. ### Do I need to allow GPTBot to show up in ChatGPT? No. GPTBot collects training data; OAI-SearchBot and ChatGPT-User are what ChatGPT's search and browsing use to find and cite pages. You can block GPTBot and still be cited, as long as the search crawlers are allowed. ## Related pages - [How do I unblock AI crawlers on Akamai?](https://see-geo.com/unblock/akamai) - [How do I unblock AI crawlers on DataDome?](https://see-geo.com/unblock/datadome) - [How do I unblock AI crawlers on Imperva?](https://see-geo.com/unblock/imperva)