How do I unblock AI crawlers on DataDome?
DataDome blocks AI crawlers when the AI-crawler category in its known-bots settings is set to block, or when a crawler isn't recognized and hits the default protection that challenges every unidentified client — and the fix is to allow the search-engine and AI-assistant categories, add a custom allow rule for any crawler DataDome doesn't recognize, and confirm the endpoints you care about aren't in a stricter mode.
Why is DataDome blocking AI crawlers on my site?
DataDome's model is strict by design: traffic is either a recognized good bot, a human that passes its JavaScript and behavioral checks, or it's challenged. A crawler can't run the challenge, so an unrecognized crawler receives a 403 carrying the x-datadome header and a captcha interstitial instead of your page — that header is what SeeGeo's audit detects when it reports a DataDome wall.
DataDome also introduced explicit AI-crawler handling in 2024–2025, separating AI assistants and search crawlers from AI data scrapers, with account-level defaults that often block the whole group. Owners who wanted to stop scraping frequently ended up blocking the assistants too.
Where does the block live in the DataDome dashboard?
Labels are current as of mid-2026 and DataDome renames screens periodically; look for the concepts rather than the exact words.
- Dashboard → Management → Bot settings / Known bots (sometimes "Verified bots"): categories such as Search Engines, Monitoring, AI Assistants / AI Crawlers, and AI Data Scrapers, each with allow or block.
- Dashboard → Management → Custom rules: allow, block or challenge by user-agent, IP, ASN, endpoint or country — the place to add a crawler DataDome doesn't categorize.
- Dashboard → Management → Endpoints (Protection vs Monitor): a page in Protection mode challenges everything unrecognized; Monitor mode only records.
- Your robots.txt, which DataDome doesn't manage — check it separately.
How do I allow AI crawlers on DataDome, step by step?
Changes apply in real time; there is no activation step.
- 1. Known bots: confirm Search Engines is allowed (it usually is) and set the AI Assistants / AI Crawlers category to Allow. Leave AI Data Scrapers on block if scraping was the problem you were solving.
- 2. For any crawler that isn't in a category — check OAI-SearchBot, ChatGPT-User, ClaudeBot, PerplexityBot — add a custom rule: condition user-agent contains the crawler name, action Allow, scope your whole site.
- 3. Endpoints: confirm your homepage and key content pages aren't under a stricter per-endpoint rule than the site default.
- 4. Verify with the crawler-user-agent requests below; a response without the x-datadome header and without a captcha page means the crawler is through.
- 5. Re-run the audit and check the crawler table.
Which AI crawlers should I allow, and which can I keep blocking?
The distinction that matters is search versus training. Search crawlers (OAI-SearchBot, ChatGPT-User, PerplexityBot, ClaudeBot, Googlebot) are what makes an assistant able to find and cite you; blocking them makes you invisible in AI answers. Training crawlers (GPTBot, Google-Extended, CCBot) feed model training and blocking them costs you nothing in visibility today. Most "DataDome blocks AI bots" settings treat both groups as one, which is exactly why owners who only meant to opt out of training end up invisible.
| Crawler | What it feeds | Allow? |
|---|---|---|
| OAI-SearchBot | ChatGPT search answers and citations | Yes — this is the one that recommends you |
| ChatGPT-User | Live fetches when a user asks ChatGPT about a page | Yes |
| GPTBot | OpenAI model training | Your call — no effect on being cited today |
| ClaudeBot / Claude-User | Anthropic's index and live fetches for Claude | Yes |
| PerplexityBot / Perplexity-User | Perplexity answers and citations | Yes |
| Googlebot | Google Search AND AI Overviews / AI Mode | Yes — blocking it removes you from Google entirely |
| Google-Extended | Gemini training (not Search) | Your call |
| Bytespider, CCBot | Third-party scrapers and training sets | Block if you like — no visibility cost |
Every crawler, one page each: what it is and how to control it · The platform toggles that block the wrong crawlers
How do I verify the wall is actually open?
Test from outside, as a crawler would — not from your browser, which is exactly the client the wall was built to admit. Run these from any terminal (or an online HTTP tester) and compare the responses:
A healthy answer is a 200 status with your real HTML. A 403, a challenge page, or a response with the vendor's mitigation header means the crawler is still blocked. If your wall verifies bots by IP range rather than user-agent, a spoofed user-agent from your laptop may still be challenged even though the real crawler gets through — in that case the definitive test is the vendor's own bot analytics, or simply re-running the audit and checking the crawler table.
# As ChatGPT's search crawler:
curl -sI -A "OAI-SearchBot/1.0" https://yoursite.com/ | head -5
# As Claude's crawler:
curl -sI -A "ClaudeBot/1.0" https://yoursite.com/ | head -5
# As a plain browser, for comparison:
curl -sI -A "Mozilla/5.0" https://yoursite.com/ | head -5
# SeeGeo's own crawler, if you want the audit itself to get through:
curl -sI -A "SeeGeoAudit/1.0" https://yoursite.com/ | head -5Frequently asked questions
What is the x-datadome header?
It's the response header DataDome adds when it has intervened on a request — typically alongside a 403 and a captcha or JavaScript challenge. If a crawler-user-agent request to your site returns that header, the crawler was blocked; its absence on a 200 response means the request reached your origin.
Does DataDome distinguish AI assistants from AI scrapers?
Yes, in recent versions of its known-bots settings: AI assistants and search-style crawlers (the ones that cite you) are a separate category from AI data scrapers (training collectors). Allow the first category to stay visible in AI answers; the second is a policy choice with no visibility cost.
Why does a custom allow rule for a user-agent work if scrapers can fake user-agents?
It's a trade-off. A user-agent allow rule admits anything claiming to be that crawler, so scope it narrowly and prefer DataDome's own verified-bot categories where they exist — they check the crawler's published IP ranges, which a spoofer can't fake. Use the custom rule only for crawlers DataDome hasn't categorized yet.