How do I unblock AI crawlers on Akamai?

Why is Akamai blocking AI crawlers on my site?

Akamai Bot Manager sorts traffic into Akamai-categorized bots (search engines, monitoring tools, and since 2024–2025 AI and large-language-model crawlers), customer-defined bots, and everything else it detects as automated. Each category carries an action — Allow, Monitor, Deny, or a challenge — and enterprise defaults lean toward Deny for anything not explicitly trusted. A crawler that arrives outside a trusted category gets a 403 with an Akamai reference number instead of your page; that reference page is what SeeGeo's audit received when it reported an Akamai wall.

Akamai also offers Content Protector and App & API Protector rules that can match crawler user-agents or rate-limit polite crawls. Because Akamai changes ship through a staged activation, a block that was added months ago can outlive whoever added it.

Where does the block live in Akamai Control Center?

Names below are current as of mid-2026; your account's security configuration may label the screens slightly differently.

• Security Configuration → Bot Manager → Akamai-Categorized Bots: the per-category action table. Look for "Search Engine Bots" and the AI / LLM crawler category (Akamai has named it "AI Bots" or "Large Language Model Crawlers" across releases).
• Bot Manager → Bot Detection (or "Unknown bots"): the default action for automated traffic Akamai can't categorize — a Deny here catches newer crawlers before they're categorized.
• Bot Manager → Custom Bot Categories / Customer-Defined Bots: where you define a crawler by user-agent string when Akamai hasn't categorized it yet.
• App & API Protector / Kona Site Defender → Custom rules: user-agent matches, rate controls and Client Reputation thresholds that can catch crawlers indiscriminately.

How do I allow AI crawlers on Akamai, step by step?

Every change below lands only after you activate the configuration — save, activate to staging, test, then activate to production.

• 1. Bot Manager → Akamai-Categorized Bots: set "Search Engine Bots" to Allow (Googlebot, Bingbot) and the AI / LLM crawler category to Monitor or Allow. Monitor lets you watch the traffic for a week before committing to Allow.
• 2. For crawlers missing from Akamai's categories — check for OAI-SearchBot, ChatGPT-User, ClaudeBot, PerplexityBot — create a customer-defined bot per user-agent under Custom Bot Categories, and give the category the Allow or Monitor action.
• 3. Bot Detection: confirm the action for unknown automated traffic isn't silently denying the crawlers above; if it is, either raise the AI category above it or set the detection to Monitor for those user-agents.
• 4. App & API Protector custom rules: search rule conditions for "bot", "crawler", "GPT", "Claude" and remove or exempt the AI crawlers; check rate controls aren't tighter than a polite one-request-per-second crawl.
• 5. Save → activate to staging → verify from staging with the crawler-user-agent requests below → activate to production.
• 6. Re-run the audit after production activation and confirm the crawler table shows the AI crawlers allowed.

Which AI crawlers should I allow, and which can I keep blocking?

The distinction that matters is search versus training. Search crawlers (OAI-SearchBot, ChatGPT-User, PerplexityBot, ClaudeBot, Googlebot) are what makes an assistant able to find and cite you; blocking them makes you invisible in AI answers. Training crawlers (GPTBot, Google-Extended, CCBot) feed model training and blocking them costs you nothing in visibility today. Most "Akamai blocks AI bots" settings treat both groups as one, which is exactly why owners who only meant to opt out of training end up invisible.

Every crawler, one page each: what it is and how to control it · The platform toggles that block the wrong crawlers

How do I verify the wall is actually open?

Test from outside, as a crawler would — not from your browser, which is exactly the client the wall was built to admit. Run these from any terminal (or an online HTTP tester) and compare the responses:

A healthy answer is a 200 status with your real HTML. A 403, a challenge page, or a response with the vendor's mitigation header means the crawler is still blocked. If your wall verifies bots by IP range rather than user-agent, a spoofed user-agent from your laptop may still be challenged even though the real crawler gets through — in that case the definitive test is the vendor's own bot analytics, or simply re-running the audit and checking the crawler table.

# As ChatGPT's search crawler:
curl -sI -A "OAI-SearchBot/1.0" https://yoursite.com/ | head -5
# As Claude's crawler:
curl -sI -A "ClaudeBot/1.0" https://yoursite.com/ | head -5
# As a plain browser, for comparison:
curl -sI -A "Mozilla/5.0" https://yoursite.com/ | head -5
# SeeGeo's own crawler, if you want the audit itself to get through:
curl -sI -A "SeeGeoAudit/1.0" https://yoursite.com/ | head -5

Re-run the free audit — the crawler table is the receipt

Does Akamai have an AI crawler category, or do I define each bot myself?

Both, depending on your account's release: Akamai added AI and LLM crawler categories to Bot Manager's categorized bots in 2024–2025, so most accounts can set one category action. Newer crawlers can lag categorization, in which case a customer-defined bot by user-agent covers them until Akamai catches up.

Why does my Akamai site return a reference number page to crawlers?

That page ("Access Denied — You don't have permission… Reference #18.xxx") is Akamai's Deny action. The reference number is the incident ID your Akamai support team can look up to tell you which rule fired — the fastest way to find the block if the category table looks fine.

How long until an Akamai change takes effect?

Akamai security configurations must be activated; staging activation is typically a few minutes and production a few more. Changes saved but not activated do nothing, which is the most common reason a fix 'didn't work' — check the activation status before assuming the setting was wrong.