What is ClaudeBot?
ClaudeBot is Anthropic's AI training crawler. Crawls pages for Claude's models and search index.
What does ClaudeBot do with your content?
Crawls pages for Claude's models and search index. Like most AI crawlers, ClaudeBot reads your pages as plain HTML — it does not run JavaScript — so content that only appears after scripts execute is invisible to it.
Should you block ClaudeBot?
This is a genuine trade-off. Allowing ClaudeBot lets future Anthropic models learn your business exists — useful when customers ask those models for recommendations. Blocking it keeps your content out of training data at the cost of that future visibility. Neither answer is wrong; it depends on what you value more.
How do you allow or block ClaudeBot in robots.txt?
Add one of these snippets to the robots.txt file at the root of your site. The first explicitly allows ClaudeBot everywhere; the second blocks it completely.
# Allow ClaudeBot
User-agent: ClaudeBot
Allow: /
# Block ClaudeBot
User-agent: ClaudeBot
Disallow: /Is robots.txt enough?
Not always. CDNs and firewalls (Cloudflare in particular) can block crawlers at the network level regardless of what robots.txt says — Cloudflare blocks AI crawlers by default for many accounts. If you want ClaudeBot to reach your site, check your CDN's bot settings too. SeeGeo's free audit checks both.
How do I check whether ClaudeBot can read my site right now?
Two checks, both needed: read your robots.txt for a group naming ClaudeBot (or the * group it falls back to), then request your homepage with ClaudeBot's user agent and confirm you get a 200 with your real HTML rather than a 403 or a challenge page. SeeGeo's free crawler check runs both in a few seconds.
How does ClaudeBot compare to similar crawlers?
ClaudeBot is one of 16 crawlers the SeeGeo audit checks individually. The closest comparisons:
| Crawler | Operator | Role | Runs JavaScript? |
|---|---|---|---|
| ClaudeBot | Anthropic | AI training crawler | No |
| Claude-User | Anthropic | on-demand AI fetcher | No |
| GPTBot | OpenAI | AI training crawler | No |
| Google-Extended | AI training crawler | No | |
| CCBot | Common Crawl | AI training crawler | No |
Why does crawler access matter? The numbers
Crawler access is where AI visibility starts or ends: a blocked crawler can't read you, and an AI that can't read you can't recommend you.
- Visitors arriving from AI assistants convert at roughly 4.4x the rate of traditional organic search on average (Semrush, cross-industry, 2026).
- AI-assistant referrals are still only about 1% of total web traffic — small, but the fastest-growing acquisition channel measured (multiple 2026 studies).
- Adobe Digital Insights (Q1 2026) measured AI-assistant visitors converting 42% better than non-AI traffic — a full reversal from the year before.
- Cloudflare blocks AI crawlers by default for many accounts, so sites are often invisible to AI without anyone having decided to be.
- Content edits alone — statistics, citations, quotable structure — can raise AI visibility on the order of 30–40% (Princeton GEO study, KDD 2024).
Frequently asked questions
What is ClaudeBot?
ClaudeBot is Anthropic's AI training crawler. Crawls pages for Claude's models and search index.
Should I block ClaudeBot?
This is a genuine trade-off. Allowing ClaudeBot lets future Anthropic models learn your business exists — useful when customers ask those models for recommendations.
Does ClaudeBot run JavaScript?
No. ClaudeBot, like most AI crawlers, reads raw HTML without executing JavaScript. If your content only renders client-side, it is effectively invisible to it.
Can ClaudeBot read my website?
Only if two things are true: your robots.txt does not disallow ClaudeBot, and your server or CDN actually serves the page when ClaudeBot asks. Firewalls can block it regardless of robots.txt, so the reliable way to know is to fetch your homepage as ClaudeBot — the free check on see-geo.com/ai-crawler-check does exactly that.