What is ClaudeBot?

What does ClaudeBot do with your content?

Collects public web pages that Anthropic may use to train Claude. Claude's search results come from a separate agent, Claude-SearchBot. Like most AI crawlers, ClaudeBot reads your pages as plain HTML — it does not run JavaScript — so content that only appears after scripts execute is invisible to it.

Should you block ClaudeBot?

Blocking ClaudeBot keeps your content out of data Anthropic may use to train Claude. Anthropic runs two other agents with separate robots.txt rules: Claude-User, which fetches pages when someone asks Claude a question, and Claude-SearchBot, which improves Claude's search results. Blocking ClaudeBot alone leaves both of those untouched.

How do you allow or block ClaudeBot in robots.txt?

Add one of these snippets to the robots.txt file at the root of your site. The first explicitly allows ClaudeBot everywhere; the second blocks it completely.

# Allow ClaudeBot
User-agent: ClaudeBot
Allow: /

# Block ClaudeBot
User-agent: ClaudeBot
Disallow: /

How do you recognise ClaudeBot in your server logs?

Anthropic publishes the ClaudeBot token but not a full user-agent string, so search your access logs for "ClaudeBot".

Anyone can copy a user-agent string, so check the request's IP address against the list Anthropic publishes for ClaudeBot.

Anthropic's documentation for ClaudeBot · Anthropic's published IP ranges

Is robots.txt enough?

Not always. CDNs and firewalls (Cloudflare in particular) can block crawlers at the network level regardless of what robots.txt says — Cloudflare blocks AI crawlers by default for many accounts. If you want ClaudeBot to reach your site, check your CDN's bot settings too. SeeGeo's free audit checks both.

How do I check whether ClaudeBot can read my site right now?

Two checks, both needed: read your robots.txt for a group naming ClaudeBot (or the * group it falls back to), then request your homepage with ClaudeBot's user agent and confirm you get a 200 with your real HTML rather than a 403 or a challenge page. SeeGeo's free crawler check runs both in a few seconds.

Run the free AI crawler check

How does ClaudeBot compare to similar crawlers?

ClaudeBot is one of 17 crawlers the SeeGeo audit checks individually. The closest comparisons:

All 17 crawlers, compared · What is an AI crawler?

Why does crawler access matter? The numbers

Crawler access is where AI visibility starts or ends: a blocked crawler can't read you, and an AI that can't read you can't recommend you.

• Visitors arriving from AI assistants convert at roughly 4.4x the rate of traditional organic search on average (Semrush, cross-industry, 2026).
• AI-assistant referrals are still only about 1% of total web traffic — small, but the fastest-growing acquisition channel measured (multiple 2026 studies).
• Adobe Digital Insights (Q1 2026) measured AI-assistant visitors converting 42% better than non-AI traffic — a full reversal from the year before.
• Cloudflare blocks AI crawlers by default for many accounts, so sites are often invisible to AI without anyone having decided to be.
• Content edits alone — statistics, citations, quotable structure — can raise AI visibility on the order of 30–40% (Princeton GEO study, KDD 2024).

What is ClaudeBot?

ClaudeBot is Anthropic's AI training crawler. Collects public web pages that Anthropic may use to train Claude. Claude's search results come from a separate agent, Claude-SearchBot.

Should I block ClaudeBot?

Blocking ClaudeBot keeps your content out of data Anthropic may use to train Claude. Anthropic runs two other agents with separate robots.txt rules: Claude-User, which fetches pages when someone asks Claude a question, and Claude-SearchBot, which improves Claude's search results.

Does ClaudeBot run JavaScript?

No. ClaudeBot, like most AI crawlers, reads raw HTML without executing JavaScript. If your content only renders client-side, it is effectively invisible to it.

Can ClaudeBot read my website?

Only if two things are true: your robots.txt does not disallow ClaudeBot, and your server or CDN actually serves the page when ClaudeBot asks. Firewalls can block it regardless of robots.txt, so the reliable way to know is to fetch your homepage as ClaudeBot — the free check on see-geo.com/ai-crawler-check does exactly that.

What is ClaudeBot's user agent?

Anthropic publishes the ClaudeBot token but not a full user-agent string, so search your access logs for "ClaudeBot". Anyone can copy a user-agent string, so check the request's IP address against the list Anthropic publishes for ClaudeBot.