AI crawlers

What is Claude-User?

Claude-User is Anthropic's on-demand AI fetcher. Fetches your page live during a Claude conversation.

What does Claude-User do with your content?

Fetches your page live during a Claude conversation. Like most AI crawlers, Claude-User reads your pages as plain HTML — it does not run JavaScript — so content that only appears after scripts execute is invisible to it.

Should you block Claude-User?

Blocking Claude-User makes your business invisible to Anthropic's AI when customers ask it for recommendations. If you want to be found, allow it; block it only if you have a specific reason to keep Anthropic's AI away from your content.

How do you allow or block Claude-User in robots.txt?

Add one of these snippets to the robots.txt file at the root of your site. The first explicitly allows Claude-User everywhere; the second blocks it completely.

# Allow Claude-User
User-agent: Claude-User
Allow: /

# Block Claude-User
User-agent: Claude-User
Disallow: /

Is robots.txt enough?

Not always. CDNs and firewalls (Cloudflare in particular) can block crawlers at the network level regardless of what robots.txt says — Cloudflare blocks AI crawlers by default for many accounts. If you want Claude-User to reach your site, check your CDN's bot settings too. SeeGeo's free audit checks both.

Frequently asked questions

What is Claude-User?

Claude-User is Anthropic's on-demand AI fetcher. Fetches your page live during a Claude conversation.

Should I block Claude-User?

Blocking Claude-User makes your business invisible to Anthropic's AI when customers ask it for recommendations. If you want to be found, allow it; block it only if you have a specific reason to keep Anthropic's AI away from your content..

Does Claude-User run JavaScript?

No. Claude-User, like most AI crawlers, reads raw HTML without executing JavaScript. If your content only renders client-side, it is effectively invisible to it.

Is your site readable to AI right now? SeeGeo's free audit checks crawler access (all 16 bots), structure, and content extractability in about a minute.

Run a free audit

Related