--- title: "What is Claude-SearchBot?" description: "What Claude-SearchBot (Anthropic) does with your content, whether to block it, and the exact robots.txt lines — in plain language." canonical: https://see-geo.com/bots/claude-searchbot language: en published: 2026-09-29 updated: 2026-09-29 publisher: SeeGeo alternates: fr: https://see-geo.com/fr/bots/claude-searchbot de: https://see-geo.com/de/bots/claude-searchbot --- # What is Claude-SearchBot? Claude-SearchBot is Anthropic's AI search index crawler. Crawls the web to improve Claude's search results — blocking it can keep you out of what Claude finds when it searches. ## What does Claude-SearchBot do with your content? Crawls the web to improve Claude's search results — blocking it can keep you out of what Claude finds when it searches. Like most AI crawlers, Claude-SearchBot reads your pages as plain HTML — it does not run JavaScript — so content that only appears after scripts execute is invisible to it. ## Should you block Claude-SearchBot? Blocking Claude-SearchBot can keep your pages out of the results Claude finds when it searches the web to answer a question — the closest thing Claude has to being found in search. It does not affect training (that is ClaudeBot) or pages a person asks Claude to open (Claude-User). If you want Claude to recommend you, leave it allowed. ## How do you allow or block Claude-SearchBot in robots.txt? Add one of these snippets to the robots.txt file at the root of your site. The first explicitly allows Claude-SearchBot everywhere; the second blocks it completely. ``` # Allow Claude-SearchBot User-agent: Claude-SearchBot Allow: / # Block Claude-SearchBot User-agent: Claude-SearchBot Disallow: / ``` ## How do you recognise Claude-SearchBot in your server logs? Anthropic publishes the Claude-SearchBot token but not a full user-agent string, so search your access logs for "Claude-SearchBot". Anyone can copy a user-agent string, so check the request's IP address against the list Anthropic publishes for Claude-SearchBot. [Anthropic's documentation for Claude-SearchBot](https://support.claude.com/en/articles/8896518-does-anthropic-crawl-data-from-the-web-and-how-can-site-owners-block-the-crawler) · [Anthropic's published IP ranges](https://claude.com/crawling/bots.json) ## Is robots.txt enough? Not always. CDNs and firewalls (Cloudflare in particular) can block crawlers at the network level regardless of what robots.txt says — Cloudflare blocks AI crawlers by default for many accounts. If you want Claude-SearchBot to reach your site, check your CDN's bot settings too. SeeGeo's free audit checks both. ## How do I check whether Claude-SearchBot can read my site right now? Two checks, both needed: read your robots.txt for a group naming Claude-SearchBot (or the * group it falls back to), then request your homepage with Claude-SearchBot's user agent and confirm you get a 200 with your real HTML rather than a 403 or a challenge page. SeeGeo's free crawler check runs both in a few seconds. [Run the free AI crawler check](https://see-geo.com/ai-crawler-check) ## How does Claude-SearchBot compare to similar crawlers? Claude-SearchBot is one of 17 crawlers the SeeGeo audit checks individually. The closest comparisons: | Crawler | Operator | Role | Runs JavaScript? | |---|---|---|---| | Claude-SearchBot | Anthropic | AI search index crawler | No | | ClaudeBot | Anthropic | AI training crawler | No | | Claude-User | Anthropic | on-demand AI fetcher | No | | OAI-SearchBot | OpenAI | AI search index crawler | No | | PerplexityBot | Perplexity | AI search index crawler | No | [All 17 crawlers, compared](https://see-geo.com/bots) · [What is an AI crawler?](https://see-geo.com/glossary/ai-crawler) ## Why does crawler access matter? The numbers Crawler access is where AI visibility starts or ends: a blocked crawler can't read you, and an AI that can't read you can't recommend you. - Visitors arriving from AI assistants convert at roughly 4.4x the rate of traditional organic search on average (Semrush, cross-industry, 2026). - AI-assistant referrals are still only about 1% of total web traffic — small, but the fastest-growing acquisition channel measured (multiple 2026 studies). - Adobe Digital Insights (Q1 2026) measured AI-assistant visitors converting 42% better than non-AI traffic — a full reversal from the year before. - Cloudflare blocks AI crawlers by default for many accounts, so sites are often invisible to AI without anyone having decided to be. - Content edits alone — statistics, citations, quotable structure — can raise AI visibility on the order of 30–40% (Princeton GEO study, KDD 2024). ## Frequently asked questions ### What is Claude-SearchBot? Claude-SearchBot is Anthropic's AI search index crawler. Crawls the web to improve Claude's search results — blocking it can keep you out of what Claude finds when it searches. ### Should I block Claude-SearchBot? Blocking Claude-SearchBot can keep your pages out of the results Claude finds when it searches the web to answer a question — the closest thing Claude has to being found in search. It does not affect training (that is ClaudeBot) or pages a person asks Claude to open (Claude-User). ### Does Claude-SearchBot run JavaScript? No. Claude-SearchBot, like most AI crawlers, reads raw HTML without executing JavaScript. If your content only renders client-side, it is effectively invisible to it. ### Can Claude-SearchBot read my website? Only if two things are true: your robots.txt does not disallow Claude-SearchBot, and your server or CDN actually serves the page when Claude-SearchBot asks. Firewalls can block it regardless of robots.txt, so the reliable way to know is to fetch your homepage as Claude-SearchBot — the free check on see-geo.com/ai-crawler-check does exactly that. ### What is Claude-SearchBot's user agent? Anthropic publishes the Claude-SearchBot token but not a full user-agent string, so search your access logs for "Claude-SearchBot". Anyone can copy a user-agent string, so check the request's IP address against the list Anthropic publishes for Claude-SearchBot. ## Related pages - [What is ClaudeBot?](https://see-geo.com/bots/claudebot) - [What is Claude-User?](https://see-geo.com/bots/claude-user) - [What is OAI-SearchBot?](https://see-geo.com/bots/oai-searchbot)