Short answer: for most small businesses, no — blocking AI crawlers removes you from the AI answers your customers increasingly rely on, and the two most common blocks we see in audits are misunderstandings, not decisions.
The longer answer depends on which bot, because "AI crawler" covers three very different jobs.
What are the three kinds of AI crawler?
Training crawlers (GPTBot, ClaudeBot, Meta-ExternalAgent, CCBot) collect pages so future models know your business exists. Blocking them is a real trade-off: your content stays out of training data, and future models learn about your competitors instead of you.
Search-index crawlers (OAI-SearchBot, PerplexityBot) build the live indexes behind AI search. Block these and the AI cannot cite you even when you're the best answer.
On-demand fetchers (ChatGPT-User, Claude-User, Perplexity-User) fetch your page live when a user asks about you mid-conversation. Blocking them means the AI answers questions about your business without being able to check your site.
Which blocks are usually mistakes?
Two patterns come up constantly in SeeGeo audits (robots.txt rules per RFC 9309):
The blanket block. A User-agent: * / Disallow: / left over from a staging site, or added "for security." It doesn't just block AI — it blocks Googlebot. Everything else on the site stops mattering until it's removed.
The Google-Extended confusion. Blocking Google-Extended does not remove you from Google Search or AI Overviews — those run on Googlebot. It only opts you out of Gemini model training. The reverse mistake is worse: some owners block Bingbot "because we don't care about Bing," not realizing AI assistants that draw on Bing's index — including parts of ChatGPT's search — lose access too.
When is blocking the right call?
Legitimate reasons exist. Publishers whose content is the product may rationally block training crawlers. Sites with user data behind logins should block everything from private paths. And any business can decide the principle matters more than the visibility. The key is making it a decision, not an accident: know which bots you're blocking, what each one feeds, and what you're giving up.
How do you check what you're blocking right now?
Three places to look, because robots.txt alone isn't the whole story:
- robots.txt — read the actual rules per bot (or run a SeeGeo audit, which evaluates all 16 major crawlers individually).
- Your CDN — Cloudflare blocks AI crawlers by default for many accounts now. Your robots.txt can say "welcome" while the firewall says no.
- Your rendering — most AI crawlers don't run JavaScript. If your content only appears client-side, you're "blocking" every AI bot without a single rule.
The uncomfortable truth: plenty of businesses that never chose to block anything are invisible to AI anyway. That's why we built the audit to check all three layers.