What is AI crawler?
An AI crawler is an automated program that reads web pages on behalf of an AI system — for training data (GPTBot, ClaudeBot), for an AI search index (OAI-SearchBot, PerplexityBot), or live, mid-conversation (ChatGPT-User, Claude-User).
What are the three kinds of AI crawler?
Training crawlers (GPTBot, ClaudeBot, Google-Extended, CCBot) collect text that future models learn from — blocking them trades future model knowledge of your business for keeping content out of training data. Index crawlers (OAI-SearchBot, PerplexityBot) feed AI search products; blocking them removes you from those answers directly. Live fetchers (ChatGPT-User, Claude-User, Perplexity-User) retrieve a page on demand when an assistant needs it mid-answer.
Why does JavaScript rendering matter so much?
Nearly all AI crawlers read raw HTML and do not execute JavaScript. A site that paints its content client-side — common with React, Vue, and site builders — serves those crawlers an effectively blank page. This is the single most damaging finding SeeGeo's audit produces, because it silently zeroes out every other effort: to an AI, the site does not say anything at all.
How do sites block AI crawlers by accident?
Two ways: leftover robots.txt rules written for another purpose, and CDN bot protection — Cloudflare in particular blocks AI crawlers by default for many accounts, at the network level, where robots.txt can't help. Sites are routinely invisible to AI without anyone having decided to be. An audit that checks each crawler individually is how you find out.
Frequently asked questions
What does AI crawler mean?
An AI crawler is an automated program that reads web pages on behalf of an AI system — for training data (GPTBot, ClaudeBot), for an AI search index (OAI-SearchBot, PerplexityBot), or live, mid-conversation (ChatGPT-User, Claude-User).
Should I block AI crawlers?
It's a real trade-off, not a default. Blocking training crawlers keeps content out of future models at the cost of those models knowing you exist; blocking index crawlers and live fetchers removes you from AI answers outright. If being recommended matters to your business, most crawlers should stay allowed.
Do AI crawlers respect robots.txt?
The major ones — from OpenAI, Anthropic, Google, Perplexity — publicly commit to honoring robots.txt (RFC 9309) and in practice do. The practical problem is usually the opposite: sites blocking crawlers unintentionally, not crawlers ignoring rules.
How do I see which AI crawlers can read my site?
Check robots.txt for each crawler's user agent, then check what your CDN does at the network layer — the part robots.txt can't show. SeeGeo's free audit tests all sixteen major search and AI crawlers individually and reports each one's access.