Every bot that decides your AI visibility
Whether ChatGPT, Claude, Perplexity, or Google can read — and recommend — your business comes down to a handful of crawlers. What each one does, and whether you should let it in.
AI search crawlers
OAI-SearchBot
Builds the index behind ChatGPT search — this is the bot that decides if ChatGPT can cite you.
PerplexityPerplexityBot
Indexes pages for Perplexity's answer engine — Perplexity cites sources on almost every answer.
AmazonAmazonbot
Feeds Alexa and Amazon's AI answers.
DuckDuckGoDuckAssistBot
Powers DuckDuckGo's AI-assisted answers.
On-demand AI fetchers
AI training crawlers
GPTBot
Collects pages to train ChatGPT's underlying models.
AnthropicClaudeBot
Crawls pages for Claude's models and search index.
GoogleGoogle-Extended
Controls whether Google may train Gemini on your content. Blocking it does NOT affect Google Search or AI Overviews.
Common CrawlCCBot
Builds a public web archive that many AI companies train models on. Blocking it quietly removes you from future models' knowledge.
ByteDanceBytespider
Trains ByteDance's models (Doubao).
AppleApplebot-Extended
Controls whether Apple may use your content for Apple Intelligence.
MetaMeta-ExternalAgent
Collects pages to train Meta's Llama models.
Search crawlers
Which of these can actually reach your site? The free SeeGeo audit evaluates your robots.txt against every crawler on this page — plus your CDN and JavaScript rendering.
Run a free audit