---
title: "AI crawlers explained: every bot that matters"
description: "Every major AI and search crawler: what it does with your content, whether to block it, and the exact robots.txt lines — in plain language."
canonical: https://see-geo.com/bots
language: en
updated: 2026-09-08
publisher: SeeGeo
alternates:
  fr: https://see-geo.com/fr/bots
  de: https://see-geo.com/de/bots
---

# AI crawlers explained: every bot that matters

Every major AI and search crawler: what it does with your content, whether to block it, and the exact robots.txt lines — in plain language.

## Pages

- [What is GPTBot?](https://see-geo.com/bots/gptbot) — What GPTBot (OpenAI) does with your content, whether to block it, and the exact robots.txt lines — in plain language.
- [What is OAI-SearchBot?](https://see-geo.com/bots/oai-searchbot) — What OAI-SearchBot (OpenAI) does with your content, whether to block it, and the exact robots.txt lines — in plain language.
- [What is ChatGPT-User?](https://see-geo.com/bots/chatgpt-user) — What ChatGPT-User (OpenAI) does with your content, whether to block it, and the exact robots.txt lines — in plain language.
- [What is ClaudeBot?](https://see-geo.com/bots/claudebot) — What ClaudeBot (Anthropic) does with your content, whether to block it, and the exact robots.txt lines — in plain language.
- [What is Claude-User?](https://see-geo.com/bots/claude-user) — What Claude-User (Anthropic) does with your content, whether to block it, and the exact robots.txt lines — in plain language.
- [What is PerplexityBot?](https://see-geo.com/bots/perplexitybot) — What PerplexityBot (Perplexity) does with your content, whether to block it, and the exact robots.txt lines — in plain language.
- [What is Perplexity-User?](https://see-geo.com/bots/perplexity-user) — What Perplexity-User (Perplexity) does with your content, whether to block it, and the exact robots.txt lines — in plain language.
- [What is Google-Extended?](https://see-geo.com/bots/google-extended) — What Google-Extended (Google) does with your content, whether to block it, and the exact robots.txt lines — in plain language.
- [What is CCBot?](https://see-geo.com/bots/ccbot) — What CCBot (Common Crawl) does with your content, whether to block it, and the exact robots.txt lines — in plain language.
- [What is Bytespider?](https://see-geo.com/bots/bytespider) — What Bytespider (ByteDance) does with your content, whether to block it, and the exact robots.txt lines — in plain language.
- [What is Amazonbot?](https://see-geo.com/bots/amazonbot) — What Amazonbot (Amazon) does with your content, whether to block it, and the exact robots.txt lines — in plain language.
- [What is Applebot-Extended?](https://see-geo.com/bots/applebot-extended) — What Applebot-Extended (Apple) does with your content, whether to block it, and the exact robots.txt lines — in plain language.
- [What is Meta-ExternalAgent?](https://see-geo.com/bots/meta-externalagent) — What Meta-ExternalAgent (Meta) does with your content, whether to block it, and the exact robots.txt lines — in plain language.
- [What is DuckAssistBot?](https://see-geo.com/bots/duckassistbot) — What DuckAssistBot (DuckDuckGo) does with your content, whether to block it, and the exact robots.txt lines — in plain language.
- [What is Googlebot?](https://see-geo.com/bots/googlebot) — What Googlebot (Google) does with your content, whether to block it, and the exact robots.txt lines — in plain language.
- [What is Bingbot?](https://see-geo.com/bots/bingbot) — What Bingbot (Microsoft) does with your content, whether to block it, and the exact robots.txt lines — in plain language.

## Frequently asked questions

### How can I check whether AI crawlers like GPTBot can read my website?

Check two things: that your robots.txt does not disallow the crawler, and that your server or CDN actually returns the page when a request arrives with that crawler's user agent. The second check matters because firewalls such as Cloudflare can block AI bots regardless of robots.txt. SeeGeo's free crawler check does both for GPTBot, OAI-SearchBot, ClaudeBot and PerplexityBot in a few seconds.

### Which AI crawlers should a small business allow?

Allow the search-type crawlers that feed answers customers see — OAI-SearchBot and ChatGPT-User for ChatGPT, Googlebot for Google's AI Overviews, PerplexityBot, and ClaudeBot — since blocking them removes you from recommendations. The training-only crawlers (GPTBot, Google-Extended, CCBot) are a genuine choice: allowing them helps future models know you exist, blocking them keeps your content out of training data.

### Does robots.txt alone control AI crawlers?

No. robots.txt is a request that well-behaved crawlers honour, but your CDN or firewall can block a crawler that robots.txt allows, and a crawler can be listed as allowed while your pages render only with JavaScript it never runs. Access is only real when the fetch succeeds and the content is in the HTML.