How do I block AI crawlers in robots.txt?
One group per bot: User-agent: GPTBot on one line and Disallow: / on the next. Bots read the most specific group that names them and fall back to User-agent: * otherwise, so a file that blocks GPTBot and allows * does what it looks like it does. Upload the result as /robots.txt at the root of your domain; changes take effect the next time each crawler fetches the file, usually within a day.
Which AI crawlers should you block, if any?
Split them by purpose. AI search crawlers (OAI-SearchBot, PerplexityBot, Amazonbot, DuckAssistBot) and on-demand fetchers (ChatGPT-User, Claude-User, Perplexity-User) are how assistants read your site to answer a customer; blocking them removes you from those answers. Training crawlers (GPTBot, ClaudeBot, Google-Extended, CCBot, Bytespider, Meta-ExternalAgent, Applebot-Extended) feed future models and do not affect today's recommendations. Search crawlers (Googlebot, Bingbot) should stay open — Bingbot in particular also feeds parts of ChatGPT's search. The crawler library has a page per bot.
Is robots.txt enough?
It is enough to block, and not enough to allow. A CDN or firewall can refuse a bot your robots.txt welcomes, which is the single most common reason a site that thinks it is open is invisible to AI. Confirm with the free crawler check, which fetches your homepage as four AI crawlers and reads the robots.txt you just uploaded.
Frequently asked questions
How do I block AI crawlers in robots.txt?
Add a group per bot: a User-agent line with the bot's token (GPTBot, ClaudeBot, PerplexityBot, Google-Extended, CCBot…) followed by Disallow: /. Put it at https://yourdomain.com/robots.txt. The generator above writes exactly those lines for the bots you tick, and leaves everything else allowed with a User-agent: * group.
Should a small business block AI crawlers?
Usually not. Blocking the AI search crawlers removes you from the answers customers get in ChatGPT, Claude, Perplexity and Gemini. If your concern is model training, block only the training crawlers — GPTBot, Google-Extended, CCBot, Bytespider, Meta-ExternalAgent, Applebot-Extended — and keep the search ones open; the 'Block training only' preset does this.
Does blocking Google-Extended remove me from Google?
No. Google-Extended only controls Gemini model training. Google Search and AI Overviews use Googlebot, so blocking Google-Extended changes neither.
Is robots.txt enough to allow AI crawlers?
No. Firewalls and CDNs such as Cloudflare can block AI crawlers at the network level regardless of robots.txt, and Cloudflare does so by default for many accounts. After uploading the file, run the free crawler check to confirm each bot actually receives your homepage.
Updated 2026-09-08.