How can I check whether AI crawlers like GPTBot can read my website?
By hand, in two steps. First open https://yourdomain.com/robots.txt and look for a User-agent: group naming the bot — GPTBot, OAI-SearchBot, ClaudeBot, PerplexityBot — or the * group it falls back to; a Disallow: / under it blocks the bot from everything. Then request your homepage with the bot's user agent, for example curl -A GPTBot https://yourdomain.com/, and confirm you get a 200 with your real HTML rather than a 403 or a challenge page. The tool above runs exactly those two steps for you.
Why would GPTBot be blocked when robots.txt allows it?
Because the block happens before robots.txt is ever consulted. Cloudflare, Akamai, DataDome and Imperva can challenge or refuse requests whose user agent is a known AI crawler, and Cloudflare blocks AI crawlers by default on many plans. Your robots.txt looks welcoming while the door is locked. Our unblock guides show where each vendor's switch is.
Which crawlers should you allow?
For being recommended today: the AI search crawlers (OAI-SearchBot, ClaudeBot, PerplexityBot) and Google's crawlers. Training crawlers (GPTBot, Google-Extended, CCBot) are a separate choice that doesn't affect current answers. The crawler library has a page per bot with the exact robots.txt lines to allow or block it, and the robots.txt generator writes the whole file from a few ticks.
Frequently asked questions
How can I check whether AI crawlers like GPTBot can read my website?
Two things have to be true: your robots.txt must not disallow the bot, and your server or CDN must actually serve the page when the bot asks for it. Checking robots.txt by hand catches the first; only fetching your homepage with the bot's user agent catches the second, which is why this tool does both.
My robots.txt allows GPTBot — why does the fetch still fail?
Because robots.txt is a request, not a gate. Firewalls and CDNs — Cloudflare, Akamai, DataDome, Imperva — can block or challenge AI crawlers at the network level regardless of robots.txt, and Cloudflare does so by default for many accounts. A 403, 429 or 503 with a firewall header means the bot never sees your content.
Which AI crawlers matter most for being recommended?
OAI-SearchBot (ChatGPT search), ClaudeBot (Anthropic), PerplexityBot and Google's crawlers decide whether an assistant can read and cite you. GPTBot, Google-Extended and CCBot feed model training rather than live answers, so blocking them affects future models, not today's recommendations.
Does the check change anything on my site?
No. It reads your robots.txt and requests your homepage a few times, identifying itself honestly as a SeeGeo check, the same way the free audit does. Nothing is stored beyond the result you see on this page.
Updated 2026-09-08.