What is robots.txt?
robots.txt is a plain-text file at a website's root that tells crawlers which parts of the site they may read, using the Robots Exclusion Protocol (RFC 9309) — and it's where AI crawler access is granted or denied.
How does robots.txt control AI visibility?
Every major AI crawler — GPTBot, ClaudeBot, PerplexityBot, Google-Extended — checks robots.txt before reading and honors what it finds. One leftover 'Disallow: /' under a wildcard user-agent, written years ago for some other reason, silently removes a site from AI training data, AI search indexes, and live retrieval at once. The file is tiny; its blast radius is not.
# Allow a specific AI crawler
User-agent: GPTBot
Allow: /
# Block one, allow the rest
User-agent: CCBot
Disallow: /What can't robots.txt control?
The network layer. CDNs and firewalls decide whether a crawler's request ever reaches your server — Cloudflare blocks AI crawlers by default for many accounts — and no robots.txt directive can override a blocked connection. Auditing access means checking both layers, which is why SeeGeo tests each crawler's real reachability rather than reading your rules and assuming.
What's the most common robots.txt mistake?
The leftover blanket block: a 'User-agent: * / Disallow: /' written for a staging site and shipped to production, or an old rule blocking a directory that modern crawlers need. Because the file fails silently — nothing breaks, you just quietly vanish — these survive for years. Re-check it after every migration, and test what crawlers actually experience rather than what the file appears to say.
Frequently asked questions
What does robots.txt mean?
robots.txt is a plain-text file at a website's root that tells crawlers which parts of the site they may read, using the Robots Exclusion Protocol (RFC 9309) — and it's where AI crawler access is granted or denied.
Do AI companies actually respect robots.txt?
The major operators — OpenAI, Anthropic, Google, Perplexity — publicly commit to it and observably comply. Edge cases exist among smaller scrapers, but the dominant real-world problem is the reverse: sites blocking legitimate crawlers by accident.
What does 'User-agent: *' do to AI crawlers?
The wildcard applies to every crawler without a more specific rule — including all AI crawlers. A 'Disallow: /' under it blocks everything from everyone, which is the single most common accidental way sites vanish from AI answers.
Where do I put robots.txt?
At the exact root: yoursite.com/robots.txt. Crawlers look only there — a robots.txt in a subdirectory is ignored entirely, and a missing file means no rules, which permits everything.