---
title: "What is robots.txt?"
description: "robots.txt controls which crawlers may read your site — now including every AI crawler. The syntax, the accidents, and what it can't control."
canonical: https://see-geo.com/glossary/robots-txt
language: en
published: 2026-09-08
updated: 2026-09-08
publisher: SeeGeo
alternates:
  fr: https://see-geo.com/fr/glossary/robots-txt
  de: https://see-geo.com/de/glossary/robots-txt
---

# What is robots.txt?

robots.txt is a plain-text file at a website's root that tells crawlers which parts of the site they may read, using the Robots Exclusion Protocol (RFC 9309) — and it's where AI crawler access is granted or denied.

## How does robots.txt control AI visibility?

Every major AI crawler — GPTBot, ClaudeBot, PerplexityBot, Google-Extended — checks robots.txt before reading and honors what it finds. One leftover 'Disallow: /' under a wildcard user-agent, written years ago for some other reason, silently removes a site from AI training data, AI search indexes, and live retrieval at once. The file is tiny; its blast radius is not.

```
# Allow a specific AI crawler
User-agent: GPTBot
Allow: /

# Block one, allow the rest
User-agent: CCBot
Disallow: /
```

[Every crawler's exact lines](https://see-geo.com/bots)

## What can't robots.txt control?

The network layer. CDNs and firewalls decide whether a crawler's request ever reaches your server — Cloudflare blocks AI crawlers by default for many accounts — and no robots.txt directive can override a blocked connection. Auditing access means checking both layers, which is why SeeGeo tests each crawler's real reachability rather than reading your rules and assuming.

## What's the most common robots.txt mistake?

The leftover blanket block: a 'User-agent: * / Disallow: /' written for a staging site and shipped to production, or an old rule blocking a directory that modern crawlers need. Because the file fails silently — nothing breaks, you just quietly vanish — these survive for years. Re-check it after every migration, and test what crawlers actually experience rather than what the file appears to say.

## Frequently asked questions

### What does robots.txt mean?

robots.txt is a plain-text file at a website's root that tells crawlers which parts of the site they may read, using the Robots Exclusion Protocol (RFC 9309) — and it's where AI crawler access is granted or denied.

### Do AI companies actually respect robots.txt?

The major operators — OpenAI, Anthropic, Google, Perplexity — publicly commit to it and observably comply. Edge cases exist among smaller scrapers, but the dominant real-world problem is the reverse: sites blocking legitimate crawlers by accident.

### What does 'User-agent: *' do to AI crawlers?

The wildcard applies to every crawler without a more specific rule — including all AI crawlers. A 'Disallow: /' under it blocks everything from everyone, which is the single most common accidental way sites vanish from AI answers.

### Where do I put robots.txt?

At the exact root: yoursite.com/robots.txt. Crawlers look only there — a robots.txt in a subdirectory is ignored entirely, and a missing file means no rules, which permits everything.

## Related pages

- [What is AI crawler?](https://see-geo.com/glossary/ai-crawler)
- [What is LLM training data?](https://see-geo.com/glossary/llm-training-data)
- [What is Google AI Overviews?](https://see-geo.com/glossary/ai-overviews)