What is CCBot?

What does CCBot do with your content?

Builds a public web archive that many AI companies train models on. Blocking it quietly removes you from future models' knowledge. Like most AI crawlers, CCBot reads your pages as plain HTML — it does not run JavaScript — so content that only appears after scripts execute is invisible to it.

Should you block CCBot?

CCBot feeds Common Crawl, a public archive many AI labs train on. Blocking it quietly removes you from the training data of future models — models that customers may ask for recommendations years from now. The trade-off is real: allow it for AI visibility, block it if keeping your content out of training data matters more to you.

How do you allow or block CCBot in robots.txt?

Add one of these snippets to the robots.txt file at the root of your site. The first explicitly allows CCBot everywhere; the second blocks it completely.

# Allow CCBot
User-agent: CCBot
Allow: /

# Block CCBot
User-agent: CCBot
Disallow: /

Is robots.txt enough?

Not always. CDNs and firewalls (Cloudflare in particular) can block crawlers at the network level regardless of what robots.txt says — Cloudflare blocks AI crawlers by default for many accounts. If you want CCBot to reach your site, check your CDN's bot settings too. SeeGeo's free audit checks both.

What is CCBot?

CCBot is Common Crawl's AI training crawler. Builds a public web archive that many AI companies train models on. Blocking it quietly removes you from future models' knowledge.

Should I block CCBot?

CCBot feeds Common Crawl, a public archive many AI labs train on. Blocking it quietly removes you from the training data of future models — models that customers may ask for recommendations years from now.

Does CCBot run JavaScript?

No. CCBot, like most AI crawlers, reads raw HTML without executing JavaScript. If your content only renders client-side, it is effectively invisible to it.