What is Bytespider?

What does Bytespider do with your content?

Trains ByteDance's models (Doubao). Like most AI crawlers, Bytespider reads your pages as plain HTML — it does not run JavaScript — so content that only appears after scripts execute is invisible to it.

Should you block Bytespider?

This is a genuine trade-off. Allowing Bytespider lets future ByteDance models learn your business exists — useful when customers ask those models for recommendations. Blocking it keeps your content out of training data at the cost of that future visibility. Neither answer is wrong; it depends on what you value more.

How do you allow or block Bytespider in robots.txt?

Add one of these snippets to the robots.txt file at the root of your site. The first explicitly allows Bytespider everywhere; the second blocks it completely.

# Allow Bytespider
User-agent: Bytespider
Allow: /

# Block Bytespider
User-agent: Bytespider
Disallow: /

Is robots.txt enough?

Not always. CDNs and firewalls (Cloudflare in particular) can block crawlers at the network level regardless of what robots.txt says — Cloudflare blocks AI crawlers by default for many accounts. If you want Bytespider to reach your site, check your CDN's bot settings too. SeeGeo's free audit checks both.

What is Bytespider?

Bytespider is ByteDance's AI training crawler. Trains ByteDance's models (Doubao).

Should I block Bytespider?

This is a genuine trade-off. Allowing Bytespider lets future ByteDance models learn your business exists — useful when customers ask those models for recommendations.

Does Bytespider run JavaScript?

No. Bytespider, like most AI crawlers, reads raw HTML without executing JavaScript. If your content only renders client-side, it is effectively invisible to it.