What is Google-Extended?
Google-Extended is Google's robots.txt control for AI use of your content, not a crawler. Controls whether Google may train Gemini on your content. Blocking it does NOT affect Google Search or AI Overviews.
What does Google-Extended do with your content?
Controls whether Google may train Gemini on your content. Blocking it does NOT affect Google Search or AI Overviews. Google-Extended never visits your site itself: Googlebot does the fetching, and Google-Extended is the robots.txt switch for the AI use of that content.
Should you block Google-Extended?
Blocking Google-Extended only opts your content out of Gemini model training. It does NOT remove you from Google Search or from AI Overviews — both of those run on Googlebot. If your goal is staying visible in Google while opting out of model training, blocking Google-Extended is a reasonable, low-risk choice.
How do you allow or block Google-Extended in robots.txt?
Add one of these snippets to the robots.txt file at the root of your site. The first explicitly allows Google-Extended everywhere; the second blocks it completely.
# Allow Google-Extended
User-agent: Google-Extended
Allow: /
# Block Google-Extended
User-agent: Google-Extended
Disallow: /Will you see Google-Extended in your server logs?
No. Google-Extended is a robots.txt token, not a crawler: Google documents no separate user-agent for it. Pages are fetched by Googlebot, and the Google-Extended rule only decides whether Google may use what Googlebot collected for AI.
Is robots.txt enough?
Not always. CDNs and firewalls (Cloudflare in particular) can block crawlers at the network level regardless of what robots.txt says — Cloudflare blocks AI crawlers by default for many accounts. If you want Google-Extended to reach your site, check your CDN's bot settings too. SeeGeo's free audit checks both.
How do I check whether Google-Extended can read my site right now?
Only robots.txt matters here: look for a group naming Google-Extended. There is no request to test, because Google-Extended never fetches anything; Googlebot's access is a separate question, which SeeGeo's free crawler check answers.
How does Google-Extended compare to similar crawlers?
Google-Extended is one of 17 crawlers the SeeGeo audit checks individually. The closest comparisons:
| Crawler | Operator | Role | Runs JavaScript? |
|---|---|---|---|
| Google-Extended | AI training crawler | n/a (robots.txt token) | |
| Googlebot | search engine crawler | Delayed rendering | |
| GPTBot | OpenAI | AI training crawler | No |
| ClaudeBot | Anthropic | AI training crawler | No |
| CCBot | Common Crawl | AI training crawler | No |
Why does crawler access matter? The numbers
Crawler access is where AI visibility starts or ends: a blocked crawler can't read you, and an AI that can't read you can't recommend you.
- Visitors arriving from AI assistants convert at roughly 4.4x the rate of traditional organic search on average (Semrush, cross-industry, 2026).
- AI-assistant referrals are still only about 1% of total web traffic — small, but the fastest-growing acquisition channel measured (multiple 2026 studies).
- Adobe Digital Insights (Q1 2026) measured AI-assistant visitors converting 42% better than non-AI traffic — a full reversal from the year before.
- Cloudflare blocks AI crawlers by default for many accounts, so sites are often invisible to AI without anyone having decided to be.
- Content edits alone — statistics, citations, quotable structure — can raise AI visibility on the order of 30–40% (Princeton GEO study, KDD 2024).
Frequently asked questions
What is Google-Extended?
Google-Extended is Google's robots.txt control for AI use of your content, not a crawler. Controls whether Google may train Gemini on your content. Blocking it does NOT affect Google Search or AI Overviews.
Should I block Google-Extended?
Blocking Google-Extended only opts your content out of Gemini model training. It does NOT remove you from Google Search or from AI Overviews — both of those run on Googlebot.
Does Google-Extended run JavaScript?
The question doesn't apply: Google-Extended fetches nothing. Googlebot fetches your pages, and the Google-Extended token only governs whether Google may use them for AI.
Can Google-Extended read my website?
Google-Extended never reads your website: Googlebot does. Google-Extended is only a robots.txt rule deciding whether Google may use that content for AI, so check your robots.txt for a group naming it.
Does blocking Google-Extended affect Google Search rankings?
No. Google-Extended only controls Gemini model training. Google Search and AI Overviews use Googlebot, so blocking Google-Extended has no effect on either.