What is Googlebot?
Googlebot is Google's search engine crawler. Google's main crawler. Powers Google Search AND the AI Overviews box at the top of results.
What does Googlebot do with your content?
Google's main crawler. Powers Google Search AND the AI Overviews box at the top of results. It follows the rules you publish in robots.txt, a small text file at the root of your site that tells automated visitors what they may read (defined by RFC 9309, the Robots Exclusion Protocol standard).
Should you block Googlebot?
Blocking Googlebot removes you from Google Search AND from AI Overviews, which are built on Googlebot's index. For any business that wants customers to find it, blocking Googlebot is almost never the right call.
How do you allow or block Googlebot in robots.txt?
Add one of these snippets to the robots.txt file at the root of your site. The first explicitly allows Googlebot everywhere; the second blocks it completely.
# Allow Googlebot
User-agent: Googlebot
Allow: /
# Block Googlebot
User-agent: Googlebot
Disallow: /Is robots.txt enough?
Not always. CDNs and firewalls (Cloudflare in particular) can block crawlers at the network level regardless of what robots.txt says — Cloudflare blocks AI crawlers by default for many accounts. If you want Googlebot to reach your site, check your CDN's bot settings too. SeeGeo's free audit checks both.
Frequently asked questions
What is Googlebot?
Googlebot is Google's search engine crawler. Google's main crawler. Powers Google Search AND the AI Overviews box at the top of results.
Should I block Googlebot?
Blocking Googlebot removes you from Google Search AND from AI Overviews, which are built on Googlebot's index. For any business that wants customers to find it, blocking Googlebot is almost never the right call..
Does Googlebot run JavaScript?
Googlebot can render JavaScript for indexing, but rendering is delayed and imperfect — server-rendered content is always the safer path.