Quick answer: we ran SeeGeo's audit engine over 107 real small-business websites — restaurants, trades, clinics, shops, professional services across the US, UK, Canada, Australia, Ireland, France, and Germany. Only 3 of 107 (2.8%) explicitly block any major AI crawler in robots.txt, so the popular "your site is blocking AI without you knowing" story is rare in this sample. The real gaps are quieter: 29 of 107 (27.1%) have no structured data anywhere, 69 of 107 (64.5%) never plainly say what the business does at the top of the homepage, and 101 of 107 graded C or worse on overall AI visibility. One genuine surprise: 27 of 107 (25.2%) already have an llms.txt file — and almost none of them put it there. Every number below comes from the actual runs, with the full methodology published at the end of this post.
How many small-business sites block AI crawlers?
3 of 107 (2.8%). Those three explicitly name AI crawlers in hand-written robots.txt groups and disallow them from the whole site. Counting wildcard rules that happen to catch AI bots too, the number stays 3 of 107 — no site in this sample is root-blocked for an AI crawler by accident of a User-agent: * rule.
That number is worth sitting with, because the industry narrative — ours included, at times — leans on the idea that small businesses are accidentally invisible to AI because something is blocking the crawlers. In this sample, that's the exception. Nearly everyone lets AI in. What happens after the crawler gets in is where the visibility is lost.
Which AI crawler is blocked most?
| Crawler | Explicit robots.txt block | Any root block |
|---|---|---|
| GPTBot (OpenAI, training) | 3 of 107 (2.8%) | 3 of 107 (2.8%) |
| ClaudeBot (Anthropic) | 3 of 107 (2.8%) | 3 of 107 (2.8%) |
| Google-Extended (Gemini training) | 2 of 107 (1.9%) | 2 of 107 (1.9%) |
| CCBot (Common Crawl) | 2 of 107 (1.9%) | 2 of 107 (1.9%) |
| PerplexityBot | 1 of 107 (0.9%) | 1 of 107 (0.9%) |
| OAI-SearchBot (ChatGPT citations) | 0 of 107 (0.0%) | 0 of 107 (0.0%) |
| Googlebot | 0 of 107 (0.0%) | 0 of 107 (0.0%) |
| Bingbot | 0 of 107 (0.0%) | 0 of 107 (0.0%) |
GPTBot and ClaudeBot tie for most-blocked at 3 sites each. Notably, OAI-SearchBot — the crawler that decides whether ChatGPT can cite a site — is blocked by nobody in this sample.
Who blocked them, and was it deliberate?
Among the 3 blocking sites: 2 show hand-written robots.txt rules with no CDN or firewall signals, and 1 pairs its robots.txt rules with a Cloudflare front. All three name the AI bots explicitly in their own robots.txt groups — these look like decisions, not defaults. 0 of 107 (0.0%) sites carry the signature of Cloudflare's managed robots.txt in this sample.
One honest caveat that cuts the other way: 31 of 107 (29.0%) of the audited sites sit behind Cloudflare, and CDN-level bot blocking that leaves no external signature is not measurable from outside. Our robots.txt numbers are exact; our CDN numbers are a floor. (A related receipt from our own run: one site served HTTP 403 to our honestly-identified audit crawler while serving a normal page to a browser — network-level bot filtering is real, which is why sites that blocked our checker were excluded from every statistic rather than counted as "blocks AI.")
The JavaScript problem is smaller than advertised
3 of 107 (2.8%) sites serve a homepage whose primary content doesn't exist unless JavaScript runs — which is how most AI crawlers read the web, so those three are effectively blank to AI. We expected more. Modern site builders appear to server-render well enough that the "your beautiful JS site is invisible" warning, while real, is also the exception in this sample.
Structured data is where the sample actually fails
- No JSON-LD structured data anywhere on the crawled pages: 29 of 107 (27.1%)
- JSON-LD on the homepage: 75 of 107 (70.1%)
- JSON-LD beyond the homepage: 69 of 107 (64.5%)
- A plain-language "what we do" statement near the top of the homepage: only 38 of 107 (35.5%) — meaning 69 of 107 (64.5%) make both humans and machines infer what the business is
- Overall grades from the audit engine: B 6 · C 38 · D 38 · F 25 — 101 of 107 (94.4%) graded C or worse
This is the actual shape of the small-business AI-visibility problem in our data: the door is open, but the room is unlabeled.
The llms.txt surprise: adoption by default
27 of 107 (25.2%) sites have an llms.txt file — a number that would suggest remarkable adoption of a contested young standard. It doesn't. We re-fetched and classified every one:
- Every Shopify store in the sample (8 of 8) carries Shopify's auto-generated "Agent Instructions" llms.txt.
- Every Wix hit carries Wix's auto-generated markdown file.
- Several WordPress hits are generated by the All in One SEO plugin, which stamps its own signature into the file.
In other words: platforms are adopting llms.txt on their customers' behalf. Most of these owners almost certainly don't know the file exists. Whether the standard matters is a separate question — our honest verdict on llms.txt is unchanged — but "who is adopting llms.txt" now has a data-backed answer: software vendors, not site owners.
What should a business owner do?
The study's practical order of operations, given what actually failed:
- Say what you do, in plain words, at the top of your homepage. The single most-failed check (64.5%) and the cheapest fix on this list.
- Add structured data — at minimum an Organization or LocalBusiness block, plus FAQ markup where you genuinely answer questions. A quarter of the sample has none at all.
- Only then worry about crawler access. Check it — it takes seconds and the rare failure is severe — but in 97% of cases in this sample, the door was already open.
SeeGeo's free audit runs all of these checks on your site in about 30 seconds, no signup. Deeper dives: should you block AI crawlers?, is your website visible to ChatGPT?, and the AI visibility glossary.
Frequently asked questions
Does blocking GPTBot remove a site from ChatGPT? Blocking GPTBot stops OpenAI's model-training crawler. ChatGPT's search citations depend on OAI-SearchBot — blocked by 0 of 107 sites here. Blocking both makes citation practically impossible; blocking only GPTBot mostly affects what future models learn.
Is robots.txt the only way sites block AI crawlers? No. CDNs and firewalls can block at the network level with no robots.txt trace. We report detected CDN signals separately (32 of 107 sites show a WAF/CDN signature), and we treat our CDN numbers as a floor because signature-free blocking is invisible from outside.
Did the three blocking sites choose to block AI? It looks that way: all three name AI bots explicitly in hand-written robots.txt groups, and none carries the signature of a CDN-managed robots.txt. In this sample, blocking was rare but deliberate.
Why is your llms.txt number so much higher than reported adoption? Because platforms ship it by default now. Shopify auto-generates an llms.txt for every store, Wix generates one, and popular WordPress SEO plugins emit one. Owner-initiated adoption in this sample is a small fraction of the 25.2%.
Can I see the data? Yes — the aggregate data, findings, methodology, and the llms.txt classification are published on GitHub: github.com/Tunisian-Aaron/ai-visibility-study-2026 (CC BY 4.0). The domain list and per-site results are deliberately not published — this study is statistics, not a wall of shame.
Full methodology
Published verbatim; runs conducted 2026-08-17.
Sample. 120 candidate domains were collected from public "best of" listicles, local directory pages, and trade-association member lists across 12 business categories (restaurants/cafés, trades, clinics, salons/fitness, independent retail, professional services, auto repair, artisan producers) and 7 countries (US, UK, Ireland, Canada, Australia, France, Germany). Selection rules: small businesses only; national chains, franchises, and enterprise-scale companies excluded during collection (43 exclusions logged with reasons); aggregator/directory domains excluded; the business's own site only; no site owned by SeeGeo or anyone we have a relationship with. This is a convenience sample — it is not random and not representative of the web. Findings describe this sample only, and we do not name the audited businesses.
Runs. Each site was crawled once on 2026-08-17 by the same engine that powers SeeGeo's public audit: homepage, up to ~8 site pages, robots.txt, sitemap, and an llms.txt probe. User-agent: SeeGeoAudit/1.0 (+https://see-geo.com/bots) — honestly identified, with an explanation page. Same-domain requests were spaced at least 1 second apart; one visit per site, with a single retry after network failure. 13 of 120 candidates were excluded at run time (8 unreachable, 3 domains moved, 2 blocked our checker); they appear in no denominator. A site that blocks or challenges our checker is recorded as "could not verify" — never counted as "blocks AI crawlers."
Definitions. Blocks bot X (explicit): robots.txt contains a user-agent group explicitly matching X, and evaluating X against the site root yields disallowed; wildcard-only disallows are tallied separately and never merged into the explicit number. CDN/WAF signals: response-header signatures only (e.g. server: cloudflare); CDN blocking that leaves no signature is not externally measurable, so CDN numbers are a floor. Primary content missing without JavaScript: the engine's deterministic verdict that the crawled HTML — no JS execution, which is what most AI crawlers see — contains no usable primary content. Structured data: parseable JSON-LD; "beyond the homepage" means at least one non-homepage crawled page carries it. llms.txt present: HTTP 200 at /llms.txt with a non-HTML body; soft-404s serving the site shell count as absent, and one initial positive was corrected to absent on re-verification (logged). Every present file was re-fetched and classified by signature (Shopify auto-generated, Wix auto-generated, SEO-plugin output, other). "What we do" statement: the engine's shipped heuristic — English, French, and German patterns over the first 1,200 characters of homepage text plus the meta description.
Limitations. Convenience sample; results describe these 107 sites on this date, nothing more. CDN-level blocking is partially detectable at best. Bot rosters change; we evaluated the engine's roster as of the run date. Some "unreachable" exclusions may themselves be network-level blocks of our checker — indistinguishable from outside, which is one more reason excluded sites never enter a denominator. Subgroup statistics are reported only where n ≥ 20. The underlying aggregate data is published at github.com/Tunisian-Aaron/ai-visibility-study-2026.