What percentage of small business websites block AI crawlers?
2.8%. 3 of 107 sites explicitly name and disallow at least one major AI crawler in robots.txt; the number is unchanged when wildcard rules are counted (3 of 107). The popular claim that small sites are “blocking AI without knowing it” is rare in this sample. 31 of 107 (29.0%) sit behind Cloudflare, where network-level blocking can happen regardless of robots.txt; of those, 1 also block in robots.txt.
| Crawler | Explicitly blocked | Effectively blocked | Share of sites |
|---|---|---|---|
| GPTBot | 3 of 107 | 3 of 107 | 2.8% |
| OAI-SearchBot | 0 of 107 | 0 of 107 | 0.0% |
| ClaudeBot | 3 of 107 | 3 of 107 | 2.8% |
| PerplexityBot | 1 of 107 | 1 of 107 | 0.9% |
| Google-Extended | 2 of 107 | 2 of 107 | 1.9% |
| CCBot | 2 of 107 | 2 of 107 | 1.9% |
| Googlebot | 0 of 107 | 0 of 107 | 0.0% |
| Bingbot | 0 of 107 | 0 of 107 | 0.0% |
How many small business websites have structured data?
75 of 107 (70.1%) carry JSON-LD on the homepage and 69 (64.5%) beyond it; 29 (27.1%) have none anywhere. Without it, a machine classifying the business has only the prose — and the next figure shows how often the prose doesn't help.
How many websites say what the business does?
38 of 107 (35.5%). The other 69 (64.5%) open with a tagline, a slideshow or a menu and never state plainly what they sell, where, and for whom. This is the most common gap in the study and the cheapest to fix: one sentence at the top of the homepage. Only 3 sites lost their content entirely without JavaScript.
How many websites have an llms.txt file?
27 of 107 (25.2%). Almost none were placed by the owner — platforms and plugins add them — and they made no measurable difference to the grade. What llms.txt is and isn't.
What grade does a typical small business website get?
| Grade | Sites | Share |
|---|---|---|
| A | 0 | 0.0% |
| B | 6 | 5.6% |
| C | 38 | 35.5% |
| D | 38 | 35.5% |
| F | 25 | 23.4% |
101 of 107 graded C or worse. SeeGeo's grades are anchored to this distribution — an A is the top decile of real sites, not an arbitrary threshold — which is why a B is already a good result. How we measure.
Does the website platform matter?
| Platform | Sites | Block any AI crawler | No structured data |
|---|---|---|---|
| wordpress | 38 | 0 | 4 |
| custom/unknown | 34 | 2 | 20 |
Only platforms with enough sites for a fair comparison are shown. WordPress sites in the sample blocked nothing and rarely lacked schema (a plugin usually adds it); custom builds accounted for most of the no-structured-data cases. Our platform defaults study covers what each platform switches on for you.
Method
Sample: 120 sites drawn from local-business categories in seven countries; 8 unreachable, 3 moved and 2 that blocked the checker were excluded, leaving 107. Each site was crawled by SeeGeo's audit engine the way an AI crawler reads it — no JavaScript, honest user agent — and scored deterministically. Aggregate data is published at github.com/Tunisian-Aaron/ai-visibility-study-2026 under CC BY 4.0 and served as JSON with a field dictionary at see-geo.com/data; individual businesses are never named. Generated 2026-08-17; the full write-up has the country and category breakdowns.
Frequently asked questions
What percentage of small business websites block AI crawlers?
2.8% — 3 of 107 real small-business sites audited in 2026 explicitly block at least one major AI crawler in robots.txt. Counting wildcard rules that happen to catch AI bots, the figure is the same 3 of 107. Blocking is rare; being unreadable is common.
How many small business websites have no structured data?
27.1% — 29 of 107 sites have no JSON-LD anywhere, so a machine has to guess what kind of business they are. 75 of 107 carry some structured data on the homepage.
How many websites say what the business does on the homepage?
Only 38 of 107 (35.5%) plainly state what the business does near the top of the homepage; 69 (64.5%) do not, which is the single most common AI-visibility gap in the study.
How many websites have an llms.txt file?
27 of 107 (25.2%) served an llms.txt in 2026 — almost all of them placed there by a platform or plugin rather than the owner. It made no measurable difference to their AI-visibility grade.
What AI visibility grade does a typical small business website get?
C or worse. 101 of 107 (94.4%) graded C, D or F on SeeGeo's audit; 6 graded B and none graded A. Grades are anchored to this same distribution, so an A means the top decile of real sites.
Updated 2026-09-08. Cite as: SeeGeo, AI visibility study of 107 small-business websites, 2026.