Quick answer: an "AI visibility score" is an estimate of a probability — how often an engine names you when a buyer asks — and every estimate has an error bar. Most tools print the number and hide the bar. As of this week SeeGeo asks each question five times per engine, reports the share of answers that named you with a 95% interval, and labels a week-over-week move "a real change" only when a paired test across your prompts says it cleared the noise. Our own latest reading is Claude 22% ±26, ChatGPT 11% ±16, Gemini 11% ±16. The second number is the honest one. It is also the one that will shrink as the method does its work.
This post is the methodology behind that change: what the research on auditing AI answers says, what we built, the arithmetic, and the one thing we have not solved.
The problem in one sentence
Ask ChatGPT "best accountant for a freelancer in Lyon" three times and you get three different lists. We measured this ourselves in September: 1,000 identical questions, the top three brands locked, everyone below them a coin flip. A single answer is therefore not a fact about your business. It is a draw from a distribution, and the only thing worth reporting is the distribution.
Three groups of researchers reached the same conclusion this year from different directions.
- A variance-components decomposition of brand answers attributes most of the total variation — on the order of three quarters, in the version we read — to simply re-running the same question on the same model. The model-and-prompt combination, the thing you can act on, is the minority of what moves.
- A finite-sample audit of repeated queries makes the obvious consequence explicit: the set of brands you observe grows with the number of calls, so any report that does not state its call count is not a measurement.
- A scoping review of brand-visibility audits proposes a validity checklist whose first items are "at least five draws per prompt and condition" and "variance reported". A separate reliability study puts numbers on the draw count: about 0.58 at five iterations, 0.74 at ten, 0.81 at fifteen. Three runs, which is what we used until this week and what most tools use when they repeat at all, sits below the bottom of that table.
None of this is exotic. It is the ordinary statistics of estimating a proportion from samples, applied to a domain that has been selling point estimates as if they were rankings.
What the scan does now
Five draws per prompt, at the engine's defaults
Every prompt in your panel is asked five times per engine, per scan, at the engine's default settings — because that is what your customer gets. We do not lower the temperature to make the numbers look steadier; a steadier number that does not describe the product your buyers use is a worse number. Search grounding still runs on the first draw only: the citations an engine retrieves change on the timescale of the web, and we showed earlier that grounded repeats return near-duplicate citations. The four extra draws cost tokens, not search fees, which is how five runs fit on every plan.
The interval comes from your prompts, not from the pile of answers
Here is the mistake that is easy to make. Twenty prompts times five runs is one hundred answers. A hundred coin flips give a proportion with a tidy ±10 point interval. So why not report 40% ±10?
Because the hundred answers are not independent. The five answers to one prompt agree with each other far more than they agree with another prompt's. In our September data, a prompt that named a brand once tended to name it every time, and a prompt that did not, did not. The real sample size is closer to the number of prompts than the number of answers.
So the interval is built the way survey statisticians handle clustered samples. Each prompt gets its own rate — five of five, two of five, zero of five. The scan's visibility is the pooled rate over all readings, exactly as before, so your trend line is continuous. But the uncertainty is the spread of those per-prompt rates divided by the square root of the number of prompts. Where that spread happens to be zero (a single prompt, or every prompt at exactly 0% or 100%) we fall back to the classic Wilson interval on the pooled count, so the bar never collapses to nothing.
A worked example, with the arithmetic you can check. Twenty prompts, five runs each. Eight prompts name you every time; twelve never do. Pooled visibility: 40 of 100 answers, 40%.
| How you compute the 95% interval | Result |
|---|---|
| Treat the 100 answers as independent coin flips (Wilson, n = 100) | 40% ±10 |
| Treat the 20 prompts as the sample (between-prompt spread ÷ √20) | 40% ±22 |
The naive bar is less than half the honest one. A tool that prints the first is not lying about the 40; it is lying about how sure it is.
A change is a change only when it beats the bar
Last week 30%, this week 38%. Up eight points. Is it?
Scans share a prompt panel, so we can ask the right question: for each prompt, did its rate go up or down, and do those per-prompt differences agree? That is a paired comparison — the mean of the differences against their own spread, over the prompts both scans asked. If the mean clears about two standard errors, the project page and the weekly digest say a real change. If it does not, they say within the noise band, and the digest's subject line says "steady" even when the pooled number moved. Fewer than three shared prompts is untestable and is never reported as a change.
This is the line we most wanted to draw. "ChatGPT visibility up 8 points" is the sentence that gets a report opened, and it is the sentence we are now unwilling to write unless the arithmetic supports it.
The model that answered is part of the record
Every scan stores which model ids answered for each engine. Providers swap models under stable product names, and a swap moves answers more than anything you can do to your site. When a jump coincides with a new model id, that is the explanation, and the record is there to show it.
What did not change
A reading counts as a mention only when the answer text names you; being linked is a separate measure, "cited", and it is only scored when the answer carried citations at all. A rank is reported only when the answer contained an explicit ranked list, labeled as the position in that one answer. A failed API call is excluded from every statistic rather than scored as invisibility. None of that is new; all of it is on the methodology page, which now describes the intervals too.
Our own numbers, with the bar
We publish our visibility live, measured by the same scanner customers get. The reading on 21 October, from ten prompts at three runs each (the last scan before the change):
| Engine | Visibility | 95% interval |
|---|---|---|
| Claude | 22% | ±26 |
| ChatGPT | 11% | ±16 |
| Gemini | 11% | ±16 |
Read plainly: we are a months-old domain in a category with incumbents, and with a ten-prompt panel we cannot distinguish 22% from 0% or from 45%. The bar is wider than the reading. That is not a flaw in the method; it is what the method is for. The old page showed "22%" and let you imagine it meant something. The next scan runs five draws, and a twenty-prompt panel would cut the width by about a third again. We would rather publish an honest ±26 and watch it shrink than a confident 22 that was never there.
The caveat we have not engineered away
Repetition fixes one kind of noise. It does not fix phrasing. The same buyer intent asked in different words can return different brands: a large vendor study of nearly 38,000 answers found mentions stable while the intent held, with a drop of about 2.4 points once the wording drifted; a preprint on paraphrase brittleness found much larger swings, with recommendation sets overlapping only 14–29% across natural paraphrases against 50–61% for plain reruns. The studies disagree on the size and agree on the shape: rankings move more than membership, and one wording is a sample of one.
The fix is a different prompt model, not a different statistic — a panel of phrasings per intent, with the intent as the cluster. We have not built that yet. Until we do, the best thing you can do is what the methodology already suggests: write your panel the way different customers would actually ask, not the same question ten ways.
Why this matters for the number you pay for
Every AI-visibility tool sells a number. The question to ask any of them, including us, is the one a scientist would ask: how many times did you ask, how far apart were the answers, and what would you have said if the move were noise? If a product cannot answer those three questions, its trend line is a random walk with a logo on it.
Ours now answers all three on the page, in the email, and in the data export. Run the free audit to see whether engines can read you; then track a panel of your customers' questions and watch the bars — not just the numbers — change.
Frequently asked questions
Why five runs and not ten? Ten would be better and the reliability curve says so. Five is the floor the published protocols ask for, and it is what fits on every plan at the current cost of a draw. Grounded draws are the expensive ones and we still take one per prompt; the four repeats cost tokens. We will raise the count when the plan economics allow, and the interval on the page will show you the improvement rather than our telling you about it.
My visibility went from 30% to 38% and the page says "within noise". Is the number wrong? No. Both numbers are the correct pooled rates for their scans. "Within noise" means that when we compare prompt by prompt, the differences do not agree enough to rule out the engines' own randomness. Keep the panel, keep scanning; a real shift shows up as a consistent lift across prompts and the label flips.
Why not lower the temperature so the answers are stable? Because your customers do not. The scan measures the product buyers use, at the settings they use it with. A stable number that describes a different product is a worse number, and zero temperature is not deterministic on production inference anyway.
Does the interval apply to the Google surface? Google is one draw per prompt by design: page one is the answer and does not re-roll within a scan. Its interval comes from the between-prompt spread alone, and the paired test works the same way across scans.
Where can I see the math? The statistics module is a single pure file with its own tests, and the methodology page states every rule in plain words. If you find a case where the interval or the verdict misleads, tell us: the engine is versioned, and corrections ship with a changelog entry.