AI search KPIs: what to measure, and how many times to measure it

Why don't traditional SEO metrics work for AI search?

Because three of the four quantities conventional SEO reporting is built on do not exist in a generative answer.

There is no rank. A generative engine composes one answer from several retrieved passages. There is no ordered list, so there is no position one. Order of mention inside an answer is sometimes recorded as a position-weighted score, but it is not a ranking and it moves between runs of the same prompt.

There are no impressions. No engine publishes how often an answer containing your brand was shown. Search Console reports Google's AI surfaces inconsistently and nothing equivalent exists for ChatGPT, Perplexity or Claude.

There is no click-through rate on an answer the user never clicks. The whole design intent of an AI answer is that the user does not need to visit the source.

iPullRank calls the result the Measurement Chasm: “A user's query might retrieve passages from your site, merge them with content from other sources, and synthesize an answer, but unless that answer includes a citation and the user clicks it, you have no direct evidence of your involvement.”

The fourth quantity, sessions, does survive — but it is small enough to be statistically thin by construction, which is the first reason you need channel metrics rather than traffic metrics as the primary KPI.

How many times must you ask before a number means anything?

A mention rate is a binomial proportion, so the arithmetic is settled and borrowed directly from survey statistics. At a true rate near 30%, ten runs give you a 95% interval of roughly ±28 percentage points. The number you are reporting is almost entirely noise.

Every one of the channel metrics above is a proportion: some number of sampled answers out of a total. That makes the precision of the estimate a function of one thing you control — how many answers you sampled.

The half-width of a 95% confidence interval for a proportion is approximately 1.96 × √(p(1−p) / n). At p = 0.3, which is a realistic mention rate for a mid-market brand in its own category, that produces this:

1. n = 10 ±28.4
2. n = 30 ±16.4
3. n = 50 ±12.7
4. n = 100 ±9.0
5. n = 250 ±5.7
6. n = 400 ±4.5

Normal approximation to the binomial, α = 0.05. Percentage points. Precision improves with the square root of n, so quadrupling the runs halves the interval.

What this means for your prompt set

Observations are prompts × runs × engines. A practical floor for portfolio-level reporting is 50 prompts, run 5 times each, per engine, per period — 250 observations, giving roughly ±6 percentage points.

The consequence most teams have not absorbed: individual prompts can almost never be reported on their own. One prompt run five times gives you five observations and an interval of roughly ±40 points. A dashboard that shows per-prompt movement week on week is showing you noise in a presentation layer.

What counts as a real change?

Detecting a difference between two periods is harder than estimating one period, because both estimates carry error. As a working rule, the minimum detectable change is roughly 1.4 times the half-width of a single period's interval.

At 250 observations per period, that puts the minimum detectable effect at about 8 percentage points. A move from 30% to 35% is, on that sample, no detected change — and should be reported in exactly those words rather than as a 5-point gain.

Three conditions that must be frozen

1. The prompt set. Adding or removing prompts changes the population being sampled, which invalidates the comparison. Version it, and when you revise it, report the old and new sets in parallel for one period.
2. The competitor set. Share of voice has a denominator. Changing who is in it changes the metric without anything changing in the world.
3. The model version, where it is exposed. A model update can move every number overnight. Record what you queried, so a step change can be attributed to the engine rather than to your work.