This is the calculation and reporting contract behind every AirPulse research page. Read it with the 30,504-response stability study and the prompt-selection methodology.
1. Define the measurement question before collecting data
State the decision the study supports. “Are we visible in AI?” is too broad. “How often is our brand mentioned for these evaluation-stage prompts on ChatGPT and Perplexity during this four-week window?” is usable.
Lock these fields before the baseline:
- target brand and approved aliases;
- exact prompt text and locale;
- prompt intent and buyer stage;
- answer engine and surface;
- run schedule and minimum valid runs;
- mention and citation matching rules;
- start and end dates;
- parser, roster and measurement version.
2. Use a prompt-engine cell as the unit
A prompt-engine cell is one exact prompt paired with one answer engine. “Best AI visibility tools for B2B SaaS” on ChatGPT is one cell. The same prompt on Perplexity is another.
Do not pool engines before calculating the primary rates. In AirPulse's frozen cohort, consecutive mention flip rates ranged from 6.2% on Gemini to 14.7% on Google AI. The gap may partly reflect the cohort, but it is large enough that a hidden average misleads.
3. Freeze the prompt roster
Every prompt gets a stable identifier. Fixing spelling after the first run creates a new version. Adding a product qualifier changes the question and needs a new prompt or a declared roster revision.
Store:
| Field | Required value |
|---|---|
| Prompt ID | Stable internal identifier |
| Prompt text | Exact text sent to the engine |
| Locale | Language and market qualifiers |
| Intent | Discovery, evaluation, comparison, validation or post-purchase |
| Buyer stage | Problem, category, shortlist, vendor or proof |
| Version | Roster version active for the run |
Selection is covered in Which AI search prompts should your brand track? Label generated suggestions as suggestions; keep observed questions from customers, sales calls or Search Console distinguishable.
4. Schedule repeated runs
Start with at least seven valid runs for brand detection and eight for source coverage per prompt-engine cell, spread across two to four weeks. These are practical floors from external research, not a theorem.
Store failed calls separately. A timeout, refusal, empty answer or parser failure is not “brand not mentioned.” Report valid runs and excluded runs.
5. Preserve one evidence unit per run
Every valid run should retain:
- prompt ID and roster version;
- engine, surface and available model identifier;
- request and response timestamps;
- raw answer text;
- parsed brand mention and matched span;
- citation URLs and normalised domains;
- available search or retrieval results;
- generated search queries where the engine exposes them;
- parser version and validation flags.
Generated queries, search results and citations are different fields. A search result shows what may have been retrieved. A citation shows what the answer exposed. A generated query shows how the system searched.
6. Calculate the primary outcomes
Mention rate
mention rate = runs with brand mention / valid runs, within one prompt-engine-window scope.
Matching rules cover approved brand names and aliases and reject unrelated words. If a reviewer changes a match, record the override and reason.
Own-domain citation rate
own-domain citation rate = runs citing the brand's domain / valid runs, where a cited URL's registrable domain belongs to the tracked brand.
Normalise scheme, www, fragments and tracking parameters. Preserve the original URL as evidence.
Third-party citation rate
Runs that cite a separate domain which mentions, validates, compares or competes with the brand. Keep the classification transparent; one source can play different roles for different questions.
Source coverage and stability
Source coverage is the distinct set of cited domains or URLs observed across runs. For consecutive sets, AirPulse uses Jaccard similarity:
Jaccard similarity = intersection / union
Report the distribution, or at least median and mean. The frozen study found a mean of 0.396 and a median of 0.333. One minus Jaccard is not the probability that each URL disappears.
7. Show uncertainty
For a binary rate, show numerator, denominator and a binomial confidence interval. Prefer a Wilson interval when samples are small or rates are near zero or one.
Overlapping intervals do not prove “no difference”; non-overlap alone does not prove a causal effect. For a content trial, compare a preregistered target set with matched control prompts and keep the time series.
Quantifying Uncertainty in AI Visibility explains why single-run point estimates give a false sense of precision.
8. Separate the four doors
An AI outcome can fail at four different stages:
- Access: can the relevant crawler or user-requested fetcher reach the page?
- Discovery/indexing: is the URL known to search and retrieval systems?
- Retrieval: is the page selected as evidence for this question?
- Answer selection: does the final answer mention the brand or cite the page?
Record these stages separately. OpenAI distinguishes OAI-SearchBot from GPTBot (publisher guidance), Perplexity distinguishes PerplexityBot from Perplexity-User (crawler documentation), and Google separates crawling, indexing and serving (Search documentation). Access does not guarantee a mention or citation.
9. Version every material change
Create a new measurement version when any of these changes:
- prompt roster or wording;
- engine or surface definition;
- model identifier where exposed;
- mention or citation parser;
- brand alias list;
- source URL normalisation;
- retry or failure handling;
- run cadence;
- page intervention being tested.
Do not draw a before/after conclusion across an undeclared measurement boundary. If a boundary cannot be reconstructed, label the series non-comparable.
10. Use controls for intervention claims
To test a page change, freeze target prompts and matched control prompts before publication. Collect a baseline, record the intervention date, then continue the same schedule. Do not rewrite prompts after launch.
The result can be:
- positive: the target moved more than controls under the registered rule;
- null: the page became retrievable but target outcomes did not materially improve;
- inconclusive: access, sample size or contamination prevents a decision.
A null result is publishable. A single citation is an observation until it repeats under the plan. The full procedure (preregistration, matched controls, checkpoints, decision rule) is the trial protocol.
11. Minimum weekly report
| Field | Example format |
|---|---|
| Prompt | Stable ID plus readable text |
| Engine | Separate row per engine |
| Window | Start date-end date, timezone |
| Valid runs | 7/8 completed |
| Mention rate | 4/7 (57.1%), with interval |
| Own-domain citation rate | 2/7 (28.6%), with interval |
| Third-party citation rate | 3/7 (42.9%) |
| Source stability | Median Jaccard plus set coverage |
| Access state | Allowed, fetched, indexed/retrievable, unknown |
| Version | Prompt roster, parser and measurement version |
The executive summary can be shorter; the underlying table stays available.
12. Privacy and tenant isolation
Public exports are aggregate-only unless a customer has approved a case study. Remove names, tenant IDs, private prompts, raw answers, API keys, account-state details and URLs that could identify a private programme. Generate downloads from a reviewed allow-list of fields, not by deleting columns after export.