Established model
A/B testing and controlled experiments
Ronald A. Fisher; adapted for the web by Ron Kohavi and others · 1935
Also known as Split testing, Randomised controlled experiments
Randomised assignment is the only method that separates what your change caused from what would have happened anyway.
Its place in the frameworkData Core›Performance Analysis
What it does
The logic is a century old and the application is recent. What the web experimentation literature added is a catalogue of the ways it goes wrong in practice — peeking at results, running many tests until one passes, and the consistent finding that the large majority of confidently expected improvements turn out to be nothing.
- Reach for it when
- Whenever a change can be given to some people and not others, and before believing any before-and-after comparison.
- Where it stops
- It answers narrow questions about small changes over short horizons. Long-term effects, changes that must ship to everyone, and anything with few users are outside it.
Ronald A. Fisher, The Design of Experiments, Oliver & Boyd, 1935; Ron Kohavi, Diane Tang & Ya Xu, Trustworthy Online Controlled Experiments, Cambridge University Press, 2020.
Why it sits at Performance Analysis
Turning numbers into understanding: starting from a question, choosing a method that can answer it, reading the result honestly, and doing something.
A model is only useful when you reach for it at the right moment. This one answers a question that arises here — so it is filed here, and nowhere else. These are the working areas it serves:
- The questionDecision-analysis practice on defining the decision before the data, and on the value of information as the test of whether analysis is worth doing.
- MethodThe distinction between observational and experimental design, and the conditions under which observational data supports a causal claim.
- Reading the resultWork on confounding, selection effects and the base-rate problems that make plausible business findings wrong.
- Acting on itEvidence-based management on the gap between findings and practice, and the closure of the loop from analysis to measured outcome.
What it touches elsewhere
Nothing in a business is decided on its own. A conclusion reached with this model at Performance Analysis lands in these other cores, whether or not anyone follows it there.
- KPI managementAnalysis usually starts because a measure moved, and the definition of that measure constrains what can be concluded.
- ReportingReporting says what happened; analysis says why, and the two are frequently confused.
- Goal CoreThe decision an analysis informs is usually about whether a goal is achievable or has been achieved.
- Market CoreAttribution and incrementality are analysis problems, and they are where causal claims most often exceed the evidence.
Filed at the same place
These answer questions that arise at Performance Analysis too. Where they disagree with this one, the disagreement is the useful part.
- Cohort analysisGroup people by when they arrived, then follow each group separately instead of averaging them together.
- The Ishikawa diagramWork backwards from an observed problem through the categories of cause that could produce it.
Elsewhere in Data Core
These are other people’s models, named here so you can go to the source and use them properly. The Omnigoal is not affiliated with their authors and is not endorsed by them; nothing of theirs is reproduced here — no canvas, no diagram, no wording. Each is described in our own words, with the originator credited, because the framework is a place to put thinking, not a replacement for the people who did it. Model names and trademarks belong to their respective owners and are used here only to refer to the work itself.
All 125 models