Marketing Channels · Testing and incrementality · Component
The result
The measured difference, with the noise band it has to exceed — because a difference smaller than normal variation is not a result.
The deliverable
What it is
The difference between test and control has to exceed the difference those regions normally show against each other. Without that band, any result can be read as significant.
Inconclusive is a common and legitimate outcome. Recording it as such rather than as a weak positive is what keeps the method honest.
One level in
What it is made of
Each element is a constituent part of the component. Follow one to see the attributes it carries.
The difference
What the test and control actually produced.
3 attributes: Difference · Volume · Period
LearnThe noise band
How much those sets normally differ anyway.
3 attributes: Normal variation · Baseline period · Exceeded
LearnThe verdict
What the test concluded.
3 attributes: Verdict · Estimated incrementality · Confidence
Learn
A difference smaller than the normal variation between your test and control regions is not a result.
The other components in testing and incrementality
The test design
What is withheld, from whom, for how long — because a badly designed test produces a confident wrong answer.
LearnThe decision
What the result changes, agreed before the test ran — because afterwards, inconvenient results get explained away.
Learn