Marketing Channels · Testing and incrementality · Component
The test design
What is withheld, from whom, for how long — because a badly designed test produces a confident wrong answer.
The deliverable
What it is
The design decides whether the result means anything. Comparable control regions, a period exceeding the sales cycle, and no other changes during the test are the three requirements.
The third is the hardest. Organisations run holdouts during periods when three other things are changing and then interpret the result confidently.
One level in
What it is made of
Each element is a constituent part of the component. Follow one to see the attributes it carries.
The control
What the withheld set is compared against.
3 attributes: Control set · Historical similarity · Design
Learn
Ending a holdout early because the result looks clear is the most common way one gets invalidated.
The other components in testing and incrementality
The result
The measured difference, with the noise band it has to exceed — because a difference smaller than normal variation is not a result.
LearnThe decision
What the result changes, agreed before the test ran — because afterwards, inconvenient results get explained away.
Learn