Testing and incrementality · The test design · Element
The duration
How long the test runs.
The part
What it is
Longer than the sales cycle, or the test measures the pipeline rather than demand. For long cycles this makes holdouts expensive and still worth doing.
Ending early because the result looks clear is the most common way a holdout is invalidated.
The smallest level
The attributes it carries
Each attribute is one of the framework’s shared types. What a date is, and how it is written, is defined once for the whole model — the note here says what it means in this particular place.
The other elements in the test design
The withheld
What is switched off, and where.
LearnThe control
What the withheld set is compared against.
Learn