Personalization · Whether it works · Component
The control
The non-personalised group held back for comparison — permanently, not just at launch.
The deliverable
What it is
Without a control there is no way to know what the personalisation contributes, and after launch the question becomes unanswerable.
A permanent small holdout costs a little revenue and answers the question continuously, including as the model degrades.
One level in
What it is made of
Each element is a constituent part of the component. Follow one to see the attributes it carries.
The holdout
Who sees the non-personalised version.
3 attributes: Holdout size · Assignment · Running since
LearnThe cost
What holding back the control costs.
3 attributes: Cost of holdout · Of total · Accepted
Learn
Where the holdout costs nothing, the personalisation is producing nothing. That is the answer.
The other components in whether it works
The customer measure
Whether it improved anything for the customer, not only for conversion — because the two can move in opposite directions.
LearnThe decision
What was kept, changed or removed as a result — because untested personalisation is a permanent cost with no established benefit.
Learn