UI and UX Optimization · Testing changes · Component
The power
Whether the available volume can detect the effect that matters — because an underpowered test produces noise presented as evidence.
The deliverable
What it is
Most organisations do not have the traffic to detect the effects they are testing for. Running the test anyway produces a result that is read as a finding.
The check is arithmetic and takes minutes: given the baseline rate, the effect size that matters and the available volume, can this test detect it?
One level in
What it is made of
Each element is a constituent part of the component. Follow one to see the attributes it carries.
The calculation
Whether the test can detect the effect.
3 attributes: Baseline rate · Available volume · Required duration
Learn
Stopping a test when the result looks good is the most common way to invalidate it.
The other components in testing changes
The measure
What the change is meant to improve, agreed in advance — because deciding afterwards means the winning metric is chosen from the results.
LearnThe result
What happened, including the nulls — because the nulls are the majority and are what keeps expectations calibrated.
Learn