Omni Core · UI and UX Optimization · Module
Testing changes
Establishing whether a change improved anything, on a measure agreed beforehand, with enough volume to be readable.
The idea
How it works
The honest finding of most testing programmes is that the majority of changes have no measurable effect. Knowing that changes how much confidence a shipped change should carry.
Underpowered tests are worse than no tests, because they produce a result that looks like evidence. A test that cannot detect the effect size that matters should not be run.
Working with it
In practice
- 01
Agree the measure before building
And the size of change that would count. Deciding afterwards means the winning metric is chosen from the results.
- 02
Check you have the volume
If the traffic cannot detect a meaningful effect, the test will produce noise presented as a finding.
- 03
Run it to the planned end
Stopping when the result looks good is the most common way a test is invalidated.
- 04
Record the null results
They are the majority and they are what keeps expectations calibrated.
One level in
The components of testing changes
A component is something that exists afterwards which did not exist before — a deliverable or a mechanism, not an intention.
A test that cannot detect the effect size that matters should not be run. It will produce noise presented as evidence.
The other modules in ui and ux optimization
Understanding use
How people actually use the thing, established by watching rather than by inferring from analytics.
LearnDesign decisions
What is being changed and why, with the reasoning recorded so a later change does not undo it for reasons nobody remembers.
LearnAccessibility
Whether the interface works for people using assistive technology, keyboards only, or with impaired vision or motor control.
Learn