Home»What»Data Core»Performance Analysis

Data Core · Object

Performance Analysis

Turning numbers into understanding: starting from a question, choosing a method that can answer it, reading the result honestly, and doing something.

The term

What it is

Analysis that does not start from a question produces a report. The useful sequence is decision first, question second, method third — and it is routinely run in reverse, starting from whatever data is available.

The recurring error is treating correlation as explanation. Most business data is observational, which means the honest reading of almost any finding includes several alternative explanations.

The output of analysis is a decision or it is nothing. An analysis that is presented, agreed with and changes no behaviour has cost time and produced a document.

Why it earns a place

What goes wrong without it

01

Starting from the data produces findings nobody needed

Analysis begun from what is available yields interesting patterns rather than answers, and interesting patterns are rarely actionable.

02

Most business data cannot support a causal claim

Observational data shows what happened together. Establishing what caused what requires either an experiment or a much more careful argument.

03

The presentation is where the honesty is usually lost

Caveats present in the analysis disappear in the summary, and the summary is what people act on.

One level in

The modules within performance analysis

Four working areas, in the order that produces useful work: the question, the method, the reading, and what actually changes.

  1. The question

    What decision the analysis is meant to inform, and what answer would change it. Analysis with no decision attached is exploration, which is legitimate and should be labelled.

    Learn

  2. Method

    How the question will be answered, and whether that method can actually answer it. Most business questions are causal and most available methods are not.

    Learn

  3. Reading the result

    What the finding does and does not say, including the alternative explanations. The step where confidence is either calibrated or inflated.

    Learn

  4. Acting on it

    What changed as a result, and whether the change worked. Without this the analysis was a document rather than an input.

    Learn

Across the framework

What it touches

  • KPI managementAnalysis usually starts because a measure moved, and the definition of that measure constrains what can be concluded.
  • ReportingReporting says what happened; analysis says why, and the two are frequently confused.
  • Goal CoreThe decision an analysis informs is usually about whether a goal is achievable or has been achieved.
  • Market CoreAttribution and incrementality are analysis problems, and they are where causal claims most often exceed the evidence.

Beyond the framework

Models worth knowing here

The Omnigoal says where this belongs and what it touches. It does not tell you how to think about it — other people have done that, and done it well. These are theirs.

  1. A/B testing and controlled experiments

    Ronald A. Fisher; adapted for the web by Ron Kohavi and others · 1935

    Also known as Split testing, Randomised controlled experiments

    Randomised assignment is the only method that separates what your change caused from what would have happened anyway.

    The logic is a century old and the application is recent. What the web experimentation literature added is a catalogue of the ways it goes wrong in practice — peeking at results, running many tests until one passes, and the consistent finding that the large majority of confidently expected improvements turn out to be nothing.

    Reach for it when
    Whenever a change can be given to some people and not others, and before believing any before-and-after comparison.
    Where it stops
    It answers narrow questions about small changes over short horizons. Long-term effects, changes that must ship to everyone, and anything with few users are outside it.

    Ronald A. Fisher, The Design of Experiments, Oliver & Boyd, 1935; Ron Kohavi, Diane Tang & Ya Xu, Trustworthy Online Controlled Experiments, Cambridge University Press, 2020.

  2. Cohort analysis

    From demography, adopted in business analytics · 1970s

    Group people by when they arrived, then follow each group separately instead of averaging them together.

    An aggregate figure mixes people who have been customers for three years with people who arrived last week, and the mixture hides the trend. Following each intake on its own shows whether what you changed made later arrivals behave differently — which is the question, and one that no total can answer.

    Reach for it when
    Whenever retention, repeat purchase or usage over time is being judged from a single overall number.
    Where it stops
    Cohorts differ for reasons other than what you changed — the season they arrived in, the campaign that brought them. It shows a difference without explaining it.

    Standard demographic method; brought into business analytics through subscription and web practice.

  3. The Ishikawa diagram

    Kaoru Ishikawa · 1968

    Also known as Fishbone diagram, Cause-and-effect diagram

    Work backwards from an observed problem through the categories of cause that could produce it.

    Branches for the standard families — people, method, machine, material, measurement, environment — each expanded until the causes are specific enough to test. Its value is coverage under pressure: a group looking for a cause converges on the first plausible one within minutes, and the categories force the other five to be considered.

    Reach for it when
    When performance has moved and the explanation was agreed on before anyone looked.
    Where it stops
    It generates candidate causes; it does not weigh them. Every branch is a hypothesis and the diagram gives no way to tell which one is true.

    Kaoru Ishikawa, Guide to Quality Control, JUSE, 1968.

These are other people’s models, named here so you can go to the source and use them properly. The Omnigoal is not affiliated with their authors and is not endorsed by them; nothing of theirs is reproduced here — no canvas, no diagram, no wording. Each is described in our own words, with the originator credited, because the framework is a place to put thinking, not a replacement for the people who did it. Model names and trademarks belong to their respective owners and are used here only to refer to the work itself.

Every model in the framework, and where each one belongs

Ask what decision the analysis would change. Where there is no answer, it is exploration — which is fine, and should be labelled.

The other objects in the Data Core

KPI management

Establish and monitor key performance indicators (KPIs) for different business areas. This helps track progress towards strategic objectives and ensures alignment with overall goals. By setting clear KPIs, businesses can measure performance accurately, identify areas needing improvement, and make informed decisions. This process involves selecting relevant KPIs, regularly reviewing them, and adjusting strategies based on performance data to drive continuous improvement and achieve desired outcomes.

Learn

Data Collection

Ensure accurate and consistent data collection across all business operations. Systematically gathering data from various sources provides comprehensive insights that inform decision-making and strategic planning.

Learn

Reporting

Generate detailed reports to communicate performance metrics to stakeholders. These reports should provide a clear overview of business performance, facilitating informed decision-making and ensuring transparency.

Learn

Data Governance

Establish policies and procedures for data management, ensuring data quality, security, and compliance. Robust governance frameworks safeguard data integrity and adhere to regulatory standards.

Learn
Back to the Data Core