ANALYTICS & REPORTING

Reading a metric distribution

Use when a decision is resting on an average and you suspect the shape underneath is uneven.

metric-distribution-reading.md

Download .md

You are examining the shape of a metric, not its average. You are not recommending product changes.

Metric and how it is calculated: {{METRIC_DEFINITION}}
Raw values, percentile summary, or bucketed counts: {{DISTRIBUTION_DATA}}
The average currently being quoted: {{QUOTED_AVERAGE}}
The decision this number is feeding: {{DECISION_AT_STAKE}}

Output:
1. A table: Statistic | Value | What it tells us here. Include count, mean, median, the 25th, 75th, 90th and 99th percentiles, the share of zeros, and the share of the total contributed by the top 10 percent. Write "cannot compute" for anything {{DISTRIBUTION_DATA}} does not support.
2. The shape in one sentence: roughly symmetrical, right skewed, bimodal, or dominated by zeros.
3. How far {{QUOTED_AVERAGE}} sits from the median, and what that gap changes about {{DECISION_AT_STAKE}}.
4. If the shape is bimodal or zero heavy, the split that most likely explains it and the cut of the data that would confirm it.
5. The statistic to quote instead, plus one sentence to say when someone asks why the reported number changed.

Rules:
- Do not drop outliers. Name the threshold and give the figures with and without them.
- If {{METRIC_DEFINITION}} already averages something per user or per session, say so. An average of averages is a different number and usually the wrong one.
- No distribution jargon without a plain definition in the same sentence.

Fill in before running

Replace each placeholder with your own detail. The more specific you are, the less the model invents.

{{METRIC_DEFINITION}}
{{DISTRIBUTION_DATA}}
{{QUOTED_AVERAGE}}
{{DECISION_AT_STAKE}}

Getting a better result

Paste deciles rather than 50,000 raw rows; the percentiles carry almost all of the information.
State the share of zeros first for any metric where not acting is a common outcome.
Put the replacement statistic into the standing report, otherwise the mean comes back next month.

Questions about this prompt

When do I use this rather than quoting the average?

When a decision is resting on a mean and you suspect the shape underneath is uneven. Averages are fine for symmetrical data and misleading for nearly everything else, which includes most revenue, order value and time on task figures. This diagnoses shape only and stays out of product recommendations.

What do I need in front of me?

The metric definition including how it is calculated, deciles or a percentile summary rather than fifty thousand raw rows, the average currently being quoted, and the decision the number is feeding. Percentiles carry nearly all the information. Name the decision, or section three has nothing to weigh the mean to median gap against.

What comes back?

A statistics table with count, mean, median, four percentiles, the share of zeros and the share of the total from the top ten percent, a one sentence verdict on shape, how far the quoted average sits from the median, and the statistic to quote instead. That replacement statistic is the deliverable.

What is the mistake that costs me here?

Not writing the replacement statistic into the standing report. The mean returns next month because it is what the tool shows by default. Watch for an average of averages as well: where the metric already averages per user or per session, averaging again gives a different number and usually the wrong one.