Use when a test has finished and you need to know whether the result means anything.
ab-test-result-reading.md
You are reviewing an A/B test result critically before anyone ships the winner.
Test: {{TEST_DESCRIPTION}}
Hypothesis as written before the test: {{HYPOTHESIS}}
Variant results: {{RESULTS_WITH_SAMPLE_SIZES}}
Runtime and traffic split: {{RUNTIME_AND_SPLIT}}
Primary metric declared in advance: {{PRIMARY_METRIC}}
Guardrail metrics: {{GUARDRAILS}}
Answer:
1. Was the test valid? Check sample size, runtime against full business cycles, split balance, sample ratio mismatch, and whether the primary metric was set before the test.
2. What does the primary metric result actually say? Give the observed difference, the uncertainty, and a plain sentence a non-analyst would understand.
3. Guardrails: did anything get worse? Say so even if the primary metric won.
4. Ship, do not ship, or run longer. One of those three, with the reason.
5. If someone points at a segment where the result looks stronger, explain why that is or is not evidence.
Rules:
- Do not call a result significant without stating the sample sizes and the metric it applies to.
- Treat any metric not declared in advance as exploratory and label it so.
- If the test ran for less than one full weekly cycle, say the result is unreliable regardless of the numbers.
- Never recommend shipping on a directional result alone. Say what it would cost to get certainty.
Replace each placeholder with your own detail. The more specific you are, the less the model invents.