QuQi
ANALYSE ET RAPPORTS

Incrementality holdout design

Use when a channel claims credit you doubt and you want to test it rather than argue about models.

incrementality-holdout-design.md
Télécharger le .md
You are designing a holdout test to measure incremental effect. You are not evaluating a finished test and you are not defending the current attribution model.

Activity to test: {{ACTIVITY_TO_TEST}}
Current reported performance and spend: {{CURRENT_PERFORMANCE}}
Units we can split on, such as regions, stores, or audiences: {{SPLITTABLE_UNITS}}
Baseline volume and its week to week variation: {{BASELINE_VARIATION}}

Output:
1. The design: what is held out, for how long, and how units from {{SPLITTABLE_UNITS}} are assigned to each group. Name the assignment method.
2. A power estimate: the smallest effect this design could detect given {{BASELINE_VARIATION}}, with the arithmetic shown, and how much longer it would need to run to halve that.
3. A table: Threat to validity | How the design handles it | Residual risk. Cover spillover between units, seasonality, other activity running at the same time, and anything in {{CURRENT_PERFORMANCE}} suggesting the baseline is already trending.
4. Decided before launch: primary metric, guardrails, and the exact comparison you will run at the end.
5. The cost of the holdout in forgone {{ACTIVITY_TO_TEST}} results, and what finding would justify paying it.

Rules:
- If the design cannot detect an effect small enough to matter, say so and stop rather than proposing a test that will end ambiguous.
- Do not present an attribution difference as an incrementality result.
- Name in advance what would make you stop the test early, and what would not.

À remplir avant de lancer

Remplacez chaque espace réservé par vos propres détails. Plus vous êtes précis, moins le modèle invente.

  • {{ACTIVITY_TO_TEST}}
  • {{CURRENT_PERFORMANCE}}
  • {{SPLITTABLE_UNITS}}
  • {{BASELINE_VARIATION}}

Obtenir un meilleur résultat

  1. Split on geography where you can; user level holdouts leak through shared devices and households.
  2. Agree the analysis with whoever owns the channel before launch, not after the result arrives.
  3. A test that only detects a 40 percent effect is not worth running on a channel you think does 10.