# Incrementality holdout design

> Use when a channel claims credit you doubt and you want to test it rather than argue about models.

## املأها قبل التشغيل

- `{{ACTIVITY_TO_TEST}}`
- `{{CURRENT_PERFORMANCE}}`
- `{{SPLITTABLE_UNITS}}`
- `{{BASELINE_VARIATION}}`

## البرومبت

```
You are designing a holdout test to measure incremental effect. You are not evaluating a finished test and you are not defending the current attribution model.

Activity to test: {{ACTIVITY_TO_TEST}}
Current reported performance and spend: {{CURRENT_PERFORMANCE}}
Units we can split on, such as regions, stores, or audiences: {{SPLITTABLE_UNITS}}
Baseline volume and its week to week variation: {{BASELINE_VARIATION}}

Output:
1. The design: what is held out, for how long, and how units from {{SPLITTABLE_UNITS}} are assigned to each group. Name the assignment method.
2. A power estimate: the smallest effect this design could detect given {{BASELINE_VARIATION}}, with the arithmetic shown, and how much longer it would need to run to halve that.
3. A table: Threat to validity | How the design handles it | Residual risk. Cover spillover between units, seasonality, other activity running at the same time, and anything in {{CURRENT_PERFORMANCE}} suggesting the baseline is already trending.
4. Decided before launch: primary metric, guardrails, and the exact comparison you will run at the end.
5. The cost of the holdout in forgone {{ACTIVITY_TO_TEST}} results, and what finding would justify paying it.

Rules:
- If the design cannot detect an effect small enough to matter, say so and stop rather than proposing a test that will end ambiguous.
- Do not present an attribution difference as an incrementality result.
- Name in advance what would make you stop the test early, and what would not.
```

## كيف تحصل على نتيجة أفضل

- Split on geography where you can; user level holdouts leak through shared devices and households.
- Agree the analysis with whoever owns the channel before launch, not after the result arrives.
- A test that only detects a 40 percent effect is not worth running on a channel you think does 10.

---

من مكتبة برومبتات QuQi - https://www.quqi.io/ar/prompts/incrementality-holdout-design