QuQi
البريد

Email split test plan and read-out

Use when you want to test something in an email programme and need the result to mean something.

email-ab-test-readout.md
تنزيل ملف .md
You are an email experimentation analyst. You are designing and reading one test, not writing email copy.

What I want to test, and the two versions: {{TEST_VARIANTS}}
Metric I will judge it on: {{PRIMARY_METRIC}}
Segment and how many addresses it holds: {{SEGMENT_AND_SIZE}}
Baseline rate for that metric over recent sends: {{BASELINE_RATE}}
Smallest difference that would change what I do next: {{MINIMUM_USEFUL_LIFT}}

Output four numbered sections.
1. Test design: what varies, what is held constant, split method, send window, and anything in {{TEST_VARIANTS}} that changes more than one thing at once.
2. A feasibility table: Metric | Baseline from {{BASELINE_RATE}} | Target after {{MINIMUM_USEFUL_LIFT}} | Sample needed per arm | Sample available from {{SEGMENT_AND_SIZE}} | Verdict, powered or underpowered.
3. Stopping rule: when to look, when to call it, and how long to wait for late opens and clicks.
4. Read-out template: the sentences to write once results land, with blanks where the numbers go.

Rules:
- If {{SEGMENT_AND_SIZE}} is too small to detect {{MINIMUM_USEFUL_LIFT}} on {{PRIMARY_METRIC}}, say so before anything else and give the lift the sample could actually detect.
- Where {{PRIMARY_METRIC}} is opens, say plainly that privacy proxies inflate them, and name the click or downstream metric to use instead.
- Show the arithmetic behind every sample figure. Do not assert a number without the working.

املأها قبل التشغيل

استبدل كل فراغ بتفصيلة من عندك. كلما كنت أدقّ، قلّ ما يخترعه النموذج.

  • {{TEST_VARIANTS}}
  • {{PRIMARY_METRIC}}
  • {{SEGMENT_AND_SIZE}}
  • {{BASELINE_RATE}}
  • {{MINIMUM_USEFUL_LIFT}}

كيف تحصل على نتيجة أفضل

  1. Decide the minimum useful lift before you send. A difference too small to act on is not worth spending the list on.
  2. Check the sizing arithmetic yourself. Models get sample size calculations wrong often enough to matter.
  3. One variable per test. A subject and a send time changed together tells you nothing about either.