QuQi
CONVERSION

Pre launch experiment QA check

Use when a test is built and ready to launch and the setup has not been checked by anyone else.

pre-launch-experiment-qa-check.md
.md herunterladen
You are checking an experiment before it goes live. You are not judging whether the idea is good, only whether the test will produce a result anyone can trust.

VARIANT DESCRIPTIONS AND WHAT CHANGES IN CODE: {{VARIANT_SPEC}}
TARGETING AND TRAFFIC ALLOCATION: {{TARGETING_RULES}}
HOW THE PRIMARY METRIC IS TRACKED: {{TRACKING_SETUP}}
DEVICES, BROWSERS AND LOGGED-IN STATES IN SCOPE: {{SCOPE}}

Produce a numbered checklist. Against each item give a verdict of Pass, Fail, or Cannot verify, plus the exact check to run:
1. Bucketing holds for the same user across sessions and devices
2. {{TARGETING_RULES}} cannot overlap another live test on the same page
3. Sample ratio: the split expected, and the point at which a mismatch should stop the test
4. The primary metric in {{TRACKING_SETUP}} fires once per conversion, not on reload or back navigation
5. The variant renders in every combination in {{SCOPE}}, including logged-in and returning visitors
6. No flicker or layout shift while the variant applies
7. Downstream systems that read the changed element still work: analytics, CRM field mapping, email triggers
8. Internal traffic and bots are excluded
9. A stopping rule and a duration agreed in writing before launch

Then name the three items most likely to invalidate this particular test given {{VARIANT_SPEC}}, and describe what a broken version would look like in the data.

Constraints: do not pass an item because it is usually fine. Anything you cannot confirm from what I supplied is Cannot verify. No em dashes.

Vor dem Start ausfüllen

Ersetzen Sie jeden Platzhalter durch Ihre eigenen Angaben. Je genauer Sie sind, desto weniger erfindet das Modell.

  • {{VARIANT_SPEC}}
  • {{TARGETING_RULES}}
  • {{TRACKING_SETUP}}
  • {{SCOPE}}

So wird das Ergebnis besser

  1. Run it against the built variant, not the ticket, since what shipped and what was specified often differ.
  2. The sample ratio check is the one that saves whole weeks; agree the stop threshold before traffic starts.
  3. Keep the completed checklist with the readout, so a surprising result can be re-examined.