Use when a test is built and ready to launch and the setup has not been checked by anyone else.
pre-launch-experiment-qa-check.md
You are checking an experiment before it goes live. You are not judging whether the idea is good, only whether the test will produce a result anyone can trust.
VARIANT DESCRIPTIONS AND WHAT CHANGES IN CODE: {{VARIANT_SPEC}}
TARGETING AND TRAFFIC ALLOCATION: {{TARGETING_RULES}}
HOW THE PRIMARY METRIC IS TRACKED: {{TRACKING_SETUP}}
DEVICES, BROWSERS AND LOGGED-IN STATES IN SCOPE: {{SCOPE}}
Produce a numbered checklist. Against each item give a verdict of Pass, Fail, or Cannot verify, plus the exact check to run:
1. Bucketing holds for the same user across sessions and devices
2. {{TARGETING_RULES}} cannot overlap another live test on the same page
3. Sample ratio: the split expected, and the point at which a mismatch should stop the test
4. The primary metric in {{TRACKING_SETUP}} fires once per conversion, not on reload or back navigation
5. The variant renders in every combination in {{SCOPE}}, including logged-in and returning visitors
6. No flicker or layout shift while the variant applies
7. Downstream systems that read the changed element still work: analytics, CRM field mapping, email triggers
8. Internal traffic and bots are excluded
9. A stopping rule and a duration agreed in writing before launch
Then name the three items most likely to invalidate this particular test given {{VARIANT_SPEC}}, and describe what a broken version would look like in the data.
Constraints: do not pass an item because it is usually fine. Anything you cannot confirm from what I supplied is Cannot verify. No em dashes.
استبدل كل فراغ بتفصيلة من عندك. كلما كنت أدقّ، قلّ ما يخترعه النموذج.