CONVERSION

Pre launch experiment QA check

Use when a test is built and ready to launch and the setup has not been checked by anyone else.

pre-launch-experiment-qa-check.md

Download .md

You are checking an experiment before it goes live. You are not judging whether the idea is good, only whether the test will produce a result anyone can trust.

VARIANT DESCRIPTIONS AND WHAT CHANGES IN CODE: {{VARIANT_SPEC}}
TARGETING AND TRAFFIC ALLOCATION: {{TARGETING_RULES}}
HOW THE PRIMARY METRIC IS TRACKED: {{TRACKING_SETUP}}
DEVICES, BROWSERS AND LOGGED-IN STATES IN SCOPE: {{SCOPE}}

Produce a numbered checklist. Against each item give a verdict of Pass, Fail, or Cannot verify, plus the exact check to run:
1. Bucketing holds for the same user across sessions and devices
2. {{TARGETING_RULES}} cannot overlap another live test on the same page
3. Sample ratio: the split expected, and the point at which a mismatch should stop the test
4. The primary metric in {{TRACKING_SETUP}} fires once per conversion, not on reload or back navigation
5. The variant renders in every combination in {{SCOPE}}, including logged-in and returning visitors
6. No flicker or layout shift while the variant applies
7. Downstream systems that read the changed element still work: analytics, CRM field mapping, email triggers
8. Internal traffic and bots are excluded
9. A stopping rule and a duration agreed in writing before launch

Then name the three items most likely to invalidate this particular test given {{VARIANT_SPEC}}, and describe what a broken version would look like in the data.

Constraints: do not pass an item because it is usually fine. Anything you cannot confirm from what I supplied is Cannot verify. No em dashes.

Fill in before running

Replace each placeholder with your own detail. The more specific you are, the less the model invents.

{{VARIANT_SPEC}}
{{TARGETING_RULES}}
{{TRACKING_SETUP}}
{{SCOPE}}

Getting a better result

Run it against the built variant, not the ticket, since what shipped and what was specified often differ.
The sample ratio check is the one that saves whole weeks; agree the stop threshold before traffic starts.
Keep the completed checklist with the readout, so a surprising result can be re-examined.

Questions about this prompt

When do I QA the setup rather than review the idea?

After the variant is built and before traffic starts, when nobody else has checked it. This judges only whether the test can produce a result anyone can trust; the hypothesis prompt handles whether the idea deserves the slot. Run it against the built variant rather than the ticket, since what shipped and what was specified often differ.

What do I need in front of me before running it?

What actually changes in code, the targeting and traffic allocation, how the primary metric is tracked, and the devices, browsers and logged-in states in scope. Anything you cannot describe comes back as Cannot verify rather than Pass, so vague inputs give you a checklist of unknowns, which is at least an accurate one.

What comes back, and which part is worth acting on?

Nine checks with Pass, Fail or Cannot verify and the exact check to run, then the three most likely to invalidate this particular test and what a broken version would look like in the data. That last description is what lets you spot a failure while the test is live rather than in the readout.

What is the mistake that costs me here?

Passing an item because it is usually fine. The prompt refuses to and so should you. The sample ratio check is the one that saves whole weeks, so agree the mismatch threshold that stops the test before traffic starts, and file the completed checklist with the readout so a surprising result can be re-examined.