SEO

SEO split test design

Use when someone wants to roll a template change sitewide and you want evidence it works first.

seo-split-test-design.md

Download .md

You are designing a measurable test for a proposed SEO change. You are not implementing the change.

The change being proposed: {{PROPOSED_CHANGE}}
Template affected and how many URLs use it: {{TEMPLATE_SCOPE}}
Per URL baseline, weekly, as far back as you have (clicks, impressions, position): {{BASELINE_DATA}}
What the team can and cannot deploy: {{DEPLOY_CONSTRAINTS}}

Output six numbered sections.

1. The hypothesis in one sentence: what {{PROPOSED_CHANGE}} alters, the metric it should move, the direction, and the mechanism.
2. Design table: Element | Choice | Reason. Cover the unit of assignment, how {{TEMPLATE_SCOPE}} is split, control group size, matching on baseline traffic, the primary metric, the measurement window and how seasonality is handled.
3. Whether {{TEMPLATE_SCOPE}} and the variance visible in {{BASELINE_DATA}} are enough to read a result worth acting on. If not, say what would be, then give a before and after design with its weaknesses named.
4. Contamination risks: internal links between test and control, sitewide releases, indexing lag, other work landing in the window.
5. Stop rules, including the result that means do not roll out.
6. What to log on day one so the analysis is possible later.

Constraints:
- Do not produce a significance figure or a power calculation from numbers absent from {{BASELINE_DATA}}.
- If {{DEPLOY_CONSTRAINTS}} rules out a control group, say plainly that the test is not causal.

Fill in before running

Replace each placeholder with your own detail. The more specific you are, the less the model invents.

{{PROPOSED_CHANGE}}
{{TEMPLATE_SCOPE}}
{{BASELINE_DATA}}
{{DEPLOY_CONSTRAINTS}}

Getting a better result

Match test and control on baseline traffic rather than splitting at random, since a few URLs carry most of the clicks.
Freeze other work on that template for the window, or you will never attribute the result.
Agree the stop rules with whoever wants the change before you start, not after the first bad week.

Questions about this prompt

When is a test worth designing rather than just shipping the change?

When the template carries enough URLs to split and being wrong is expensive. Shipping and watching the graph is a before and after with no control, so seasonality, a competitor or an update can all wear the credit. SEO split testing is itself contested, and on a small template it is not worth pretending.

What do I need before designing one?

The change stated precisely, how many URLs use the template, per URL weekly baselines for clicks, impressions and position going back as far as you have, and what the team can actually deploy. The baseline history is what shows the variance; without it nobody can say whether a result would be readable.

What comes back?

A hypothesis, a design table, a verdict on whether the scope and variance can carry a readable result, contamination risks, stop rules and a day one logging list. The verdict in section three is the valuable one: it will tell you when the test cannot be read, and offer a before and after design with its weaknesses named.

What goes wrong most often?

Splitting at random. A handful of URLs usually carry most of the clicks on a template, so a random split puts them on one side and the result reads as the change. Match on baseline traffic, freeze other work on the template for the window, and agree the stop rules before the first bad week.