Why every offer program needs a holdout group

The question attribution cannot answer

Attribution tells you which touchpoint a conversion followed. It cannot tell you whether the conversion needed that touchpoint. Google describes its Conversion Lift studies as a controlled experiment: one group sees the ads, a control group does not, and the difference in conversions is the lift caused by the ads. The same logic applies to on-site offers, recommendations and messages. The group that does not see them is the holdout.

A simple example

Suppose a free shipping message is shown to 90% of visitors and withheld from 10%. Visitors who saw it convert at 3.3%; the holdout converts at 3.0%. Attribution would credit the message with every conversion among those who saw it. The holdout shows that 3.0 of those 3.3 percentage points would have happened anyway, so only about one conversion in eleven was incremental.

If that small difference costs margin on every order, the program can look successful in reports and still lose money. The holdout is what makes that visible.

Keeping the holdout honest

Assign visitors randomly and keep each one in the same group for the whole test, so a returning visitor does not switch sides. Decide the size, duration and success metric before starting, and report uncertainty, not just the point estimate: a 0.3-point difference can be noise with little traffic.

A holdout means some visitors deliberately do not get the treatment. That is the cost of knowing whether it works. Measure contribution per visitor, not only conversion rate, so a lift that gives away more than it earns does not count as a win.