Filtering A/B Test Groups by Outcome Breaks Randomisation

A/B testing looks simple: run the experiment, compare the groups, declare a winner. Yet well-designed experiments are often undermined at the analysis stage, not by technical faults but by how the groups are compared, and the decisions that follow are made with misplaced confidence.

A common cause is selection bias introduced after randomisation. Randomisation makes the groups comparable only as a whole. Once the analysis filters them on behaviour that happens after assignment, the groups being compared are no longer the randomised groups, and that comparability is lost.

The GA4 BigQuery export makes this easy to do by accident. A typical query compares 'users who completed purchase X' in variant A and variant B. If completing purchase X is an outcome the variant can influence, filtering on it first selects different kinds of users in each group. A variant that brings more marginal users through to purchase can then look worse on per-buyer metrics even when it is better for the business. The nested event_params and user_properties fields make such filters easy to write, so each one needs checking against this rule.

The remedy is intention-to-treat (ITT) analysis. Once a user is randomised, they stay in that group for the analysis whatever they do afterwards, including users who never saw the variant because of a technical glitch or an early exit. A dbt staging model such as stg_experiments, holding one row per user with the identifier and assigned_variant recorded at the moment of randomisation, keeps that link intact. Metrics are then computed over all assigned users, with outcomes such as purchases in the numerator rather than used as a filter.

In observational studies, where there is no randomisation, selection bias is harder to handle and calls for methods such as propensity score matching or instrumental variables, typically in R or Python. These depend on consistent identifiers and complete event streams, which a privacy-first collection layer such as Robin Signals helps provide.

When reviewing A/B results, the key question is whether the comparison still reflects the original randomised groups. Admitting that a 'win' is uncertain can be unwelcome, but it is the standard a data-driven culture depends on.