Unit of Analysis in A/B Tests: Users, Sessions or Events

When A/B tests fail, the cause is not always weak variants or low statistical power. Often the problem sits deeper in the design: the unit of analysis. It is easy to overlook when tools aggregate data automatically, and getting it wrong can invalidate an experiment.

The unit of analysis is the independent entity that receives the treatment and on which the outcome is measured: a user, a session or an individual event. The choice determines how randomisation works, how metrics are aggregated and what the results mean. The Stable Unit Treatment Value Assumption (SUTVA) depends on these units being correctly defined and isolated; if they are not, statistical tests become unreliable.

The event-level GA4 BigQuery export offers flexibility, and with it responsibility. A common mistake is to aggregate metrics such as conversions per session without checking that the session matches the unit of randomisation. When users are randomised but each session is analysed as an independent observation, sessions from the same user are correlated, the variance is underestimated and p-values look more significant than they are. Experiments with exactly this mismatch have produced spuriously significant results.

The opposite mistake is assigning variants on session_start. A user with several sessions can then see different variants over time, and cookie expiry, re-entry or browser changes add further switching. Mixed exposure blurs the treatment effect and can hide a good variant or favour a poor one. The safer default is to assign at user level on first exposure, using user_id where available and otherwise user_pseudo_id, and to persist that assignment, for example in a dbt-modelled user_experiment_assignments table in BigQuery, so every later session inherits it.

The right unit depends on the treatment. In a pricing experiment, showing the same user different prices across sessions creates carryover effects and violates SUTVA, so the user is the right unit. For a subtle UI change on a page that users revisit, where only the immediate interaction matters, the session or even the page view can serve as the unit, provided the treatment is applied consistently within it.

Before looking at p-values or confidence intervals, define and validate the unit. Model the raw data, whether from the GA4 BigQuery export or a first-party stream such as Robin Signals, in dbt around that unit, and make sure the randomisation scheme matches it. Where the analysis unit and the randomisation unit differ, use methods that account for it, such as aggregating to the randomisation unit or using cluster-robust standard errors.