Heterogeneous Treatment Effects: Why Flat A/B Tests Hide Wins

A/B tests often return a non-significant average effect. The average, however, assumes that every user responds to the change in the same way, and that assumption rarely holds. The event-level GA4 export in BigQuery holds enough detail to test it.

Segments respond differently to the same change. A revised onboarding flow might help first-time visitors from organic search while slowing down returning users who know the old flow. Averaged together, the two effects can cancel out and produce a flat result. Features dropped for showing no impact have later proved clearly positive for specific high-value groups once the results were broken down.

The analysis needs clean, user-level features. Model the GA4 events in dbt into one row per experiment participant, with pre-treatment attributes such as traffic source, purchase history, engagement level and device category. Then run subgroup analyses or, better, fit a regression with interaction terms between the variant and each attribute. The interaction coefficients show which segments the change helped and which it harmed.

Two safeguards keep the results honest. Segment only on characteristics measured before assignment, because attributes the treatment itself can change introduce bias. And define the segments before the test starts, or correct for multiple comparisons: test twenty subgroups at a 5% threshold and one of them is likely to look significant by chance.

Segmentation depends on a stable picture of each user. As cookie-based identifiers lose reliability, first-party behavioural data carries more of that load, and a consent-aware first-party event stream such as Robin Signals can supply the user-level attributes the analysis requires.

Declaring a test 'insignificant' on the average alone can mean missing hidden value, or shipping a change that harms a specific segment. Analysing heterogeneous effects replaces a binary yes/no with a usable conclusion: 'a clear win for segment A; segment B needs another iteration.'