Network Effects Break the Independence Assumption in A/B Tests

Most A/B tests assume that users in the treatment and control groups do not affect one another, part of what causal inference calls the Stable Unit Treatment Value Assumption (SUTVA). The assumption is violated more often than analysts acknowledge, and conclusions about feature impact suffer as a result.

The usual cause is a network effect, or spillover. If a treated user shares content through a new 'share to social' feature, their connections in the control group may engage with it and receive an indirect, diluted form of the treatment. Referral schemes and collaborative features carry the same risk.

Spillover contaminates the control group and biases the estimated treatment effect. When the spillover is positive, the control group benefits too and the effect is usually underestimated; in other set-ups, such as treated and control users competing for the same limited inventory, it can be overestimated. Either way, product decisions rest on distorted evidence.

The standard mitigation is cluster randomisation. Instead of randomising individual users, assign whole clusters, such as communities, households, companies or IP ranges, so that most interactions stay within one arm. The analysis must then follow the design: with clusters as the unit of randomisation, outcomes should be analysed at cluster level or with cluster-robust standard errors, otherwise the variance is underestimated. Fewer, larger units also reduce statistical power, so the test usually needs to run longer.

Clustering depends on identifiers that link related users. Collecting them in a privacy-preserving way, for example with a server-side solution such as Robin Signals, provides the context these designs need without compromising user privacy.

Assessing interference is a practical requirement, not an academic exercise. Before running a test, ask whether treated users can influence control users; if they can, randomise at the level where that influence stops.