Short-Term A/B Test Wins Can Hide Long-Term Losses

A/B testing underpins product and marketing optimisation, but a single p-value is often treated as the end of the analysis. That shortcut misses effects that only appear over time.

A standard test compares group averages within a fixed window. It does not capture novelty effects that fade once users get used to a change, spillover between control and treatment, or consequences that materialise months later. The GA4 export in BigQuery holds event-level data for every user, yet analyses commonly reduce it to one outcome metric over a few weeks.

In one observed case, a test showed a statistically significant short-term increase in conversions. Following the assigned cohorts for several months, segmented and modelled in dbt, revealed that the 'winning' variant produced markedly higher churn among high-value customers. The short-term gain concealed a long-term loss.

A few checks catch this. Plot the treatment effect by day or week since first exposure: an effect that shrinks steadily points to novelty. Keep tracking both groups by their original assignment after the test closes, on retention and revenue rather than only the initial conversion metric. Survival analysis suits the retention side, because it handles users whose churn has not yet been observed. Where a clean test is impossible, for example because of network effects or a platform-wide change, time-series methods such as CausalImpact can estimate the effect instead.

All of this depends on accurate, consistent and privacy-compliant user-level data. Without it, long-horizon tracking produces unreliable conclusions.

The p-value is a first signal, not the conclusion. Investigating why a test came out as it did, which users drove the result and what happened afterwards takes more work, but it separates a lasting improvement from a temporary spike.