Implement Causal Inference Before You Claim Impact

Correlation in analytics data is everywhere; causation is rare. When a proper A/B test is not feasible — redesign rollouts, pricing changes, policy updates — quasi-experimental methods are essential. Even standard A/B tests can be compromised by broken randomisation or contamination between groups, so results that look too clean deserve scrutiny.

Difference-in-differences compares treatment and control groups across pre/post periods. Regression discontinuity exploits natural thresholds (e.g., users who barely qualify for a segment vs. those who barely don't). Synthetic control methods construct a counterfactual from weighted combinations of unaffected segments.

Python's causalimpact library (a port of Google's R package) automates Bayesian structural time-series analysis. It builds a counterfactual prediction of what would have happened without the intervention, then compares it with the observed data, with proper uncertainty quantification.

Stop saying 'the redesign increased conversion by 15%' when what you mean is 'conversion went up 15% after the redesign and we hope nothing else changed.' Causal inference methods give you the statistical machinery to make defensible claims about impact.

The practical workflow: identify your intervention date, select control segments (pages, markets, or user groups unaffected by the change), verify that they track the treated series in the pre-period (the parallel-trends check in difference-in-differences), then run the analysis. CausalImpact's output includes point estimates, credible intervals, and the posterior probability of a genuine effect.