Assignment vs Exposure: Defining 'Treated' in A/B Tests

Analysts spend much effort on p-values and confidence intervals, yet the definition of 'treatment' often gets little scrutiny. It deserves more, because a loose definition biases the result before any statistics are run. It should be fixed before launch, in the data model as well as in the testing tool.

The core question is whether a user counts as treated because they were assigned to a variant or because they actually experienced it. Many assigned users never see the change: they leave before the element loads, or never reach the page where it appears. Counting all of them as treated dilutes the effect, lowering the observed lift and inflating p-values, and good features get discarded.

This is the distinction between intention-to-treat (ITT) and treatment-on-the-treated (TOT). ITT compares all assigned users and preserves the randomisation; it measures what launching the change does to the whole population. TOT asks what the change does to the users who actually experience it. In a GA4 BigQuery export this takes two steps: identify the users (user_pseudo_id, or user_id where set) assigned to each variant, then separately identify those who demonstrably reached the treatment, using events rather than user properties.

The second step contains a trap. Comparing exposed treatment users with the whole control group is not a TOT estimate, because users who reach the element differ systematically from those who do not. A valid control consists of users who could have received the treatment: those who reached the same point but were shown the original. Log that trigger in both arms, for example with a custom event such as cta_impression fired when the element renders in view, and compare triggered users with triggered users. Generic events are weak proxies: enhanced-measurement scroll fires only at 90% page depth, and view_item records a product view, not sight of a CTA. Where trigger logging is impossible, dividing the ITT effect by the share of assigned users who were exposed gives a TOT estimate under standard assumptions.

The control group can also be contaminated. If control users encounter elements of the variant, for instance through cached pages or components outside the test's scope, the comparison captures less than the full effect. Checking the control group for treatment events is a cheap delivery test, and a server-side collection layer such as Robin Signals makes such touchpoints easier to instrument consistently.

Ignoring these distinctions produces results that are diluted at best and misleading at worst. An effective feature can be discarded because unexposed users mask its effect, or an effect can be credited to a variant that was never properly delivered. Precise definitions, set before data collection begins, are what make the causal claim defensible.