Diagnostics · Causal Inference · Method
Observation suggests a cause; intervention proves it. How a staged software rollout becomes a randomized experiment — and how to settle causation when you can’t randomize.
The diagnosis series found a cause by watching: a charging code (P0AE0) concentrated in cars on firmware ≥ v5.1, pointing to a cold-throttle bug, and a v5.2 patch was proposed. But “the curve bent after we shipped” is not proof — coincidence, seasonality, regression to the mean, and other concurrent changes can all bend a curve. This post climbs the ladder of evidence from observation to intervention. We show why adjust-and-watch is limited by the confounders you never measured; draw the causal graph that says what to adjust for and what not to (the collider trap); then make the rollout interventional. A staged rollout, when the order is randomized, is a clean randomized experiment — treatment versus control, the difference a real causal effect. When you cannot randomize, two workhorses recover causation from luck: difference-in-differences across not-yet-updated cohorts, and instrumental variables / natural experiments where assignment is as-good-as-random. We close with how to run it well — power, a safe ramp, guardrails, a stopping rule — and a worked proof that v5.2 fixes the firmware cluster.
The whole diagnosis series, until now, has been about watching — reading evidence the fleet handed us and inferring a cause. Watching is powerful, but it has a ceiling. To prove a cause you eventually have to push on the world and see what moves.
The previous post left us with a strong observational story: the code P0AE0 concentrated in cars running firmware ≥ v5.1, a cold-throttle bug was the suspect, and a v5.2 patch was proposed. The natural next move is to ship v5.2, watch the curve, and declare victory when it bends. That move is a trap. There is a ladder of evidence, and shipping-and-watching sits only on the bottom rung:
Why isn’t “the curve bent after we shipped” proof? Because at least four other things can bend a curve at the same time:
None of these is exotic. Each has, in real programs, masqueraded as a successful fix. The cure is to stop relying on a single before/after line and instead build a comparison — a group that did not get the change but lived through the same week. This post is about constructing that comparison, cleanly when you can randomize and cleverly when you cannot.
Part A — Why Observation Isn’t Proof
You can only adjust for what you measured. The causal graph shows where the danger hides — and which variables you must not touch.
Observation’s honest defense is adjustment: if the firmware-≥-v5.1 cars also tend to be newer, or driven harder, or in colder regions, you control for those and compare like with like. It works — but only for the variables you actually recorded.
A confounder is a common cause of both the suspected cause and the outcome. It creates a correlation that is not a causal effect. Suppose newer cars both received firmware v5.1 and are garaged in colder northern dealerships. Cold weather raises the charge-throttle rate directly. Now firmware and the code are correlated — but partly because cold is driving both the firmware rollout schedule and the fault. Adjust for cold and the firmware effect might shrink, hold, or vanish.
The unbeatable problem is the confounder you did not measure. You can stratify by region, mileage, and build week because those are in the data. You cannot stratify by “owner charges on a faulty third-party home charger,” or “cars from a sub-lot with marginal connectors,” if you never logged it. Adjustment is blind to unrecorded common causes, and there is no statistical test that reveals one — it is, by definition, invisible to the data you have.
This is the ceiling. Observation can make a cause plausible and rule out the confounders you thought of, but it can never close the door on the one you didn’t. To shut that door you stop letting the world assign firmware and assign it yourself — which is what the rest of this post builds toward.
A causal graph is a picture of who causes whom: a node for each variable, an arrow from cause to effect. Drawn honestly, it tells you exactly which variables to control for — and, just as important, which ones you must leave alone.
The graph is a DAG — directed (arrows point from cause to effect) and acyclic (no loops; nothing causes itself round a cycle). You do not need equations to use one. You need two ideas: the back-door path and the collider.
A back-door path is a sneaky route from your suspected cause to the outcome that runs backwards through a common cause. In Figure 1, firmware ← cold → code is a back-door path: it carries correlation that is not the firmware effect. The rule is simple: adjust for the variable on a back-door path (here, cold) and the path is blocked, leaving only the honest front-door effect you care about.
A collider is the opposite trap. A collider is a variable that two arrows point into — a common effect, not a common cause. The dangerous rule is: do not adjust for a collider. Controlling for a common effect opens a spurious path that was closed, manufacturing a correlation out of thin air. This is the mistake that catches careful analysts: more controls are not always better.
A concrete collider for our fleet: imagine cars enter the warranty database (K) if they have either a firmware quirk or a hardware complaint. Among cars in that database, the two start to look negatively related — knowing a car got in without firmware trouble makes a hardware cause more likely. Restrict your analysis to flagged cars and you have conditioned on a collider, and invented a relationship that does not exist in the fleet. Draw the graph first; let it choose your controls.
Part B — Make It Interventional
Randomize the rollout and you get a clean causal effect for free. Can’t randomize? Difference-in-differences and instrumental variables recover causation from luck.
Here is the quiet power of a software fleet: you control who gets the update, and when. If you assign v5.2 by a coin flip instead of by schedule, the staged rollout is a randomized controlled experiment — the gold standard — and you barely had to do anything extra.
Randomization does something no amount of adjustment can. By deciding treatment with a coin flip, you guarantee that the treated and control groups are, on average, identical on everything — region, mileage, charger habits, the marginal-connector sub-lot, and every confounder you never measured. The unmeasured cold-region problem from §02 evaporates: cold cars are equally likely to be in treatment or control. The difference in their outcomes can then only be the firmware.
That “assign it yourself” move is what causal inference calls the do-operator. There is a difference between seeing a car on v5.2 (which is tangled with why it got v5.2) and setting a car to v5.2 by force. Plain-words version: instead of asking “what’s the fault rate among cars that happen to run v5.2?” you ask “what’s the fault rate when we make a random half of the fleet run v5.2?” The second question has a clean answer; the first is contaminated by every back-door path.
One ethical and practical note: control cars are not abandoned — they are simply not yet updated, and they get v5.2 the moment the experiment reads out. The randomized phase is short (days to a couple of weeks) and small (often 5–10% of the fleet), which is enough to measure the effect while exposing very few cars to the delay.
input eligible VINs · effect to detect δ · guardrail metrics G
n ← power_sample_size(δ, baseline_rate, α=0.05, power=0.8)
for each VIN in sample(eligible, 2×n):
arm[VIN] ← random(treatment, control) # the coin flip — closes every back door
push v5.2 to treatment arm only
for each ramp step in [canary, 10%, 50%, 100%]:
observe code_rate[treatment], code_rate[control]
if any g in G breaches limit: halt & roll back # guardrail
if SPRT_decision(treatment, control) = ACCEPT: break
effect ← rate[control] − rate[treatment] # with 95% CI
return effect, CI # then ship v5.2 to everyone
Sometimes you cannot randomize — the rollout already went out by region, or safety forbids withholding a fix, or the change is all-or-nothing. You can still recover a causal effect if some cars got the update and others did not yet, and you watched both before and after.
Difference-in-differences (DiD) is the trick. Take a treated cohort (got v5.2) and a comparison cohort (not-yet-updated, still v5.1). Measure each one’s code rate before and after the rollout. Then take two differences: the change in the treated group, minus the change in the comparison group. The second subtraction cancels out whatever shared trend hit both groups — spring warming, a charger-network change, the general seasonal dip. What survives is the part unique to having received the fix.
The whole method rests on one assumption you must defend: parallel trends. Absent the treatment, the two cohorts would have moved in parallel. You check it by looking backwards — if the two lines tracked each other for months before the rollout, parallel trends is credible; if they were already diverging, DiD is not trustworthy and the estimate is suspect.
input treated & comparison cohorts · rollout time t0
# check the assumption first — refuse to proceed if it fails
assert parallel_pretrend(treated, comparison, before=t0)
Δ_treated ← rate(treated, after=t0) − rate(treated, before=t0)
Δ_comparison ← rate(comparison, after=t0) − rate(comparison, before=t0)
effect ← Δ_treated − Δ_comparison # the difference of differences
return effect, CI(cluster=cohort) # cluster SEs by cohort
There is a third way, for when neither a clean randomization nor a parallel comparison is available, but the world happened to assign the update in a way that is as-good-as-random. You borrow that lucky randomness as an instrument.
An instrument is a variable that nudges who gets the treatment but has no other path to the outcome. The classic fleet instrument is rollout order or regional scheduling. Suppose the OTA server pushes v5.2 to cars in alphabetical order of VIN, or by which regional data center has spare bandwidth this week. Those things plausibly affect whether a car has v5.2 yet, but have no direct effect on the charge-throttle fault. That makes them an instrument: the variation in v5.2 exposure they create is as-good-as-random, and you can use it to estimate the firmware effect free of confounding.
The logic in plain words: the instrument only touches the outcome through the treatment. So if cars that got v5.2 early (for an unrelated scheduling reason) show fewer faults than cars scheduled late, and the only thing the schedule changed was v5.2 timing, the fault difference must trace to v5.2. Two conditions must hold — the instrument really does move treatment (relevance), and it touches the outcome only through treatment (the exclusion restriction). The second is an assumption you argue, not test; a scheduling order that also correlates with climate or car age would violate it.
Natural experiments are the same idea found in the wild: a policy, a deadline, a supply hiccup, or an administrative cutoff that split otherwise-similar cars into treated and untreated by accident. When you cannot run the experiment, the best you can do is find the one the world already ran for you.
Part C — Run It Well
Power, a safe ramp, guardrails, and a stopping rule turn a good idea into a trustworthy result — and a worked proof that v5.2 is the cause.
A randomized rollout is only as good as its design. Four decisions separate a clean read from a fleet incident: how many cars, how fast to ramp, what to watch besides the target, and when to stop.
Sample size and power. Before you start, ask: how big an effect do I need to detect, and how many cars give me a fair chance of seeing it? Power is the probability the experiment catches a real effect of the size you care about. Bigger effects need fewer cars; subtle ones need many. For a drop from a 9% baseline that you would act on at, say, 2 points, a few thousand cars per arm is ample; to detect a fraction of a point you would need far more. Compute this first — an underpowered experiment that reads “no effect” tells you nothing.
A safe ramp. Do not flip the whole fleet at once. Ramp: a tiny canary (a few hundred cars) to catch catastrophes, then 10%, then 50%, then 100%, pausing at each step to read guardrails. The randomized comparison rides along inside the early steps, so you measure the causal effect at 10% before committing the fleet.
Guardrail metrics. The patch might fix the charge code and break something else. Guardrails are the metrics you refuse to harm — battery thermal events, range, other DTCs, OTA failure rate. If any guardrail breaches its limit at any ramp step, halt and roll back, no matter how good the target metric looks.
A stopping rule. Decide in advance when you have enough evidence to stop — do not peek repeatedly and stop the moment it looks good (that manufactures false positives, exactly the multiplicity trap from the rigorous post). A sequential rule like the SPRT, covered in When Is the Evidence Enough?, lets you stop as soon as the evidence crosses a pre-set bar while controlling the error rate.
Put it together on the running fleet. We will prove v5.2 is the cause of the improvement two independent ways — a randomized canary and a difference-in-differences cross-check — and they should agree.
The randomized canary. From the eligible cohort (firmware ≥ v5.1), we randomly assign a 10% slice: roughly 5,000 cars split into 2,500 treatment (pushed v5.2) and 2,500 control (held on v5.1 for two weeks). Over the window, treatment throws the charge code at 1.2% (30 cars), control at 9.0% (225 cars). The randomized causal effect is the difference of rates, with a confidence interval on the difference:
# treatment (v5.2) vs control (v5.1), n = 2500 each
treatment : 30/2500 = 1.2% 95% CI [0.77 , 1.63]
control : 225/2500 = 9.0% 95% CI [7.88 , 10.12]
causal effect = 9.0 − 1.2 = −7.8 points 95% CI [−9.0 , −6.6]
CI excludes 0 by a mile → the patch causes the drop
relative reduction ≈ 87% # 1.2 / 9.0
# residual 1.2% is the non-firmware tail (connector cluster, noise)
The DiD cross-check. Independently, compare the treated cohort against a not-yet-updated cohort (a region still on v5.1), before and after the push. This guards against the small chance that the randomization window coincided with something else. The two estimates should land in the same place:
| Cohort | Before | After | Change (Δ) |
|---|---|---|---|
| Treated (got v5.2) | 9.0% | 1.4% | −7.6 pts |
| Comparison (still v5.1) | 8.6% | 8.2% | −0.4 pts |
| DiD effect | Δ_treated − Δ_comparison | −7.2 pts | |
The comparison cohort drifted down just 0.4 points over the window (a little seasonal warming), so the raw before/after of the treated group would have over-credited the fix by that much. Differencing it out gives a DiD effect of −7.2 points — within the confidence interval of the randomized estimate. Two methods, built on different assumptions, agree.
The patch causes an ~7–8 point absolute drop in the charge code (about an 87% relative reduction), confirmed by a randomized comparison and reproduced by a difference-in-differences cross-check that nets out the seasonal trend. This is intervention-grade proof, not a post-hoc story — ship v5.2 fleet-wide and continue the ramp.
The residual 1.2% in the treated arm is real and is not firmware — it is the separate connector cluster and noise from the earlier posts. Proving v5.2 fixes the firmware cause says nothing about that tail; it gets its own diagnosis and, if warranted, its own experiment. Proving one cause never closes the others.
Treat “the curve bent after we shipped” as a hypothesis, not a conclusion. Move up to a quasi-experiment or a randomized rollout before you claim a cause.
Coincidence, seasonality, regression to the mean, and concurrent changes each mimic a fix. A comparison group is the only thing that rules them out at once.
Adjustment defeats the confounders you measured and is blind to the ones you didn’t. No test reveals an unmeasured common cause — only randomization defeats it.
Adjust for confounders on a back-door path; never adjust for a collider. More controls are not always better — conditioning on a common effect invents correlations.
Assign the update by coin flip and the arms match on everything, measured or not. The difference in outcomes is then a clean causal effect.
When you can’t randomize, difference out the shared trend across a not-yet-updated cohort — but check the pre-period parallelism first, or the estimate is suspect.
Rollout order or regional scheduling can be an instrument if it moves treatment but touches the outcome only through it. The exclusion restriction is argued, not tested.
Compute power up front; ramp canary → 10% → 50% → 100% behind guardrails; and use a pre-set stopping rule instead of peeking until it looks good.
A randomized estimate confirmed by a DiD cross-check is far stronger than either alone. And proving one cause never closes the others — the residual tail gets its own proof.
This post draws on causal inference, econometrics, and the practice of online controlled experiments. Entry points, grouped by the question they answer.
J. Pearl & D. Mackenzie, The Book of Why (2018). The accessible introduction to causal graphs, confounders, colliders, and the ladder from seeing to doing.
J. Pearl, Causality (2nd ed., 2009). The formal treatment — DAGs, the back-door criterion, and the do-operator that defines intervention.
G. Imbens & D. Rubin, Causal Inference for Statistics, Social, and Biomedical Sciences (2015). The potential-outcomes framework, randomization, and instrumental variables in one place.
R. A. Fisher, The Design of Experiments (1935). The founding text of randomization as the basis of causal inference.
J. Angrist & J.-S. Pischke, Mostly Harmless Econometrics (2009). Difference-in-differences, instrumental variables, and natural experiments, explained for practitioners.
D. Card & A. Krueger, “Minimum Wages and Employment” (American Economic Review, 1994). The canonical difference-in-differences study and a model of the parallel-trends argument.
R. Kohavi, D. Tang & Y. Xu, Trustworthy Online Controlled Experiments (2020). Staged rollouts, guardrail metrics, power, ramps, and stopping rules from large-scale practice.
These are orientation points into large literatures, cited to show the lineage of the method rather than as the source of any single claim here. Titles and years are given so you can find them.