Diagnostics · Fleet Scale · Rigor
The numbers under the fleet method — is the split real, did you fool yourself by looking everywhere, can you catch the next one early, and did the fix actually work?
The fleet post built the logic; its rates and splits were illustrative. Here we put numbers under the figures and close the gaps. We make “is this cohort split real?” a calculation — confidence intervals, a chi-square test, the Mantel–Haenszel stratified rate — then name the trap that voids it: scanning many attributes for the best split manufactures false positives, fixed by Bonferroni/FDR and honest power. We move from reactive to proactive with control charts and CUSUM that catch a forming cluster at fifty cars; handle gradient boundaries with logistic risk-scoring and interactions; pick the number of clusters with BIC; turn the light forecast into Weibull hazards with censoring and B10 life; quantify the campaign as prevented-loss vs cost against a recall threshold; verify the fix by watching prevalence bend; and catch phantom patterns where a bad update lights a code fleet-wide with no real fault.
A fleet investigation can feel finished the moment a clean cohort boundary appears. But “feels clean” is not “is real,” and a fleet campaign costs too much to launch on a feeling.
The previous post ended with two causes and two campaigns. Every step there was sound logic — but four questions were left on the table, and each one can overturn a verdict:
None of this is heavy mathematics. It is a handful of calculations and habits that turn a persuasive narrative into one that survives an auditor, a regulator, and a budget meeting. We keep the 6,000-car scenario throughout.
Part A — Is It Real?
Before you act on a cohort split, two checks: could this gap be noise, and did I find it by honest test or by fishing?
The cohort boundary said: firmware ≥ v5.1 fails at 9%, older firmware at 1%. Is that gap solid, or could it be the luck of the draw? Three quick calculations settle it.
First, put error bars on each rate. A measured rate is an estimate; its uncertainty shrinks with sample size. The 95% confidence interval for a proportion is roughly p ± 1.96·√(p(1−p)/n). With 3,000 cars in each group the intervals are tight and nowhere near each other — so the gap is not noise.
# firmware ≥ v5.1 vs < v5.1 (n = 3000 each)
fw ≥ v5.1 : 270/3000 = 9.0% 95% CI [7.98 , 10.02]
fw < v5.1 : 30/3000 = 1.0% 95% CI [0.64 , 1.36]
CI = p ± 1.96·√(p(1-p)/n) → intervals don’t overlap
chi-square (2×2) χ² ≈ 202 → p < 10⁻⁴⁵ # overwhelming
rate difference = 8.0 points 95% CI [6.9 , 9.1]
# but small groups stay uncertain even when the effect is huge:
suspect lot, only 8 cars: 7/8 = 88% 95% CI ≈ [47 , 99]
real, but too few cars to ACT on with confidence yet
When you compare two groups that also differ on a confounder — say two plants that shipped different firmware mixes — never pool the raw counts (that is how Simpson’s paradox sneaks in). Instead compute the effect within each firmware stratum and combine them with the Mantel–Haenszel weighted average. It yields a single adjusted number that respects the strata, so a real plant effect survives and a fake one (driven only by firmware mix) collapses to nothing.
Here is the subtle fleet mistake that voids an otherwise careful analysis: you tried plant, firmware, build week, supplier, climate, trim, drive type… and reported the one that looked significant. That habit manufactures false positives.
Each test has a 5% chance of a false alarm by luck. Run twenty of them and you expect about one spurious “significant” split even if nothing is wrong. The fix is to raise the bar in proportion to how many places you looked.
m ← number of attributes tested # plant, fw, week, lot, climate, trim…
for a in attributes: p[a] ← test_split(a)
threshold ← 0.05 / m # Bonferroni — or control FDR (Benjamini–Hochberg)
return { a : p[a] < threshold } # survivors are trustworthy
# better still: pre-state the hypothesis, or confirm the winner on a held-out
# slice of VINs the scan never saw. and make sure each stratum has enough cars.
Correction guards against false positives; power guards against false negatives. A real cause confined to a 30-car lot can hide simply because the stratum is too small to reach significance. Before concluding “no effect,” check you had enough cars in that cell to have found one.
Part B — Better Structure
Real sub-populations don’t announce how many they are, and real boundaries are often gradients, not crisp rules. Two refinements to §§3–4 of the fleet method.
De-mixing asked you to cluster the affected cars. But how many clusters — two, three, seven? Splitting by eye invents structure that isn’t there; refusing to split hides structure that is.
The honest way is to fit the data with 1, 2, 3, … sub-populations and let a model-selection score pick. The Bayesian Information Criterion (BIC) rewards fitting the data and penalizes adding clusters, so it lands on the smallest number that genuinely helps. For our fleet, BIC bottoms out at two — the firmware group and the connector group — and gets worse if you force a third.
Two practical refinements. Use soft assignment — let a car belong, say, 70% to one cluster and 30% to another — because some cars genuinely sit on the edge or carry both faults. And keep that noise/singleton tail explicit, rather than forcing every odd car into the nearest group where it will distort the cohort boundary.
A crisp rule — “firmware ≥ v5.1” — is the lucky case. Often the boundary is soft: risk rises with something rather than switching on at a line.
If the connector fault gets more likely the more a car cold-fast-charges, there is no clean cutoff — risk climbs smoothly. The right tool is logistic risk-scoring: a model that turns several factors into a probability of failure for each car. Instead of “in or out,” you get a risk score, and the campaign targets the high-risk tail.
And sometimes no single factor is the cause — two are only dangerous together. Cold weather alone is fine; high mileage alone is fine; cold and high-mileage together trips the fault. Statisticians call this an interaction (or effect modification), and you find it by testing the combination, not just the parts.
Part C — From Snapshot to Stream
Waiting until thousands of cars fail is optional. Watch the rate live to catch a cluster while it is small — and turn the hand-wavy projection into a real survival curve.
The onset curve in the previous post was read in hindsight. The same idea, run live, becomes an early-warning system — one that trips while a new cluster is fifty cars, not five thousand.
A raw rate chart is too slow: a small rise hides in the week-to-week wobble until it is large. The trick is to accumulate the little excesses instead of looking at each week alone. A CUSUM (cumulative sum) keeps a running total of how far the rate sits above its normal baseline; a persistent small shift piles up and crosses an alarm line quickly, even though no single week looked alarming.
input weekly rate · baseline p0 · slack k · alarm threshold h
S ← 0 # running cumulative excess
for each week w:
S ← max(0, S + (rate[w] − p0 − k)) # add this week’s excess over baseline
if S > h: alarm("emerging cluster") # fires early, while counts are small
# run one monitor per build cohort (plant × firmware × lot) to localize fast
Running a CUSUM per build cohort does the de-mixing in advance: the monitor that trips tells you which sub-population is moving, so you arrive at the cohort boundary of §04 almost for free — and weeks earlier.
The previous post extended the onset curve with a dashed line and called it a forecast. To put real numbers on “how many more, by when,” you need the shape of the failure over a car’s life — its hazard.
The hazard is the chance a still-healthy car fails in the next interval. Its shape tells you which kind of fault you have, and the Weibull distribution captures all three shapes with one parameter, β:
One more thing the forecast must respect: censoring. Most cars have not failed yet — they are still on the road, or left the data when sold or scrapped. They are not “successes” to ignore; they are partial information (“survived at least this long”). Reliability methods use censored data correctly; treating un-failed cars as if they never will is how you under-forecast a wear-out problem.
Part D — Decide & Close the Loop
A campaign is a decision under cost and a hypothesis about the future. Make the trade-off explicit — then watch the fleet to confirm the fix actually bent the curve.
Whether and how widely to act is an economic decision with one safety override. Lay the numbers side by side per cluster.
Cost of inaction = the failures the forecast predicts × the cost of each (warranty repair + roadside + goodwill + liability). Cost of action = the campaign (per-car cost × how many cars you cover). Act when the loss you prevent exceeds what the campaign costs — and let the forecast’s slope set the urgency. The one exception that overrides the math: if the failure is a safety hazard, it crosses into mandatory-recall territory regardless of the economics.
A campaign is a prediction: “ship this and the failures stop.” At fleet scale you can check the prediction — the same kill-switch discipline from the single-car method, now read off the prevalence curve.
After the fix, the new-cases rate for the treated cohort should bend down toward baseline within a predictable window. If it does, the diagnosis is confirmed and the matter is closed. If it does not — if the curve keeps climbing — then either the fix was wrong, it was not applied to the right cars, or the cause was misdiagnosed. A flat or rising curve after remediation re-opens the case; it does not get explained away.
The most embarrassing fleet mistake is to launch a campaign against a fault that never existed. At population scale, the lying sensor of the single-car method becomes a lying detector — and it can light up thousands of cars at once.
A bad over-the-air update to the diagnostic logic itself, a mis-calibrated threshold, or a telematics parsing bug can set the same code across the whole fleet with no physical fault behind it. The danger is that it looks like a massive, urgent problem. The good news is that a phantom has a different shape from a real cohort fault:
Before any fleet campaign, ask the §06 question of Harder Cases at population scale: does physical telemetry corroborate the code, or is the code the only thing that changed? If a thousand cars threw the code but none of them show the underlying physical signal, you have a phantom — fix the diagnostic, and do not recall a single car.
A gap is only real if the intervals stay apart. With small groups, even a huge rate difference may be noise you cannot yet act on.
Scanning many attributes manufactures a fake winner. Raise the bar (Bonferroni/FDR), pre-state the hypothesis, or confirm on a held-out slice of VINs.
Before declaring “no effect,” check the cell had enough cars to have found one. Absence of significance is not significance of absence.
Pooling raw counts across groups that differ on a confounder invites Simpson’s paradox. Compute within strata, then weight them together.
Fit 1, 2, 3… sub-populations and take the BIC minimum. Keep a noise tail; allow soft membership for cars on the edge.
When risk rises smoothly, model it and target the high-risk tail. When only a combination of factors fails, you must test the combination.
Accumulate small excesses instead of glancing at weekly rates. A per-cohort monitor catches a forming cluster at fifty cars and tells you which group is moving.
The hazard shape names the fault (infant mortality vs wear-out); B10 life gives a concrete number. Un-failed cars are partial information, not zeros.
A campaign isn’t done until the treated cohort’s curve bends down. And before any campaign, confirm physical telemetry corroborates the code — or you may be chasing a phantom.
This post leans on statistics, reliability engineering, and quality control. Entry points, grouped by the question they answer.
A. Agresti, Categorical Data Analysis (2nd ed., 2002). Proportions, contingency-table tests, and the chi-square behind “is the split real?”
A. Agresti & B. Coull, “Approximate Is Better Than ‘Exact’ for Interval Estimation of Binomial Proportions” (1998). Trustworthy confidence intervals for rates, especially with small counts.
N. Mantel & W. Haenszel (1959). The stratified estimate that combines subgroup evidence without the Simpson reversal.
Y. Benjamini & Y. Hochberg, “Controlling the False Discovery Rate” (1995). The modern, less-conservative alternative to Bonferroni when testing many attributes.
S. F. Assmann et al., “Subgroup Analysis and Other (Mis)uses of Baseline Data” (Lancet, 2000). A clear-eyed warning about finding spurious subgroups — the multiplicity trap in practice.
G. Schwarz, “Estimating the Dimension of a Model” (1978). The Bayesian Information Criterion used to pick the number of sub-populations.
G. McLachlan & D. Peel, Finite Mixture Models (2000). Mixtures, soft assignment, and choosing the number of components.
D. Hosmer & S. Lemeshow, Applied Logistic Regression (2nd ed., 2000). Risk-scoring and interaction terms for gradient boundaries.
E. S. Page, “Continuous Inspection Schemes” (1954). The original CUSUM — accumulating small shifts to detect them fast.
D. C. Montgomery, Introduction to Statistical Quality Control. Control charts and the practice of monitoring rates over time.
M. Basseville & I. Nikiforov, Detection of Abrupt Changes (1993). The theory of change-point detection behind early warning.
W. Weibull, “A Statistical Distribution Function of Wide Applicability” (1951). The distribution whose shape parameter distinguishes infant mortality from wear-out.
W. Q. Meeker & L. A. Escobar, Statistical Methods for Reliability Data (1998); W. Nelson, Applied Life Data Analysis (1982). Hazard fitting, censoring, B10 life, and warranty projection.
H. Raiffa, Decision Analysis (1968). Framing the campaign as a cost-and-uncertainty decision, with a safety override.
These are orientation points into large literatures, cited to show the lineage of the method rather than as the source of any single claim here. Titles and years are given so you can find them.