Diagnostics · Prevention · Method
Finding the cause is half the job. The other half is making sure it never comes back — feeding the fix into design, process, and test so the fault is engineered out.
The previous posts in this series found the cause. This one asks the harder question: how do you make sure it never happens again? Fixing the affected cars is containment — it stops the bleeding, but it does not change the design, the process, or the test that let the fault through. Recurrence is the real failure. We walk the disciplined corrective-action loop (8D / CAPA, Deming’s PDCA), separate the three actions people conflate — contain, correct, prevent — and then feed the confirmed cause backward: into the FMEA as a documented failure mode, into the process control plan and its control charts, and into a poka-yoke that makes the failure physically impossible. We hunt the escape point — the test that should have caught it — add the missing test, and write the case into institutional memory so the next program inherits the fix. We run our two known causes — the firmware cold-throttle bug and the high-resistance connector lot — all the way through the loop, end to end.
There is a quiet moment after a root cause is confirmed when the team believes the job is done. It is not. Knowing why the cars failed and ensuring they stop failing — permanently, across every program that follows — are two different problems, and the second is the harder one.
Across this series we tracked one charging code, P0AE0, through a fleet of 6,000 EVs and found two root causes: a firmware cold-throttle bug that misbehaves on builds ≥ v5.1, and a high-resistance charge connector from a bad supplier lot (Supplier-B, build weeks 12–18). The fleet post decided two campaigns; the rigorous follow-up verified the fix actually bent the curve. So we are done?
No. A campaign re-flashes firmware and swaps connectors on the cars that already have the fault. That is containment — necessary, urgent, and entirely backward-looking. It says nothing about the next firmware build, the next supplier lot, or the next program that reuses the same connector. If the bug that wrote the bad firmware is still in the codebase, and the process that let a bad lot ship is still in place, and the test suite that missed the cold-charge fault is still green, then the fault is contained, not prevented. It is waiting to come back.
The discipline that closes this gap has a name in quality engineering: the corrective-action loop. Its oldest and simplest form is W. Edwards Deming’s PDCA cycle — Plan, Do, Check, Act — the idea that improvement is not a one-shot fix but a turning wheel: you plan a change, do it, check whether it worked, and then act to standardize it so the gain holds and the next turn starts from higher ground. Recurrence prevention is the “Act” that most teams skip.
A root-cause analysis that ends at the fix is half an analysis. The cause is an asset: spend it on the design, the process, and the test — so the next car, the next lot, and the next program inherit a fault that has been engineered out, not merely cleaned up.
Part A — The Disciplined Loop
The whole point of a formal loop is to make “we fixed the batch” an unacceptable place to close the case. Two ideas: the eight-step loop, and the three actions it forces you to keep separate.
Ford’s 8D (“Eight Disciplines”) and the broader CAPA (Corrective And Preventive Action) framework exist for exactly one reason: to stop teams declaring victory after containment. They lay the work out as a sequence you are not allowed to short-circuit.
The structure matters because it puts three distinct deliverables in three distinct boxes — an immediate containment, a corrective action that removes the cause, and a preventive action that stops the cause from recurring elsewhere — and then refuses to close until you have verified each one and standardized the result.
The eight disciplines map cleanly onto the boxes above. D1–D3 form the team, describe the problem precisely, and put an interim containment in place — the campaign that re-flashes and re-connectors the affected cars. D4 is the root-cause work the rest of this series covered. D5–D6 choose and implement the permanent corrective action. D7 — the discipline most often skipped — takes preventive action so the cause cannot recur. D8 recognizes the team and closes the case only after the prevention is standardized.
input a confirmed problem with a verified root cause
contain(problem) # D1-D3: stop the bleeding on cars in the field
cause ← root_cause(problem) # D4: from the rest of this series
fix ← choose_correction(cause) # D5-D6: removes THIS cause
apply(fix)
if not verify(fix): return reopen(problem) # curve must bend (Fleet RCA, Made Rigorous)
# D7 -- the part teams skip: prevent recurrence, not just this batch
for layer in [design, process, test]:
update_fmea(layer, cause) # document the failure mode, re-score RPN
add_control(layer, cause) # poka-yoke / SPC / lint / HIL test
escape ← find_escape_point(cause) # why did validation miss it?
apply(close_escape(escape)) # add the test that would have caught it
standardize(cause, fix) # D8: into guidelines + knowledge base
return closed # next program inherits the fix for free
Notice the loop never returns “closed” on the strength of containment alone. It demands a verified correction, a prevention applied at every layer, a closed escape point, and a standardized record. PDCA is the same wheel, smaller: Plan the change, Do it, Check that it worked, Act to lock it in — and turn again.
Most failed corrective actions are not wrong — they are incomplete, because someone did one of the three actions and called it all three. Naming them apart is the single most useful habit in this whole discipline.
The three differ by what they change and how far forward they reach. Containment changes the affected units. Correction changes the cause for this design. Prevention changes the system so the cause cannot recur in any future unit. They are a staircase, and you have to climb all three steps.
Run our two causes up the staircase to feel the difference. For the firmware bug: containment re-flashes the affected cars; correction patches the offending code in the current build; prevention is a compile-time guard or lint rule that makes the class of bug impossible to write again. For the connector: containment swaps connectors on the bad-lot cars; correction quarantines the lot and tightens incoming inspection; prevention is a physical key on the connector so a non-conforming part cannot be installed at all. Only the third step buys you a future without the fault.
Part B — Feed The Cause Back
A verified root cause is a gift to the design and process records. Three places to spend it: the FMEA, the control plan, and an error-proofing change that needs no human to remember it.
The FMEA — Failure Mode and Effects Analysis — is the living register of every way a design (DFMEA) or process (PFMEA) can fail. A confirmed root cause that is not in the FMEA is a lesson the organization is about to forget. Putting it there is the cheapest prevention you will ever do.
An FMEA scores each failure mode on three axes from 1 to 10 and multiplies them into a Risk Priority Number:
The trick that makes the FMEA preventive rather than merely documentary: you record the cause, then you change the controls so occurrence or detection improves, and you re-score. The RPN falls because the system genuinely got safer — not because you wished it lower.
input a failure mode with current controls
S ← severity(mode) # 1..10 -- effect on the customer; rarely changeable
O ← occurrence(mode) # 1..10 -- how often the cause arises
D ← detection(mode) # 1..10 -- 10 = controls never catch it
RPN ← S × O × D # prioritize the high numbers first
# prevention = drive O down (poka-yoke, SPC) or D down (add a test):
after action: O′,D′ ← re_score(mode)
assert S × O′ × D′ < RPN # if it didn’t drop, the action didn’t work
Two cautions on RPN. First, do not let a low product fool you: a failure with severity 9 deserves attention even at a modest RPN, which is why the newer AIAG-VDA handbook replaces raw RPN with Action Priority — severity-first triage rather than a single multiplied number. Second, the DFMEA and PFMEA must agree: the connector cause lives in the PFMEA (a process let a bad part through) while the firmware cause lives in the DFMEA (a design flaw); a real fix updates whichever register owns the cause, and often both.
Lowering the FMEA’s occurrence score is a promise; a process control plan with statistical process control (SPC) is how you keep it. The move is to chart the cause, upstream, instead of waiting for the symptom to reappear downstream.
The high-resistance connector did not fail at random — it failed because a manufacturing parameter drifted. Suppose the root cause was an under-spec crimp force on the connector terminals from Supplier-B. The reactive habit is to test finished connectors for resistance and reject the bad ones. The preventive habit is to put a control chart on the crimp force itself, on the line, so a drift toward the failure region is caught before a single bad connector is built.
The control plan formalizes this: for each characteristic that the FMEA flagged as risky, it names the parameter to measure, the method, the sample size and frequency, the control limits, and the reaction plan when a point goes out of bounds. The firmware cause gets the software equivalent — a controlled build process, signed releases, and a regression gate — so that “a bad build reached the fleet” becomes a charted, gated event rather than a surprise.
SPC catches the cause; a poka-yoke stops it from ever happening. Shigeo Shingo’s error-proofing principle is the top of the staircase: redesign the part or the step so the wrong outcome is physically impossible, not just flagged after the fact.
A detection control still depends on someone reading the chart and reacting in time. A poka-yoke removes that dependency. The classic distinction: a warning poka-yoke alerts you that something is wrong; a control poka-yoke makes the wrong action mechanically impossible. Prevention prefers the latter.
static_assert) so the signed/unsigned cast behind the cold-throttle bug will not build. Neither defense depends on anyone remembering anything.This is the difference between a control that can erode and one that cannot. A keyed connector does not get tired on the night shift; a build that refuses to compile the unsafe cast does not depend on a reviewer catching it. When you can reach the top of the staircase — making the failure impossible — do it, because every layer below it relies on a human or a chart that can lapse.
Part C — Catch It Earlier Next Time
If the fault reached customers, some test let it through. Find that gap, plug it, and write the whole case into a memory the next program will actually read.
Every fault that reaches a customer passed through a validation process that was supposed to stop it. The point where it slipped through is the escape point, and finding it is a second, parallel root-cause analysis — not of the fault, but of the detection failure.
This is the distinction between the occurrence root cause (why did the fault happen?) and the escape root cause (why didn’t we catch it?). A complete corrective action closes both. You can make the cause rarer and still ship it again if the test that should catch it does not exist — so D7 prevention is incomplete until the missing test is in the suite.
Closing the escape is what converts a single fix into a permanent one. The connector lot gets an incoming-inspection gate it never had; the firmware gets a cold-charge test and a unit test for the exact cast. Both feed straight back into the FMEA’s detection score from §04 — the escape analysis and the RPN re-score are the same act seen from two sides.
A fix that lives only in one team’s heads dies when the team disbands. The final discipline is to write the case into institutional memory — a knowledge base, an FMEA library, a design guideline — so the next program inherits the lesson without re-suffering it.
This is the “Act” of PDCA at the organizational scale and the “D8” of the 8D loop: standardize the gain. A lesson that is not captured is a recurrence waiting for the institutional memory to fade. The strongest form is not a document someone has to remember to read — it is a lessons-learned entry wired into a reusable FMEA template and a design checklist, so the prevention shows up automatically the next time a similar part or feature is designed.
The link to earlier posts is direct: the diagnostic reasoning that found the cause is only worth its cost if its conclusion is preserved. A knowledge base of past root causes also accelerates future diagnosis — the next time P0AE0 appears, the first question is “is this the firmware mode, the connector mode, or something new?”, and the library answers two of those instantly.
Theory earns its keep when it runs end to end. Here are our two confirmed causes — the firmware cold-throttle bug and the connector lot — pushed through every step of the loop, from containment to a standardized guideline.
Contain: OTA re-flash to a corrected build for every car on ≥ v5.1. Correct: patch the signed/unsigned cast that wraps negative at sub-zero temperatures. Prevent — design: a lint rule plus static_assert that rejects the lossy cast at compile time, so the build fails rather than ships. Prevent — FMEA: add the failure mode to the DFMEA; detection drops from 7 to 2 once the test exists; RPN 294 → 24. Escape: validation never charged below 0°C — add a cold-charge HIL test and a unit test pinning the cast. Standardize: “no lossy numeric casts on sensor paths” enters the firmware design guideline and the CI gate.
Contain: recall and swap connectors on the ~120 VINs traced to the bad lot. Correct: quarantine the lot; issue a supplier 8D on the under-spec crimp force. Prevent — process: SPC control chart on crimp force on the supplier line, with a reaction plan that stops the line on a drift. Prevent — design: connector keying so a non-conforming lot physically cannot seat. Prevent — FMEA: add to the PFMEA; occurrence drops from 8 to 2, detection 6 to 2; RPN 336 → 28. Escape: no incoming inspection measured crimp force — add it as a gate. Standardize: crimp-force SPC and connector keying enter the standard control plan for all future programs using this connector family.
| Step | Firmware bug | Connector lot |
|---|---|---|
| Contain | OTA re-flash, fw ≥ v5.1 | Swap connectors, ~120 VINs |
| Correct | Patch the cast | Quarantine lot, supplier 8D |
| Prevent (occurrence) | Lint rule / static_assert | Connector keying |
| Prevent (detection) | Cold-charge HIL + unit test | Crimp-force SPC + incoming gate |
| RPN before → after | 294 → 24 | 336 → 28 |
| Standardize | Firmware design guideline + CI gate | Control plan + keyed-connector standard |
Read the table as a promise: for each cause, the fault is now rare (occurrence down), caught if it recurs (detection down), impossible at one layer (keying / compile guard), and written where the next program will find it (standardized). That is what “closed” means — not “the batch is fixed” but “the fault has been engineered out.”
Write all three down explicitly for every cause. A corrective action that does only one of the three and calls it done is the most common way faults come back.
A bent curve proves the containment worked; it says nothing about recurrence. Do not close until prevention is applied and the lesson is standardized (8D’s D7–D8).
Ask both “why did it happen?” and “why didn’t we catch it?” A rarer cause still ships if the missing test stays missing.
Update the FMEA’s occurrence or detection only when a real control changed it. A wished-down number is worse than no number — it hides the risk.
Make it impossible if you can (keying, compile guard); catch it with SPC if you can’t; rely on a human remembering only as a last resort.
Control the crimp force, the build gate, the upstream knob — so a drift trips before a single bad unit exists, not after the field codes appear.
Design causes go to the DFMEA, process causes to the PFMEA — and often both must move. Severity-first triage (Action Priority) beats chasing a single RPN.
A reusable FMEA entry, a checklist line, or a lint rule is inherited automatically by the next program. A document someone must remember to read is forgotten.
This post draws on quality engineering, reliability, and the lean production tradition. Entry points, grouped by the question they answer.
Ford Motor Company, Team Oriented Problem Solving (TOPS) / Global 8D. The eight-discipline structure that separates containment from corrective and preventive action and refuses to close early.
W. Edwards Deming, Out of the Crisis (1986). The PDCA / PDSA cycle and the case that quality is a system to be improved continuously, not a defect to be inspected out.
AIAG & VDA, FMEA Handbook (2019). The current joint standard — structured analysis and severity-first Action Priority in place of raw RPN.
AIAG, Potential Failure Mode and Effects Analysis (FMEA) (4th ed., 2008). The earlier reference that defined severity, occurrence, detection, and the Risk Priority Number.
IATF 16949 & ISO 9001. The quality-management standards that mandate documented corrective action, control plans, and preventive action across the automotive supply chain.
Shigeo Shingo, Zero Quality Control: Source Inspection and the Poka-Yoke System (1986). The principle of error-proofing — designing the step so the defect cannot occur, rather than inspecting for it after.
Taiichi Ohno, Toyota Production System: Beyond Large-Scale Production (1988). The origin of the Five Whys and the discipline of asking until the true, addressable cause is reached.
James Reason, Human Error (1990). The Swiss-cheese model of layered defenses — an accident happens only when holes in successive barriers line up, which is exactly what closing an escape point prevents.
These are orientation points into large literatures, cited to show the lineage of the method rather than as the source of any single claim here. Titles and years are given so you can find them.