Diagnostics · Evidence · Method
How to tell whether your measurements can actually reach a conclusion — and what to do when they can’t, or when they disagree with each other.
The companion post showed how to weigh messy, unequal-trust evidence into a single cause. It assumed the evidence was good enough to conclude. Often it is not. The core idea here, in plain words: more data is not the same as enough data. A thousand readings that look identical under two explanations cannot separate them. So sufficiency is never “do I have a lot?” — it is “can what I have tell my hypotheses apart, to the confidence this decision needs?” From that we build a sufficiency gate, a way to choose the single most useful next measurement, and a disciplined treatment of contradiction — with a worked EV example throughout.
It feels like a question about quantity. It is really a question about whether your data can tell two stories apart.
Picture two explanations for the same fault. If every measurement you own looks the same whether explanation A is true or explanation B is true, then collecting a thousand more of those same measurements changes nothing. You do not have too little data — you have data that cannot discriminate. Conversely, a single well-chosen reading that comes out one way under A and the other way under B can settle the matter on its own.
So replace the vague question with a sharp one:
Can the evidence I have separate my surviving hypotheses, by a margin large enough for the decision I have to make? If yes, conclude. If no, the useful next step is not “collect more” — it is “collect the one thing that would tell them apart.”
There is no such thing as “enough data” in the abstract. Sufficiency is always measured against three things.
The hypothesis set. Data is sufficient to separate the explanations you are currently considering. Add a new candidate tomorrow and yesterday’s sufficient data may no longer be enough — it never had to rule out that new story. This is why sufficiency is checked after framing and re-checked whenever the hypothesis list changes.
The confidence bar. How sure do you need to be? That is set by the cost of being wrong, and you should set it before looking at the data. Replacing a $12 sensor needs a low bar; pushing an over-the-air change to a safety system, or scrapping a production lot, needs a high one. The same evidence can be “enough” for the cheap decision and “nowhere near enough” for the expensive one.
The trust of the sources. A margin built from one shaky sensor is not the same as the same margin built from three independent, high-trust signals. As in the prequel, what counts is trust-weighted, independent evidence — not raw counts.
Evidence is sufficient when it separates these hypotheses, by this margin, using sources you trust — for a bar you fixed by the cost of being wrong, before you peeked.
Bundle the idea into a gate the conclusion must pass. Fail any check and you are not done — but you learn exactly why, which tells you what to do next.
Written as code, the gate returns not just yes/no but which exit you are taking and what to do about it.
input H surviving hypotheses · E evidence · bar required confidence
output CONFIRMED · INCONCLUSIVE(next) · INCOHERENT(reason)
if high_trust_conflict(E): return INCOHERENT("sources disagree") # §07
P ← posterior(H, E)
if margin(P) < bar: # checks 1 & 2
t ← next_measurement(H, E) # §05 value of information
return INCONCLUSIVE(t)
if orphan_anomalies(top(P), E): # check 3: coverage
return INCONCLUSIVE("re-frame: add a hypothesis")
if not survives_leave_one_out(top(P), E): # check 4: robustness
return INCONCLUSIVE("corroborate with an independent source")
return CONFIRMED
The first and most important check. If two explanations make the same prediction on every sensor you have, no amount of that data will choose between them.
Engineers call this observational equivalence; a detective would call it two suspects with the same alibi. The cure is a discriminating measurement — a single observation the two suspects do not share.
An EV throws a “reduced power” warning that shows up only under hard acceleration. The high-trust evidence on hand: the warning’s freeze-frame and a CAN log showing the pack terminal voltage sags when current is high. Two explanations survive the referee:
Predicts: terminal voltage sags under load.
Predicts: terminal voltage sags under load.
Both predict exactly what we measured. On the data we have, they are indistinguishable — the sufficiency gate fails at check 1.
If you must collect one more thing, collect the thing that shrinks your uncertainty the most. That idea has a name: value of information.
List the measurements you could take. For each, ask: how differently would it read under the surviving hypotheses, and how much would that split the posterior? The best test is the one that, on average, leaves you most certain afterward. In plain terms — pick the question whose answer you genuinely can’t predict, because that is the one that carries the most information.
input H hypotheses with current posterior · candidate tests T
output the single most informative test to run next
for t in T:
before ← uncertainty(posterior) # e.g. entropy of P(H)
gain[t] ← before − expected_uncertainty_after(t) # avg over t’s outcomes
return argmax gain[t] # break ties by cost / time / risk
We collect the module voltages. Now the two suspects part ways — because they make different predictions about this sensor:
Discrimination got us a leader. Three more checks decide whether the lead is trustworthy enough to act on.
Look at the gap between the top hypothesis and the runner-up, and remember §02: the gap must clear the bar set by the cost of being wrong, and it must be carried by high-trust, independent evidence. A 0.55-to-0.45 split is a leader in name only. A 0.94-to-0.06 split built on one good measurement plus the referee is a real one.
A sufficient explanation accounts for all the salient facts, not just the convenient ones. If your winning cause explains the voltage sag but leaves a second symptom — say, the warning also appearing once on a cold morning at full charge — then you have an orphan anomaly. That is a warning sign: either the cause is wrong, or a second fault is present and you have only found one. Unexplained leftovers mean re-frame, not conclude.
Ask the uncomfortable question: if my single most influential piece of evidence turned out to be wrong, would the verdict still stand? If everything hinges on one sensor that could be miscalibrated, you have a single point of failure dressed up as a conclusion. “Rich enough” means the answer survives a leave-one-out.
The instinct when two clues clash is to average them or pick the one you like. Both are wrong. A contradiction is a clue in its own right — it is telling you something specific.
Work through it in order:
First, does the trust hierarchy already settle it? If a high-trust source disagrees with a low-trust one — a freeze-frame against a vague customer report — there is no real conflict. The grade decides; keep the higher-trust signal and move on. Most “contradictions” dissolve right here.
If two high-trust sources clash, that is a real alarm. Two things you both trust cannot both be right. The most likely explanation is not that physics broke — it is that one of them is not what you think it is: a sensor is faulty, the firmware on the car is not the version in the repo, or two readings were taken at different points or different times (a provenance mismatch). Treat a high-trust conflict as a provenance bug until proven otherwise, and go re-audit. Resolving it usually is a finding.
If nothing fits well — support scattered across everything — the true cause is missing. When no hypothesis gathers real weight and the evidence pulls in all directions, the lesson is not “pick the least-bad option.” It is that your list is incomplete. Go back and frame a new candidate.
Formal evidence theory measures how much of your belief lands on combinations that cannot all be true. A lot of that “conflict mass” is a mathematical way of saying the same thing: your sources disagree more than chance allows, so stop combining and start questioning — either a source is corrupt, or the right answer is not on your list yet.
A diagnosis procedure that can only output “here is the answer” will lie to you when it has to. A good one can end in three honest states.
Pick up where §05 left off: the module-voltage test localized the drop to the connector, pushing H1 (bad connector) to 0.94. Discrimination passed. Then a contradiction appears.
Voltage localization says the loss is at the connector (→ H1). But the connector temperature log reads cool — and a high-resistance joint passing 142 A should get hot. Two trusted readings, opposite stories.
Following the high-trust-conflict rule, we check provenance. The temperature probe is mapped to the wrong connector, and its sample was logged at key-off, not under load. The reading was never measuring the suspect joint.
Re-measure the correct joint under load: it runs hot. The contradiction is gone, and it added an independent line of support for H1. Now two unrelated signals — voltage localization and temperature — agree.
Run the final gate. Discrimination: passed (module test). Margin: 0.96, high-trust. Coverage: explains the sag, the load-only timing, and the heat. Robustness: survives leave-one-out, because voltage and temperature are independent. Conflict: resolved. The procedure exits at Confirmed.
A corroded high-voltage busbar connector raises resistance, dropping voltage and tripping the reduced-power limit under high current. Disconfirming test: clean and torque the joint — if the voltage drop and heat vanish, confirmed; if they persist, re-open H2.
And the honest counterfactual: had the re-audit found the temperature probe correct and the joint genuinely cool, with modules also fine — nothing fitting — the procedure would have exited at Incoherent and sent us back to frame a new hypothesis (a failing inverter stage, perhaps), not forced a false verdict.
Part II — The Math, Made Concrete
Part I built the intuition with illustrative bars. Here we show the actual arithmetic — a real Bayesian update, the entropy that defines “information,” the sequential test that says stop now, and the cost rule that says stop collecting. Everything below is reproducible by hand.
The posteriors and the information-gain bars in Part I were round numbers for the eye. Here is the calculation that actually produces them, on the EV example.
Before the module-voltage test, the two suspects are tied: prior P(H1) = P(H2) = 0.5. The test can come out two ways — the drop is localized at the connector (call it Dₜ, favouring H1) or at a module (favouring H2). From bench data we know the test is reliable but not perfect:
P(Dₜ | H1) = 0.9 and P(Dₜ | H2) = 0.1
We run it and observe Dₜ (drop at the connector). Bayes’ rule just normalizes prior × likelihood:
| Hypothesis | Prior | Likelihood P(Dₜ|·) | Prior × Lik | Posterior |
|---|---|---|---|---|
| H1 · bad connector | 0.50 | 0.90 | 0.450 | 0.90 |
| H2 · weak module | 0.50 | 0.10 | 0.050 | 0.10 |
| total | 1.00 | — | 0.500 | 1.00 |
The shortcut worth memorizing is the odds form, because it makes combining evidence trivial: posterior odds = prior odds × likelihood ratio. If an independent second clue — the connector running hot — carries a likelihood ratio of 4, you just multiply:
odds = 1 × 9 × 4 = 36 → P(H1) = 36 / 37 ≈ 0.97
That multiplication is only valid if the two clues are independent given the hypothesis — the same rule that stopped us double-counting ten echoing complaints in the prequel. Correlated evidence does not multiply.
Uncertainty is measured with Shannon entropy, H = −Σ p·log₂(p). A 50/50 split is maximum confusion — exactly 1 bit. After the test pushes us to 90/10, the confusion drops to 0.47 bit. The information gain of the test is the difference.
# before the test — total confusion
H_before = -(0.5·log₂0.5 + 0.5·log₂0.5) = 1.00 bit
# after observing Dₜ — posterior 0.90 / 0.10
H_after = -(0.9·log₂0.9 + 0.1·log₂0.1) = 0.137 + 0.332 = 0.47 bit
information gain = 1.00 - 0.47 = 0.53 bit # this is the VoI number
Instead of fixing how many measurements to take in advance, take them one at a time and stop the moment the running total of evidence is decisive. This is Wald’s sequential probability ratio test — the cleanest formal answer to the title question.
Keep a running tally of the log-likelihood ratio: every new measurement adds log[ P(data|H1) / P(data|H0) ] to the total. Evidence for H1 pushes the tally up; evidence against pushes it down. Set two lines:
The beauty is that the thresholds are set directly by the error rates you will tolerate — A = log((1−β)/α), B = log(β/(1−α)) — so “enough” is defined by the risk you accept, exactly as in §02. A cheap decision uses tight thresholds and stops early; a safety-critical one uses wide thresholds and demands more.
input stream of measurements · thresholds A > 0 > B (from α, β)
S ← 0 # running log-likelihood ratio
for each measurement x:
S ← S + log[ P(x|H1) / P(x|H0) ]
if S ≥ A: return ACCEPT_H1 # enough — fault
if S ≤ B: return REJECT_H1 # enough — not this
return KEEP_SAMPLING # still between the lines
The sequential test stops when the evidence is decisive. But sometimes it never gets decisive — and chasing certainty has a price. The second stopping rule weighs the value of the next measurement against its cost.
Every candidate test has a value (its information gain, from §09, converted into how much it reduces your expected loss) and a cost (time on the lift, a teardown, a customer kept waiting, the risk of the test itself). The rule is simple:
Keep collecting while the best remaining test’s value of information exceeds its cost. Stop when even the most useful test left costs more than it would save — and decide now, on the current best hypothesis, accepting its residual risk.
Two forces work against each other. Each test you run leaves the next one with less to add — diminishing returns — so value falls as you go. Cost, meanwhile, holds steady or rises (the cheap tests get used up first). Where the curves cross is where you stop.
loop:
if sufficiency_check(H, E, bar) = CONFIRMED: return decide(H)
t* ← next_measurement(H, tests) # most informative remaining (§05)
if value(t*) ≤ cost(t*): # not worth it
return decide_under_uncertainty(top(H)) # stop, act on best
E ← E + collect(t*); tests ← tests − t*
Together the two rules cover both ways an investigation ends well: the sequential test stops you the instant the evidence is conclusive, and the cost rule stops you when conclusiveness is no longer affordable. Neither lets you drift — collecting data forever because you are afraid to decide.
The method in habits you can carry into the bay.
If you cannot say which single observation would change your mind, you are not ready to collect — or to conclude. The question comes before the data.
Decide how sure you must be from the cost of being wrong, in advance. Setting the bar after seeing the data is how motivated reasoning sneaks in.
Ten more identical logs move nothing. One orthogonal sensor can settle it. Spend the next measurement where the information is.
Don’t average it away. Check wiring, sensor mapping, firmware version, and when/where each reading was taken. The mismatch is usually the discovery.
A flat, scattered posterior is not bad luck — it is a signal the true cause is not on your list. Re-frame instead of forcing a choice among the wrong options.
If you cannot conclude, say so — and name the exact measurement that would. That is a useful, honest deliverable, not a failure.
If the verdict collapses without one clue, you have a single point of failure. Corroborate before you act on anything expensive.
The best time to guarantee you can tell faults apart is at design — sensor placement and testability analysis. Diagnosability is a property you can engineer, not just hope for.
The method braids together a few well-established lines of work. Good entry points, grouped by the question they answer.
A. Wald, Sequential Analysis (1947). The sequential probability ratio test — keep sampling only until the evidence ratio crosses a threshold. The original formalization of “stop when it is enough.”
C. E. Shannon, “A Mathematical Theory of Communication” (1948). Entropy as the measure of uncertainty that “margin” and “information gain” in this post are built on.
R. A. Howard, “Information Value Theory” (1966). The formal value of information: what a measurement is worth before you take it.
D. V. Lindley, “On a Measure of the Information Provided by an Experiment” (1956). Information gained from an experiment; the foundation of Bayesian optimal experimental design.
K. Chaloner & I. Verdinelli, “Bayesian Experimental Design: A Review” (1995). A readable survey of how to choose the most informative experiment.
M. Blanke, M. Kinnaert, J. Lunze, M. Staroswiecki, Diagnosis and Fault-Tolerant Control (Springer). Structural diagnosability and fault isolability — exactly when a set of measurements can or cannot separate given faults.
L. Ljung, System Identification: Theory for the User. Identifiability: the conditions under which data can actually pin down what you are trying to estimate.
R. Reiter, “A Theory of Diagnosis from First Principles” (1987). Diagnoses as explanations consistent with the observations; the formal idea of a “conflict” set.
J. de Kleer & B. C. Williams, “Diagnosing Multiple Faults” (1987). Generating candidates from conflicts, and reasoning about more than one fault at once.
G. Shafer, A Mathematical Theory of Evidence (1976). Belief functions and the notion of conflict mass when independent sources disagree.
P. Lipton, Inference to the Best Explanation (2004). The philosophy of abduction the whole diagnostic method rests on.
K. Ishikawa, Guide to Quality Control. The practical, shop-floor side: cause-and-effect (fishbone) analysis and disciplined root-cause work.
These are entry points to large literatures, cited for orientation rather than as the specific source of any single claim here. Titles and years are given so you can find them; consult the originals for exact statements.