Logic and Reasoning

The health of a thinking system

A thinking system is never handed perfect knowledge. Its health shows not in how it performs when everything is known, but in how it conducts itself when the knowledge is bad.

Knowledge goes bad in three ways, and each is a distinct reasoning regime with its own right moves and its own way of failing. It can be uncertain, where you have information but it is noisy and only probable. It can be conflicting, where your sources disagree. Or it can be missing, where there are gaps, including the gaps you do not know you have. A healthy reasoner behaves differently in each, and the master skill, the genuinely cognitive one, is metacognitive: recognizing which regime you are in and switching how you think, instead of charging ahead as if the ground were solid.

Each regime has a signature pathology, and you have met all three in passing across the series. Here they get named. Uncertainty mishandled becomes overconfidence. Conflict mishandled becomes either suppression or paralysis. Missing knowledge mishandled becomes hallucination, the confident filling of a void. Pick a situation and see which regime it is and what the healthy move would be.

These are not three separate topics but one skill seen three ways: knowing the state of your own knowledge and matching your reasoning to it. The rest of this part takes each regime in turn, then returns to the loop that keeps the whole thing honest.

Reasoning under uncertainty

The most common bad-knowledge state is also the most underestimated: you have information, but it is uncertain, and the danger is treating it as if it were not.

The first move of a healthy reasoner is to represent uncertainty as more than a single guess: a range, a distribution, a confidence attached to the claim, not a bare point estimate pretending to be a fact. The second is to know which kind of uncertainty it faces. Aleatoric uncertainty is the irreducible randomness of the world; no amount of study tells you which way a fair coin will land. Epistemic uncertainty is your own ignorance, and more information can shrink it. The distinction is not academic: it tells you whether to gather more, which only helps with epistemic uncertainty, or to hedge and accept, which is all you can do with aleatoric. Confusing them wastes effort chasing irreducible noise, or treats reducible ignorance as fate.

Calibration is the health signal

A healthy reasoner's confidence matches its accuracy. When it says seventy percent, it is right about seventy percent of the time. Overconfidence, saying ninety and being right sixty, and underconfidence, saying sixty and being right ninety, are both pathologies, and overconfidence is the dangerous one because it hides risk behind a confident voice. Push the curve below and watch what each looks like.

Two more disciplines round it out. Uncertainty propagates, and a healthy process carries it through rather than quietly rounding each step to certain. A chain of ten steps each ninety percent sure is not ninety percent sure overall; it is about thirty-five. And a healthy reasoner is willing to abstain, to say "I do not know" or "not enough to decide." A system that always produces an answer and never abstains is not confident, it is broken.

When the evidence conflicts

Sometimes the trouble is not too little information but too much of the wrong kind: sources that disagree. The instinct to average them, or to keep the one you like, is usually wrong.

Conflict has many causes: noisy sensors, unreliable sources, adversarial inputs, genuine ambiguity, or two models looking at different slices of the world. The first healthy move is to weigh by reliability rather than average blindly. A precise instrument and a rumor are not to be split down the middle; you combine them in proportion to how much each has earned your trust. The second is belief revision: when new and stronger evidence contradicts a standing conclusion, you retract the conclusion rather than hold both. This is the defeasible reasoning from Part I in action, and it is non-monotonic, which is the striking part. Learning more can make you believe less. Step through it.

When the conflict is between arguments rather than raw data, the rule from Part I returns: an argument with a live, unanswered rebuttal is defeated. Resolving conflict is often deciding which argument survives the attacks on it, not which is stated most loudly. And at the limit there is a deeper hazard. In classical logic a single contradiction is catastrophic, because from a contradiction anything follows, so one bad fact can corrupt the entire knowledge base. A healthy system reasons paraconsistently: it quarantines a contradiction so the rest of its knowledge stays usable while the clash is investigated, rather than letting one inconsistency bring everything down.

The deeper point is that a contradiction is information, not noise. It often points straight at the faulty sensor, the wrong assumption, or the adversarial input. A healthy system surfaces and investigates conflict; the two pathologies are suppressing it, by silently picking a side, and freezing on it, unable to act at all.

When knowledge is missing

The hardest bad-knowledge state is the gap: not what you believe wrongly, but what you do not know at all, including the things you do not know that you do not know.

It begins with a fork that decides everything: the closed-world versus the open-world assumption. Under the closed world, anything not known to be true is taken as false, the way a flight not listed in the database does not exist. Under the open world, what is not known is merely unknown, not false. The real world is open, and treating it as closed is a classic and costly error. In diagnosis it is the difference between "no fault" and "no evidence of a fault." Flip it.

This is why "I have not found it" must never quietly become "it is not there." A healthy system fills gaps only with flagged assumptions and abduction, the best provisional explanation, never with facts it does not have. And when a gap is reducible and the answer matters, the healthy move is to go and get the information, which is the value of information and the experimenter from earlier: distinguish reducible gaps, where you investigate, from irreducible ones, where you decide robustly under them.

Hardest of all is the difference between known and unknown unknowns. The gaps you can name, you can plan for. The gaps you cannot name have only three defenses: margins, robustness, and a watch for surprise, since an anomaly or out-of-distribution alarm is the one signal that warns you have walked into a gap you never charted.

The maintenance loop for thinking

Put the three regimes together and a single discipline emerges, the one this whole series has circled: keep the thinking honest by watching it, diagnosing it, and correcting it, the way you would any system you depend on.

The loop is the familiar one, turned inward on the reasoning itself. Monitor the health signals. Diagnose which regime is being mishandled. Repair, by recalibrating, resolving the conflict, gathering the missing piece, or handing back to a human. Verify that the correction held. It is propose, verify, and gate, applied not to an answer but to the quality of the thinking that produced it.

References and further reading

Sources behind Part V, on uncertainty, evidence, belief revision, and reasoning with incomplete or inconsistent knowledge.

1. Knight, F. H. (1921). Risk, Uncertainty and Profit. The distinction between measurable risk and true uncertainty.
2. Der Kiureghian, A. & Ditlevsen, O. (2009). Aleatory or epistemic? Does it matter? Structural Safety, 31(2).
3. Lichtenstein, S., Fischhoff, B. & Phillips, L. D. (1982). Calibration of probabilities. In Judgment under Uncertainty.
4. Guo, C. et al. (2017). On calibration of modern neural networks. ICML.
5. Pearl, J. (1988). Probabilistic Reasoning in Intelligent Systems. Morgan Kaufmann. Bayesian networks.
6. Shafer, G. (1976). A Mathematical Theory of Evidence. Princeton. Combining evidence under ignorance and conflict.
7. Alchourron, C., Gardenfors, P. & Makinson, D. (1985). On the logic of theory change (AGM). Journal of Symbolic Logic, 50(2).
8. Pollock, J. L. (1987). Defeasible reasoning. Cognitive Science, 11(4).
9. Reiter, R. (1980). A logic for default reasoning. Artificial Intelligence, 13.
10. Reiter, R. (1978). On closed world data bases. In Logic and Data Bases. The closed-world assumption and negation as failure.
11. Priest, G. (1987). In Contradiction. A study of paraconsistent logic.