A Plain-Language Field Guide · Part 2 of 7
← Part I · The Foundations and the Human MindPart II · Limits, Machines, and Problem-Solving
Where logic runs out, how reasoning gets rebuilt inside machines, how to choose under uncertainty and over time, how people really decide, and how the mind solves real problems.
Logic and reasoning rest on foundations that, when you push on them, turn out to be surprisingly unsettled. This is the honest part.
Earlier we saw that an argument can be valid yet false. The full vocabulary is worth having. An argument is valid if the conclusion must follow from the premises. It is sound if it is valid and its premises are actually true. Validity is checkable from the form alone. Soundness needs you to step outside logic and check the world. Logic alone can never tell you whether an argument is sound, only whether it is valid.
Inductive reasoning powers science, prediction, and ordinary expectation. David Hume pointed out something uncomfortable in 1748: there is no non-circular way to justify it. Why believe the future will resemble the past? Because it always has? That argument itself assumes the future resembles the past, so it is circular. We rely on induction every second and cannot give it an airtight defence. We use it because it works, not because it is proven.
Try to justify any belief and you can always ask "why?" again. This regress (named the Agrippan or Munchhausen trilemma) seems to leave only three exits, none fully comfortable.
Even the third door has a famous crack. In 1931 Kurt Godel proved that any formal system powerful enough to describe ordinary arithmetic must contain true statements it can never prove from within itself, and it cannot prove its own consistency. The dream of a single mechanical procedure that settles every mathematical truth is impossible in principle. This matters far beyond mathematics: it is a hard ceiling on what any purely formal reasoning machine, ours or a computer's, can guarantee about itself.
Every human mode of reasoning has a machine counterpart. Building reliable AI is largely the work of choosing the right mode for each job and catching it when it slips.
The same families that organise human thought map cleanly onto engineering. Diagnostics, prognostics, and agentic AI are, underneath, deliberate choices about which kind of inference to run and how much to trust it. The table below lines them up, with a vehicle-health example for each since that is where these distinctions earn their keep.
| Human mode | Machine counterpart | Where it lives in vehicle health |
|---|---|---|
| Deduction | Rule engines, type systems, formal verification, hard safety gates | Threshold and interlock logic that must never be wrong |
| Induction | Supervised learning, generalisation from data | Data-driven remaining-useful-life and prognostic models |
| Abduction | Inference to the best explanation, hypothesis ranking | Fault diagnosis and root-cause isolation from symptoms |
| Bayesian | Probabilistic inference, sensor fusion, belief updating | Survival models and confidence on a fired alarm |
| Analogy | Case-based reasoning, retrieval-augmented generation | Matching a new fault signature to known historical cases |
| Defeasible | Belief revision, non-monotonic agent state | Retracting a diagnosis when fresh telemetry arrives |
It is tempting to read a language model's quick answer as System 1 and its step-by-step "chain of thought" as System 2. There is something to that picture: forcing a model to lay out intermediate steps is, like the bat-and-ball problem, a way of refusing the fast answer and doing the slow work. But the analogy is loose. A model has no separate deliberate faculty switching on; the visible reasoning is itself generated text, which can be fluent and still wrong.
That is precisely why the failure modes look so human. Models confabulate explanations after the fact, anchor on the framing of a prompt, and confirm whatever the question presupposes. These are the same biases from Section 05, reappearing in a different substrate. Designing trustworthy agents means treating their reasoning the way good engineering treats any fallible reasoner: keep humans in the loop at the points that matter, gate the high-stakes deductive steps, and never mistake a confident chain of words for a sound argument.
The oldest ambition in logic was to turn reasoning into pure calculation. It failed in the way it had to, and then succeeded by changing shape. That story is the deepest root of how AI reasons today.
The idea that thinking could be reduced to mechanical symbol-shuffling runs from Leibniz, who imagined a universal calculus in which any dispute could be settled by saying "let us calculate," through Boole's algebra of logic to Frege, who built the first real predicate logic. David Hilbert turned it into a concrete program: write all of mathematics as a symbol game with explicit rules, then prove from the outside that the game is complete (every truth provable) and consistent (no contradiction). If that had worked, reasoning would have been, in principle, a crank you could turn. Mechanized reasoning is the direct heir of this dream.
Section 07 gave you Godel. For the machine story the sharper facts are about decidability. Godel killed completeness and self-certified consistency. Church and Turing then proved that validity in predicate logic is undecidable: no algorithm always halts with a correct yes or no about whether an arbitrary statement is a logical truth. The crucial subtlety is that it is semi-decidable. A proof procedure can systematically confirm every genuine theorem, but on a non-theorem it may simply run forever. So for any reasoning machine, silicon or otherwise, the situation is fixed: you can build a sound, complete proof enumerator, but you cannot bound how long it searches. That one fact is both the license for automated theorem proving and its permanent ceiling.
Engineers routed around the wall by trading completeness or guaranteed termination for things that actually run. Resolution and unification gave automated theorem provers a single mechanizable rule. SAT solvers tame propositional logic (decidable, but hard) with conflict-driven clause learning, and SMT solvers like Z3 extend that to arithmetic and arrays, quietly powering most modern verification. Logic programming turned deduction into a language (Prolog, Datalog). The Curry-Howard correspondence, which says proofs are programs and propositions are types, gave proof assistants like Lean, Coq, and Isabelle, where a machine checks a proof with total rigor even when it cannot find one. And model checking with temporal logic exhaustively verifies finite-state systems against a specification, which is the branch closest to safety-critical engineering.
Now place this against the certainty hierarchy from Section 03. Everything above is the deductive, guarantee-bearing machinery. Large language models sit at the opposite pole: they are fundamentally inductive engines trained to continue plausible text. They excel at the ampliative, pattern-matching modes and are structurally weak at sound deduction, because nothing in next-token prediction enforces validity. Chain-of-thought is not proof search; it is heuristic search through token space that resembles reasoning and carries no soundness guarantee. That is exactly why confident, fluent, wrong multi-step reasoning is the signature failure, the machine echo of the System 1 overreach from Section 05.
The reliable systems refuse to pick a side. They exploit a deep asymmetry: finding a proof is hard, but checking one is cheap and total. So a fallible neural component proposes and a sound formal component disposes. Each one's weakness is the other's strength. Step through the loop below to see the checker catch a plausible but unjustified move that the proposer was happy to assert.
This is no longer theoretical. Open neural theorem provers built on the Lean 4 proof assistant now solve competition-level mathematics by exactly this loop, training on a binary correct-or-incorrect signal that the verifier emits.
The recurring recipe is striking: a model decomposes a hard problem into subgoals, drafts candidate proof steps, and the Lean checker accepts only what truly follows, with reinforcement learning driven entirely by that verifier's verdict. The same idea generalises beyond mathematics. Tools like solver-backed frameworks translate a natural-language problem into formal logic, hand it to Z3 or a similar engine, and use the solver's feedback to repair their own mistakes. Autoformalization, the translation of informal statements into machine-checkable form, is the bridge that makes all of it possible.
The pattern is directly useful well outside mathematics. Runtime verification is just checking an execution trace against a temporal-logic specification, which is the formal-methods view of a health monitor. Safety-critical automotive and avionics work increasingly leans on model checking and SMT-backed verification to discharge requirements. A deductive gate sitting downstream of a fallible agent is the generate-and-verify pattern in miniature: let the agent propose, let a small verifier guarantee. And turning written requirements into checkable formal properties is autoformalization applied to engineering rather than to olympiad algebra. The lesson is the same at every scale. Do not look for a flawless reasoner. Place each kind of inference where it belongs, and build the check that catches it when it drifts.
Everything so far was a part. This is the whole. Real problem-solving is a loop that orchestrates logic, all three reasoning modes, memory, experiment, and critical thinking at once.
None of the mental acts in this guide works alone. Solving a problem means conducting them together, and the cleanest way to see how they fit is one idea from Allen Newell and Herbert Simon: a problem is a starting state, a goal state, and a set of moves between them. Solving it means finding a path. The catch is that the space of possible move-sequences is astronomically large, so the real story of human problem-solving is not raw search but every trick the mind uses to avoid searching most of it.
Not every problem yields to the same approach, and recognising the type is itself part of solving it. Three distinctions carry most of the weight.
A sudoku has fixed rules, states, and a goal. "Write a strong proposal" has none of these nailed down, so much of the work is just defining the problem at all.
Some problems fall to steady step-by-step search. Others sit at zero progress until a sudden reframe makes the whole thing click at once.
A logic puzzle needs little outside knowledge. Diagnosing a fault needs a great deal, so stored expertise matters far more than raw search.
The three modes from Section 03 are not tools you choose between. They turn as a cycle. Abduction is the creative front end that generates candidate ideas, guesses about what might work. Deduction works out the consequences of a candidate so you can test it in your head before paying to try it. Induction reads patterns from what you have already seen and biases which candidates are even worth generating. Guess, derive, test, generalise, guess again. That loop is the scientific method, and it is also what you run when you debug code or trace a fault.
Past knowledge is the biggest lever, and it is the main thing separating an expert from a novice. The classic chess studies found that masters do not search more moves than amateurs; they search better, because memory hands them large pre-compiled patterns that prune the space at a glance. Expertise is mostly a deep library of recognised situations with moves attached. Analogy is the active form: mapping the structure of a problem you have already solved onto a new one that merely looks different on the surface.
On top of knowledge sit heuristics, the shortcuts that replace exhaustive search. Means-ends analysis (keep reducing the gap between where you are and the goal), working backward from the goal, and breaking one hard problem into subgoals are the workhorses. Polya's four steps still hold: understand the problem, devise a plan, carry it out, then look back. And Simon's bounded rationality runs through all of it. We do not optimise, we satisfice, stopping at the first good-enough answer because the perfect one usually costs more than it is worth.
Every one of these moves runs through a tiny workspace. Human working memory holds only a handful of items at once, the famous "magical number seven, plus or minus two," and in practice often fewer. This one constraint explains a surprising amount. It is why we break problems into subgoals, so each piece fits in the workspace. It is why experts feel less overloaded, since each of their chunks counts as a single item. And it is why we reach for paper, diagrams, and whiteboards. Writing a problem down is not just record-keeping. It offloads the workspace onto the world so the mind can work on something larger than it could ever hold at once. Most serious problem-solving is this partnership between the head and an external scaffold.
Here is the central claim made visible. The same maze, solved two ways. Blind search expands outward in all directions with no sense of where the goal is. Heuristic search uses a cheap estimate of distance-to-goal to head the right way. Both find a path. Look at how many squares each one has to examine to get there.
Blind search is correct and finds the shortest path, but it examines almost the whole grid. The heuristic examines only a fraction of it. It carries no guarantee of the shortest route, though on an open layout like this it often lands on one anyway. That trade, far less effort for a weaker promise, is satisficing in one picture, and it is most of what makes human problem-solving fast enough to be useful.
How you encode a problem can be the whole difference between impossible and trivial. The classic case: take a chessboard, remove two opposite corner squares, and try to cover the remaining 62 squares with 31 dominoes, each covering two side-by-side squares. People shuffle placements for a long time and fail. The right representation settles it in one line.
Stop thinking about squares and think about colour. Every domino, however you lay it, covers exactly one light and one dark square. The two corners you removed are the same colour, so the board now has 32 of one colour and 30 of the other. Thirty-one dominoes would need 31 of each. It cannot be done, and you never had to try a single arrangement. The brute-force search was enormous; the right representation made it vanish. Choosing how to see a problem is often the most powerful single move available.
Means-ends analysis and subgoaling are easiest to feel by doing. Move the whole stack from the left peg to the right. Only the top disk of a peg moves at a time, and a larger disk may never rest on a smaller one. The key that unlocks it is recursive subgoaling: to move the big stack, first get everything above it out of the way onto the spare peg. Try it.
Notice the cost. The fewest moves needed is two to the power of the number of disks, minus one, so each disk you add roughly doubles the work. Brute force is hopeless even for modest sizes. Yet the recursive idea ("move the top n minus one aside, move the big one, then move the stack back") compresses that exponential mess into a rule a child can follow. That is decomposition earning its keep: a problem too big for the workspace becomes a short stack of problems that each fit.
Much of what looks like creativity is really analogy: solving a new problem by mapping it onto an old one with the same deep structure. The famous demonstration uses two stories. A general must take a fortress, but every road to it is mined to blow if a large force crosses, so he splits his army into small groups that converge from all directions at once. Separately, a doctor must destroy a tumour with radiation strong enough to kill it, but that dose would destroy the healthy tissue it passes through. The solutions are the same idea wearing different clothes.
The catch is that analogy is powerful but fragile. In the original studies, only about one person in ten spontaneously used the fortress story to crack the radiation problem unless they were told the two were related. We store solutions by their surface details, not their deep structure, so the useful match sits right there and we walk past it. Training in any field is largely about learning to file problems by structure instead, which is exactly what lets an expert say "this is really the same as that."
Sometimes thinking is not enough and you have to act on the world to learn. This is the explore-exploit tradeoff: spend moves probing for information, or commit to the best option you have so far. The quality of your probes matters, and it is where most people quietly fail. Wason's number-sequence experiments showed that when testing a hypothesis, people overwhelmingly look for evidence that confirms it and almost never try to break it. Strong experimentation deliberately designs tests that could prove you wrong. It is the most valuable and least natural move in the whole repertoire.
Three machines pay out at different unknown rates. You have a limited run of pulls. Find the best machine, but remember: every pull spent testing a weak one is a pull you did not spend earning from the best one. When do you stop exploring and start exploiting?
The tension is real and unavoidable. Explore too little and you may exploit the wrong option forever, confident and wrong. Explore too much and you spend your whole budget learning and never cash in. There is no formula that dissolves this, only better and worse balances, which is why it shows up everywhere from choosing a restaurant to allocating a research budget. Good problem-solvers feel the shift: probe aggressively when the cost of a mistake is low and information is cheap, then commit once the evidence is in.
If expertise is mostly a library of recognised patterns, the obvious question is how the library gets stocked. The answer is not raw hours but deliberate practice: repeated work at the edge of your ability, on problems just hard enough to fail at, with feedback that tells you exactly what went wrong. Over time, small chunks combine into larger schemas, and whole classes of problem collapse into a single recognised template. A revealing study asked novices and expert physicists to sort problems into groups. Novices grouped by surface features, the problems that mentioned springs together, the ones with inclined planes together. Experts grouped by deep principle, putting an inclined-plane problem and a spring problem in the same pile because both were really about conservation of energy. That shift, from sorting by how a problem looks to sorting by what it is, is most of what expertise is, and it is why an expert's first representation of a problem is already half the solution.
Critical thinking is the layer that watches the loop, and in the dual-process terms of Section 05 it is System 2 auditing System 1. It asks: am I making progress or just staying busy? Is this assumption load-bearing or am I taking it on faith? What would change my mind? Is this evidence any good? It catches the biases the earlier sections catalogued and decides when to abandon a failing strategy. A few quieter acts round it out: choosing a good representation (the same problem is intractable in one framing and obvious in another), insight and restructuring (the sudden reframe that escapes a stuck approach, often after stepping away to incubate), and abstraction (stripping detail to see the underlying structure).
The same machinery that makes us efficient also has predictable failure modes. Most stuck problem-solving is one of these traps rather than a lack of raw intelligence. Tap each to see it and how to break out.
You need to prop a door but only "see" the heavy stapler as a stapler, never as a doorstop.
Break out: describe objects by their properties (heavy, flat, rigid) rather than their names, which frees them for new uses.
A method that worked on the last five problems gets reused on a sixth where a far simpler solution exists, and you miss it entirely.
Break out: after solving, deliberately ask "is there a simpler way?" The familiar path actively hides the better one.
You test only cases you expect to pass, so the bug that breaks your assumption is never the one you check.
Break out: design the test that would prove you wrong, not the one that would prove you right.
The first plausible explanation feels like the answer, so the search stops before the real cause is found.
Break out: force at least one rival hypothesis before committing. "What else could do this?"
Pure hill-climbing reaches a small peak and stops, because every next step looks worse, even though a much higher peak is nearby.
Break out: accept a temporary step backward, or restart from a different point, to escape the false summit.
Endless gathering and weighing of options with no commitment, as the cost of deciding quietly exceeds the cost of a wrong choice.
Break out: set a stopping rule in advance. Satisfice on purpose: good enough, decided by a deadline.
Abstract loops are easy to nod along to and hard to use, so here is the whole cycle on one concrete problem: a car that will not start. Watch every earlier idea show up in sequence.
Frame it precisely. Not "the car is broken" but "the engine cranks but does not catch." That reframing already rules out a dead battery and points at fuel or spark.
Pull up what you know. Engines need fuel, air, spark, and compression. Recall similar past cases and the analogy that a flooded engine behaves a certain way.
Generate candidate explanations that would account for "cranks but will not catch": no fuel reaching the cylinders, or no spark. Two live hypotheses.
Reason out a consequence you can check. If it is fuel, then the rail pressure will read low. If it is spark, a pulled plug will be dry and show no arc. Each hypothesis makes a different prediction.
Run the cheap, discriminating test first. Pull a plug. It is wet with fuel, so fuel is arriving. That single probe was designed to split the two hypotheses, not to confirm a favourite.
Fuel is present, so the spark hypothesis survives and fuel is ruled out. Critical thinking checks the inference, then sends you back to step 3 with a sharper question: why no spark? Coil, or sensor? The loop turns again, now over a much smaller space.
That is the entire monograph in miniature. Representation narrowed the space, memory supplied the candidates, abduction proposed, deduction turned each proposal into a testable prediction, a well-chosen experiment cut the options in half, and critical evaluation decided what survived and aimed the next pass. Real diagnosis spirals through this loop several times, each turn cheaper than the last. It is exactly the structure behind systematic fault isolation in engineered systems, where the same discipline is written down as a procedure so it does not depend on any one person remembering to do it.
Put together, a real episode runs like the diagram below, with metacognition steering the entire time and a fallback path for when you get stuck.
Reasoning tells you what is likely. It does not tell you what to do. Turning beliefs into a choice, when you are unsure and the outcomes differ in how much they matter, is the job of decision theory.
The core idea is simple and strong. Weigh each action by the value of its outcomes and how likely those outcomes are, then pick the action with the best expected value. A 10 percent chance of a costly failure against a 90 percent chance of nothing is not "probably fine"; it is a weighted average you have to compute, because the costs are not equal. This is expected utility, and it is the bridge from what you believe to what you do.
It also forces a distinction the everyday word hides. Risk is when you know the odds, like a fair die. Uncertainty proper is when you do not even know the odds, which is the usual case in the real world, and it calls for caution, robustness, and gathering more information rather than a confident sum.
Gathering more information has its own price and its own payoff, and decision theory makes the trade exact. The value of information is how much a piece of evidence can improve your decision, measured as the expected cost it lets you avoid. A test is worth running only when its value of information exceeds what it costs. A test that cannot change what you would do is worth nothing, however cheap. A test that flips an expensive decision can be worth a great deal. Work a case.
A fault is either a sticking valve (60 percent, repair costs 2) or a bad sensor (40 percent, repair costs 5). Fix the wrong part and you pay for it and still have to do the right repair. A test costs 0.5 and reveals the true fault.
This is the first-principles backbone of the experimenter from the problem-solving loop, and of the "which test do I run next" skill an agent can learn: at each step, choose the test whose value of information per unit cost is highest. A reinforcement-learning agent trained on information-gained-per-cost is, in effect, learning a fast amortised estimate of value of information. The principle says what is optimal; the learned skill makes it cheap.
Expected value treats a guaranteed 50 and a coin flip for 0 or 100 as equal. People do not, and often should not. We maximize expected utility, not expected value, and utility usually curves: the second 50 is worth less than the first, so a sure thing can rightly beat a fair gamble with the same average. That curvature is risk aversion; where the curve bends the other way you get risk seeking. The shape is not a flaw in reasoning. It reflects that outcomes matter in proportion to what they change for you, not their face value, which is why a possible catastrophic loss is weighted far more heavily than its raw expected cost.
Expected utility needs probabilities. When you truly lack them, you fall back on rules that do not require them, trading some average performance for protection against being badly wrong, which is usually the right trade when a failure is catastrophic.
| Rule | Picks the action that | Use when |
|---|---|---|
| Maximin | has the least-bad worst case | a failure would be catastrophic |
| Minimax regret | you will least regret in hindsight | you fear looking back on a clear mistake |
| Robust | does acceptably across many possible worlds | the model itself is uncertain |
Real decisions rarely optimize a single number. Cost, reliability, weight, and latency pull against each other. The organizing idea is the Pareto frontier: the set of options where you cannot improve one objective without giving up another. Everything off the frontier is simply worse and can be discarded. Choosing among the points on the frontier is a value judgment about the trade, which belongs to a human, not to the optimizer.
A single choice is the easy case. Most real decisions are sequences, where today's action changes tomorrow's options, and a move that looks best right now can be a trap. Handling that is the heart of control and of reinforcement learning.
The standard frame is the Markov decision process. You are in a state, you choose an action, you receive a reward and land in a new state, and you repeat. The goal is not the best immediate reward but the best total reward over the whole sequence. That is what makes it hard: a good action now can set up a bad state later, so you cannot simply decide greedily step by step.
The breakthrough is to define the value of a state as the best total reward obtainable from it onward, and to notice that value obeys a recursion. The value of a state is the immediate reward plus the value of the best state it leads to. That self-referential rule is the Bellman equation, and it lets you solve a long sequence by stitching together one-step choices, working backward from the goal. This is exactly the value function asserted in Part III. Here is where it comes from.
Acting over time also forces a tension the one-shot case never had: do you exploit the best option you know, or explore an uncertain one that might be better? Explore too little and you lock onto a mediocre habit; explore too much and you waste effort. The bandit game back in the problem-solving section was this tension in miniature, and balancing it well is, formally, a value-of-information calculation spread across time.
Everything so far has been normative, the ideal a perfect agent would follow. People depart from it, and the departures are regular enough to predict, which makes them part of the picture rather than mere noise.
Kahneman and Tversky's prospect theory captures the main ones. We judge outcomes as gains and losses from a reference point, not as final states, so the same result feels different depending on where you started. Losses loom larger than equal gains, by roughly two to one, which is loss aversion. We overweight small probabilities and underweight large ones. And because of all this, the framing of a choice, as a gain to secure or a loss to avoid, can flip the decision even when the options are identical.
The framing effect is easiest to feel by living it. Below is one set of options, identical in outcome, described two ways. Try each.
A fleet of 600 units will fail without intervention. Two repair programs, A and B, with identical real outcomes, described either as units saved or units lost.
The numbers never changed; 200 saved is 400 lost. The frame moved the choice. These are not signs of stupidity but the fingerprints of a mind built for scarcity and real stakes. They matter for design: present the same facts two ways and you will get two decisions, so a system reporting to humans must watch its own framing, and an agent that learns from human feedback will inherit these biases unless you correct for them.
Reasoning is not free. Every step costs time and computation, and at some point more thinking stops being worth it. Deciding how much to think is itself a decision, and a good reasoner makes it well.
This is metareasoning, reasoning about your own reasoning. The resource-rational view says the right amount of deliberation is the amount whose expected improvement in the decision exceeds its cost. That is value of information turned inward: keep thinking only while the next thought is likely to change the answer by more than it costs to have. It explains why experts often decide faster than novices, not despite knowing more but because they know which thoughts are worth having and skip the rest.
This closes the loop on the reasoning-strategies section. Choosing chain-of-thought versus a full tree search is a metareasoning call, and an orchestrator that escalates effort only for high-stakes problems is doing exactly this. Anytime algorithms, which return a usable answer whenever they are interrupted and improve it given more time, are the engineering expression of the same idea.
Five quick checks. Pick an answer to see whether it holds and why.