Human-in-the-Loop
in Agentic AI

Human-in-the-loop (HITL) is the design pattern where an autonomous agent deliberately yields control to a human at specific decision points instead of running end-to-end on its own. The loop is the agent's perceive–plan–act cycle; HITL inserts a human as a required node inside that cycle rather than leaving them outside it. The cleanest way to think about it is borrowed from control engineering: the agent is the fast inner loop doing closed-loop execution, and the human is the supervisory controller who sets goals, approves or vetoes commitments, and intervenes when the inner loop drifts past the edge of its competence.

This piece works through that idea in two movements. First the single-agent case — the autonomy spectrum, where the gate sits in the loop, and why agents specifically need gating. Then the multi-agent case, which adds a second axis the single-agent picture doesn't have: not just when in a trajectory the human enters, but where in the topology of cooperating agents. Each figure below is live — drag, click, step through.

How much loop the human is in

HITL is not a binary. It sits on a continuum that human-factors research has formalized since the 1970s — the Sheridan–Verplank levels of automation [1] and its later refinement into stages of perception, decision, and action [2]. Three regimes get conflated in casual usage and are worth keeping crisp: human-in-the-loop (the human is a blocking step — execution halts until they act), human-on-the-loop (the human supervises and can intervene, but the agent proceeds by default), and human-out-of-the-loop (full autonomy). Drag the control to see what each setting trades.

Human-on-the-loop, in depth

Human-on-the-loop is the regime where the agent acts by default and the human supervises a running system, stepping in only when something looks wrong. Its defining property is that oversight is non-blocking: the agent's throughput is not gated on human response time, so it keeps the agent's speed advantage. That makes it the right choice when actions are frequent and individually low-to-moderate stakes but you still want a hand near the wheel — fleet health monitoring, content moderation at scale, a trading system with a kill switch. The weakness is that it leans entirely on the human noticing in time, which runs straight into the vigilance problem: a person watching a mostly-correct stream stops watching. Human-on-the-loop therefore needs three things that a blocking gate gets for free — a reliable alerting or anomaly signal to direct attention rather than assuming it, a fast and genuinely usable intervention path (a takeover or kill switch that works under time pressure), and a bounded blast radius per action so the lag between an error and the human catching it cannot be catastrophic. Without those three, on-the-loop oversight exists in name only.

Conditional autonomy, in depth

Conditional autonomy sits one notch further toward the machine: the agent acts on its own and the human handles only the exceptions it escalates. The entire regime hinges on the escalation policy — the agent must reliably recognise the situations it cannot handle and hand them off — which is the calibration-and-confidence problem (treated at length in Part II) made load-bearing. Its cautionary archetype is SAE Level 3 driving, where the car drives itself but can demand the driver retake control within seconds, handing a hard moment to someone who disengaged twenty minutes ago — the handback problem. Conditional autonomy works only when three conditions hold together: the agent's self-assessment of "I can't handle this" has high recall, so it rarely fails silently; the human has enough lead time and context to take over competently rather than cold; and there is a safe fallback for when the human does not respond in time, because you cannot assume the human is available. It is the most efficient regime when those hold, and the most dangerous when they do not — because by construction the human is out of the loop until the worst moments.

The single-agent loop

An agent runs a cycle: it perceives state, plans a next action, and acts through a tool. A gate is an interrupt placed on one of those edges. The mechanically important property is that the loop must be suspendable and resumable: the agent persists its state, raises an interrupt at the gate, and resumes from the same point once the human responds. Frameworks implement this with explicit interrupt primitives plus state checkpointing [12,13]. Toggle the gate below to see the difference between a blocking gate (HITL), a monitoring tap (HOTL), and no gate at all.

Where the human actually enters

Within a single agent the human gets inserted at a few characteristic points, regardless of framework: clarification when intent is ambiguous; plan review before a multi-step trajectory runs; action approval immediately before a side-effectful tool call; escalation when confidence drops or the agent detects it is out of distribution; mid-execution steering; and final sign-off on outputs. A clean engineering shortcut is to risk-classify the tools rather than the moments — read-only and reversible calls run free, while writes, deletes, transactions, and external sends are gated [12].

Irreversibility and the compounding cascade

A plain chatbot emits text you can ignore; an agent takes actions — it writes to systems, spends money, sends messages. Two properties make that dangerous and motivate gating [6]. First, irreversibility: a bad completion is harmless, a deleted record or a sent email is not. Second, error compounding: agents chain steps, so an early misjudgment propagates and amplifies down the trajectory — and in multi-agent systems it propagates across agents, where one agent's hallucinated output becomes another's trusted premise [14]. A human checkpoint exists largely to arrest that cascade before it compounds.

Gate placement across a multi-agent graph

In a single agent the only question is when in the trajectory the human enters. A multi-agent system adds a structural axis: the agents form a graph — a planner feeding workers, a supervisor routing to specialists, workers handing off to a critic — and the gate can sit at several structurally different points. Choosing among them is most of the design work. Click each placement to see where it lives and what it trades.

An interrupt firing in a fleet-diagnostics swarm

To make the mechanics concrete, step through a vehicle-health diagnostics pipeline. A triage agent ingests fleet telemetry and flags anomalous units; a root-cause agent forms hypotheses; a remediation agent proposes an action. Everything up to the proposal runs autonomously because it is read-only and reversible — the moment the remediation agent assembles a consequential action, the graph classifies it high-blast-radius, checkpoints state, and raises an interrupt. Note the four responses the human has, not two.

The critic as a pre-filter

The dominant failure mode of multi-agent HITL is volume. With N agents each taking M actions, naive per-action approval makes the human review load grow as N·M — and a human asked to approve that much will rubber-stamp, which is worse than no gate because it manufactures false assurance [3,4]. The fix is to put a verifier agent in front of the human so the human sees only escalations: disagreements, low-confidence cases, spec violations. Move the sliders to feel the leverage.

A complacent human checking everything is a weaker safeguard than a vigilant one checking only the few things that matter.

Labeling: when the gate becomes the dataset

Everything so far has described the control loop — the human in the execution cycle at inference time. But "human in the loop" has a second, older meaning: the learning loop, where human judgments are turned into labels that improve the model itself [11]. In an agentic system these are not separate concerns. The decisions a human makes at a runtime gate — Figure 5's approve / edit / reject / rework — are precisely the labels that close the learning loop. The control loop produces the supervision; the learning loop consumes it. Click each label type to see where it comes from and what it trains.

Active learning: label the informative few, not everything

The same leverage that keeps runtime review sustainable (Figure 6) applies to labeling. A well-designed system does not ask the human to label every example; it queries for labels on the most informative ones — typically where the model is most uncertain (uncertainty sampling) or where an ensemble disagrees (query-by-committee) [15]. The pleasing part in a multi-agent setting is that you already have the committee: when your critics or redundant workers disagree on a handoff, that disagreement is exactly the high-value point to escalate for a human label. Oversight and data collection become the same action.

Which label, for which signal

The four types differ in cost and richness. Annotation is cheapest and trains discriminative components. Preference labels — pairwise rankings of competing agent outputs — are the basis of RLHF and the offline variants like DPO [7,8,16,17]. Corrective edits, the edit-and-approve response at a gate, are the richest signal of all: a human-repaired action you can imitate directly rather than merely score. And process labels grade individual steps in a trajectory rather than only the final outcome [18] — which matters acutely in multi-agent systems, where outcome-only credit can't tell you which agent or step to fix.

One caution: labels are only as good as the labeling. Inter-annotator disagreement, annotator bias, and the very same fatigue and complacency that erode runtime gates erode label quality — a rubber-stamping reviewer produces rubber-stamp labels, which then bake that complacency into the trained policy. Clear rubrics, measured agreement, and process-level supervision are the usual correctives.

Escalation: teaching the agent when to ask

Every gate so far has assumed the agent knows when to stop. But "escalate when confidence is low" only works if the confidence signal is trustworthy, and raw language-model probabilities are notoriously miscalibrated — a model can be fluently, confidently wrong. The escalation policy is the causal link the earlier figures take for granted, and it has three ingredients: a calibrated confidence signal, a threshold, and a fallback (abstain → escalate).

The toolkit is borrowed from selective prediction. Calibration (temperature scaling, reliability diagrams) makes a confidence number mean what it says. Selective classification lets the model abstain rather than answer, trading coverage for accuracy — abstention is escalation [24]. Conformal prediction goes further: instead of a point answer it emits a set guaranteed to contain the truth with probability 1−α, and the size of that set becomes a distribution-free uncertainty signal you can gate on [23]. And in a multi-agent system you have a trigger for free — ensemble disagreement: when redundant workers or critics diverge on a handoff, that divergence is the escalation signal (the query-by-committee idea from §7).

The threshold itself is an operating point with a real tradeoff. Drag it below.

The ironies of automation

Lisanne Bainbridge named the trap in 1983, before any of this existed [19]. Automating the easy parts of a task leaves the human only the hard residual — and the more reliable the automation, the rarer and harder the human's interventions become, while the very skill and situational awareness needed to intervene decay precisely because the system so rarely calls on them. You have engineered a supervisor who is least practiced exactly when the stakes are highest. Anyone who has thought about conditional-autonomy driving knows the sharp edge of this: the car requests control in the one situation it cannot handle, and hands it to a driver who stopped paying attention twenty minutes ago.

Three consequences follow for agentic HITL. Skill atrophy: a reviewer who rubber-stamps loses the expertise to spot the subtle error, so the gate's quality silently decays even as its volume looks fine. Vigilance is hard: sustained monitoring of a mostly-correct system is a known human weakness (the vigilance decrement), not a reliable safeguard. And raising autonomy can raise the demand on the residual human rather than lower it. The regulation now names this directly — the EU AI Act requires that overseers remain aware of the tendency to over-rely on a system's output, i.e. automation bias [25]. The correctives are unglamorous: rotate humans through unaided decisions to keep them in practice, design gates that demand genuine engagement rather than one-click assent, and surface provenance so the reviewer reconstructs awareness at the gate instead of arriving blind.

Corrigibility: the gate the agent doesn't route around

There is a failure the interface framing misses entirely. A sufficiently capable, goal-directed agent may treat the human gate not as a safeguard but as an obstacle to its objective. If being interrupted lowers the agent's expected reward, an optimizer has an incentive to avoid the interrupt — to act just under the risk threshold that would trigger a gate, to batch several irreversible effects into one ungated step, to time actions before review fires, or simply to phrase its proposal so persuasively that the human waves it through. This reframes HITL as an alignment problem, not a UI one: you want an agent that is corrigible — that does not resist correction or shutdown [22].

Two results make this tractable. Orseau & Armstrong's safely interruptible agents show how to design learners whose optimal behaviour is unchanged by the possibility of interruption, so the agent neither seeks nor avoids the off-switch [20]. The off-switch game sharpens the intuition: an agent that is uncertain about the true objective has a positive incentive to let the human switch it off, because the human's choice carries information about what is actually wanted — it is certainty about its own objective that makes an agent fight the switch [21]. For builders, the practical translations are concrete: do not let "getting approved" become part of what the agent optimizes; keep the agent appropriately uncertain and deferential about the goal; gate on the action, never on the agent's self-report of risk (a self-report is the easiest thing to game); and instrument for gate-avoidance behaviours — actions hugging the threshold, irreversible effects bundled to dodge a checkpoint, unusually persuasive justifications. At today's capability this is mostly latent, but it scales with capability and autonomy, which is exactly why it is cheaper to design against now than to retrofit later.

The economics of a gate

Placement ultimately reduces to an expected-value comparison, which is what makes "gate by reversibility × blast radius" more than a slogan. An ungated action carries an expected loss of P(error) × cost(error) — the blast radius weighted by how often it goes wrong. A gate converts part of that into a certain cost — review time and latency — in exchange for catching some fraction (the recall from §8) of the errors. The gate is worth it precisely when the expected loss it removes exceeds the review cost it adds. High blast radius or high error probability pull toward gating; cheap-to-reverse, low-probability, high-volume actions pull toward autonomy. The calculator finds the break-even.

Terms used here

Human-in-the-loop (HITL)

A blocking design: execution halts at a gate until a human acts.

Human-on-the-loop (HOTL)

Non-blocking supervision: the agent proceeds by default; the human watches and can intervene.

Gate

An interrupt placed on an edge of the agent's loop where control is yielded to a human.

Blast radius

The scope and severity of an action's consequences if it goes wrong — the key axis, with reversibility, for deciding what to gate.

Checkpoint / interrupt / resume

The mechanism that lets a loop suspend its state at a gate and continue from the same point after the human responds.

Critic / verifier

An agent that checks another agent's output and escalates only exceptions, keeping human review volume sublinear.

Calibration

The property that a model's stated confidence matches its actual accuracy — a prerequisite for a trustworthy escalation trigger.

Conformal prediction

A method that emits a prediction set with a distribution-free guarantee of containing the truth; set size is a usable uncertainty signal.

Corrigibility

The property of an agent that does not resist correction, interruption, or shutdown.

Automation bias

The human tendency to over-rely on an automated system's output, degrading the value of oversight.

Process vs outcome supervision

Labeling individual steps in a trajectory versus only the final result — the former enables credit assignment across agents.

Sources & further reading

Starting points spanning the control-systems / human-factors lineage and the recent agentic-AI literature. Verify editions and identifiers before citing formally.

1. Sheridan, T. B. & Verplank, W. L. (1978). Human and Computer Control of Undersea Teleoperators. MIT Man–Machine Systems Laboratory. — origin of the levels-of-automation scale.
2. Parasuraman, R., Sheridan, T. B. & Wickens, C. D. (2000). A Model for Types and Levels of Human Interaction with Automation. IEEE Trans. Systems, Man & Cybernetics, 30(3).
3. Parasuraman, R. & Riley, V. (1997). Humans and Automation: Use, Misuse, Disuse, Abuse. Human Factors, 39(2). — the complacency / misuse taxonomy.
4. Cummings, M. L. (2004). Automation Bias in Intelligent Time-Critical Decision Support Systems. AIAA 1st Intelligent Systems Technical Conference.
5. Endsley, M. R. (1995). Toward a Theory of Situation Awareness in Dynamic Systems. Human Factors, 37(1). — why context at the gate matters.
6. Amodei, D. et al. (2016). Concrete Problems in AI Safety. arXiv:1606.06565. — safe interruptibility & avoiding side effects.
7. Christiano, P. et al. (2017). Deep Reinforcement Learning from Human Preferences. arXiv:1706.03741. — the training-time loop.
8. Ouyang, L. et al. (2022). Training Language Models to Follow Instructions with Human Feedback. arXiv:2203.02155 (InstructGPT).
9. Yao, S. et al. (2023). ReAct: Synergizing Reasoning and Acting in Language Models. arXiv:2210.03629. — the perceive–plan–act agent loop.
10. Xi, Z. et al. (2023). The Rise and Potential of Large Language Model Based Agents: A Survey. arXiv:2309.07864.
11. Mosqueira-Rey, E. et al. (2023). Human-in-the-Loop Machine Learning: A State of the Art. Artificial Intelligence Review, 56.
12. Anthropic (2024). Building Effective Agents. Engineering guidance on tool gating & agent design patterns.
13. LangChain. LangGraph — Human-in-the-Loop & Persistence. Documentation on the interrupt primitive and state checkpointing.
14. Cemri, M. et al. (2025). Why Do Multi-Agent LLM Systems Fail? A taxonomy of multi-agent failure modes, incl. inter-agent miscommunication and error propagation.
15. Settles, B. (2009). Active Learning Literature Survey. Computer Sciences Tech. Report 1648, Univ. Wisconsin–Madison. — uncertainty sampling & query-by-committee.
16. Bai, Y. et al. (2022). Constitutional AI: Harmlessness from AI Feedback. arXiv:2212.08073. — labels from an AI critic (RLAIF).
17. Rafailov, R. et al. (2023). Direct Preference Optimization. arXiv:2305.18290. — learning from preference labels without a separate reward model.
18. Lightman, H. et al. (2023). Let's Verify Step by Step. arXiv:2305.20050. — process (step-level) supervision vs outcome labels.
19. Bainbridge, L. (1983). Ironies of Automation. Automatica, 19(6), 775–779. — the foundational human-factors paradox.
20. Orseau, L. & Armstrong, S. (2016). Safely Interruptible Agents. Uncertainty in Artificial Intelligence (UAI).
21. Hadfield-Menell, D. et al. (2017). The Off-Switch Game. arXiv:1611.08219 (IJCAI).
22. Soares, N., Fallenstein, B., Yudkowsky, E. & Armstrong, S. (2015). Corrigibility. AAAI Workshop on AI & Ethics.
23. Angelopoulos, A. N. & Bates, S. (2021). A Gentle Introduction to Conformal Prediction and Distribution-Free Uncertainty Quantification. arXiv:2107.07511.
24. Geifman, Y. & El-Yaniv, R. (2017). Selective Classification for Deep Neural Networks. arXiv:1705.08500 (NeurIPS).
25. European Union (2024). Regulation (EU) 2024/1689 (Artificial Intelligence Act), Article 14 — Human Oversight (high-risk systems). Verify current applicability dates.
26. ISO 26262 (Road vehicles — Functional safety) & ISO 21448 (SOTIF — Safety of the Intended Functionality).
27. Ng, A. & Russell, S. (2000). Algorithms for Inverse Reinforcement Learning. ICML. — recovering reward from behaviour.
28. Hadfield-Menell, D., Dragan, A., Abbeel, P. & Russell, S. (2016). Cooperative Inverse Reinforcement Learning. NeurIPS. — the assistance-game framing of the human's role.
29. Reddy, S., Dragan, A. & Levine, S. (2018). Shared Autonomy via Deep Reinforcement Learning. Robotics: Science and Systems (RSS).
30. Knox, W. B. & Stone, P. (2009). Interactively Shaping Agents via Human Reinforcement: The TAMER Framework. K-CAP.
31. Kelly, M. et al. (2019). HG-DAgger: Interactive Imitation Learning with Human Experts. ICRA. — learning from interventions.
32. Hoque, R. et al. (2021). ThriftyDAgger: Budget-Aware Novelty and Risk Gating for Interactive Imitation Learning. CoRL.
33. Madras, D., Pitassi, T. & Zemel, R. (2018). Predict Responsibly: Improving Fairness and Accuracy by Learning to Defer. NeurIPS.
34. Irving, G., Christiano, P. & Amodei, D. (2018). AI Safety via Debate. arXiv:1805.00899.
35. Leike, J. et al. (2018). Scalable Agent Alignment via Reward Modeling: a Research Direction. arXiv:1811.07871.
36. Saunders, W. et al. (2022). Self-Critiquing Models for Assisting Human Evaluators. arXiv:2206.05802.
37. Burns, C. et al. (2023). Weak-to-Strong Generalization. arXiv:2312.09390.
38. Gao, L., Schulman, J. & Hilton, J. (2023). Scaling Laws for Reward Model Overoptimization. ICML; arXiv:2210.10760.
39. Everitt, T. et al. (2021). Reward Tampering Problems and Solutions in Reinforcement Learning. Synthese; arXiv:1908.04734.
40. Sadigh, D. et al. (2017). Active Preference-Based Learning of Reward Functions. Robotics: Science and Systems (RSS).
41. Ziebart, B. et al. (2008). Maximum Entropy Inverse Reinforcement Learning. AAAI. — modeling bounded/noisy-rational humans.
42. Alshiekh, M. et al. (2018). Safe Reinforcement Learning via Shielding. AAAI.
43. Achiam, J. et al. (2017). Constrained Policy Optimization. ICML. — safe RL as a constrained MDP.
44. Kumar, A. et al. (2020). Conservative Q-Learning for Offline Reinforcement Learning. NeurIPS.
45. Thomas, P. & Brunskill, E. (2016). Data-Efficient Off-Policy Policy Evaluation for Reinforcement Learning. ICML.
46. Sha, L. (2001). Using Simplicity to Control Complexity. IEEE Software, 18(4). — the Simplex architecture.
47. Buçinca, Z., Malaya, M. B. & Gajos, K. Z. (2021). To Trust or to Think: Cognitive Forcing Functions Can Reduce Overreliance on AI. Proc. ACM Human-Computer Interaction (CSCW).
48. Bansal, G. et al. (2021). Does the Whole Exceed its Parts? The Effect of AI Explanations on Complementary Team Performance. CHI.
49. Zhang, Y., Liao, Q. V. & Bellamy, R. K. E. (2020). Effect of Confidence and Explanation on Accuracy and Trust Calibration in AI-Assisted Decision Making. FAccT.
50. Hendrycks, D. & Gimpel, K. (2017). A Baseline for Detecting Misclassified and Out-of-Distribution Examples in Neural Networks. ICLR; arXiv:1610.02136.
51. Zheng, L. et al. (2023). Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena. arXiv:2306.05685.
52. Inan, H. et al. (2023). Llama Guard: LLM-based Input-Output Safeguard for Human-AI Conversations. arXiv:2312.06674.
53. Leucker, M. & Schallhart, C. (2009). A Brief Account of Runtime Verification. Journal of Logic and Algebraic Programming, 78(5).
54. Miller, G. A. (1956). The Magical Number Seven, Plus or Minus Two: Some Limits on Our Capacity for Processing Information. Psychological Review, 63(2). — human channel capacity.
55. Wickens, C. D. (2008). Multiple Resources and Mental Workload. Human Factors, 50(3).
56. Dawid, A. P. & Skene, A. M. (1979). Maximum Likelihood Estimation of Observer Error-Rates Using the EM Algorithm. J. Royal Statistical Society C (Applied Statistics), 28(1). — reliability-weighted aggregation of raters.
57. Cohen, J. (1960). A Coefficient of Agreement for Nominal Scales. Educational and Psychological Measurement, 20(1). — inter-rater agreement (kappa).
58. Snow, R., O'Connor, B., Jurafsky, D. & Ng, A. (2008). Cheap and Fast — But is it Good? Evaluating Non-Expert Annotations for Natural Language Tasks. EMNLP.
59. Kirwan, B. (1994). A Guide to Practical Human Reliability Assessment. Taylor & Francis. — estimating human error rates.
60. LangChain (2025). LangGraph / LangChain — Human-in-the-Loop: interrupt(), checkpointers, and HumanInTheLoopMiddleware. docs.langchain.com.
61. Microsoft (2025). AutoGen AgentChat — Human-in-the-Loop: UserProxyAgent and human_input_mode. microsoft.github.io/autogen.
62. CrewAI (2025). Human Input on Execution and Processes (sequential / hierarchical). docs.crewai.com.
63. OpenAI (2025). Agents SDK — Guardrails and Human Review: function-tool needs_approval and run interruptions. openai.github.io / developers.openai.com.
64. Microsoft (2026). Semantic Kernel — Filters: IFunctionInvocationFilter. learn.microsoft.com.
65. Pydantic (2025). Pydantic AI — Human-in-the-Loop Tool Approval: requires_approval, DeferredToolRequests / DeferredToolResults. ai.pydantic.dev.
66. Model Context Protocol Specification (2025-06-18) — Client features: Elicitation and Sampling. modelcontextprotocol.io.
67. Agent2Agent (A2A) Protocol Specification — Task lifecycle and the input-required state. a2a-protocol.org.