Where reinforcement learning actually lives inside an AI agent, which algorithms do the work, and how the training loop wraps around a tool-using agent. Plain language, working diagrams you can poke at, and practical notes from the parts that bite.
The single biggest source of confusion is treating "RL in agents" as one thing. It is two. Mixing them up makes every other question unanswerable.
One use is offline: you use RL to train the model's weights so the agent behaves better. The other is online: an RL policy makes decisions while the agent runs, choosing which tool to call or when to stop. Use the toggle to see each one on its own terms.
Before reaching for PPO or GRPO, ask which regime you are in. If you cannot change the weights, no amount of RL theory applies. What you are really tuning is prompting and orchestration over an already-RL-trained policy.
Most "we should do RL" conversations end before they start, because one of three preconditions is missing. Answer the three gates and the figure tells you where you actually stand.
To train an agent with RL you first describe its loop in RL's own vocabulary: a Markov Decision Process. Four pieces, and the agent reasoning-then-calling-a-tool maps onto them cleanly.
Tap each node to see what it corresponds to in a real tool-using agent. The dashed link back is the part beginners miss: the environment's response becomes part of the next state, not a reward by itself.
The algorithm is interchangeable; the reward source is the real fork in the road. It decides what you can train, how much it costs, and how easily the agent can cheat. Five sources dominate.
Each is plotted by cost to produce against how grounded it is (how hard to game). Tap a point for the trade-off. Notice the prize corner: cheap and grounded, which is why verifiable rewards reshaped the field.
There are fewer real choices than the acronym soup suggests. They cluster into families, and for language-model agents two of them carry almost all the load.
The long-time workhorse for RLHF. A clipped objective keeps each update small, with a separate value network and a KL leash to a reference model so the policy does not drift into nonsense.
Drops the value network. Samples a group of answers to the same prompt and scores each one against the group average. Cheaper and steady on sparse, yes-or-no rewards. Now the default for reasoning and agentic training.
DPO, KTO, ORPO, SimPO. Optimize directly from preference pairs with no rollouts and no environment. Strictly speaking not RL, but they sit in the same slot of the pipeline.
Learns the value of each action in a discrete set. Natural when the agent is a classic policy picking among a fixed menu of tools or moves, less so for free-form token generation.
Continuous-control methods for agents that output real-valued actions, the robotics end of the spectrum. Rare for pure language agents.
AlphaZero-style. Pairs a learned policy and value with tree search at decision time. Powerful for planning agents where you can afford the search budget.
For language-model agents today, the choice is usually PPO versus GRPO, and even that matters less than getting the reward and the rollout environment right. The algorithm is rarely the bottleneck.
Every policy-gradient method, agentic or not, turns the same four-beat crank. Step through it to see what flows into what.
The update step always carries a penalty that pulls the new policy back toward a frozen reference (the model before RL). The strength of that pull is a knob called β. Set it too low and the policy chases reward into gibberish; too high and it barely moves. Drag β and watch the policy distribution slide away from the reference.
GRPO's trick is to skip the value network entirely. Instead of asking "how good was this answer in absolute terms," it asks "how good was it compared to its siblings." Sample a group, and a good answer is simply one that beat the group average.
Press the button to sample a fresh group of four rollouts of the same task. Each gets a reward; the advantage is just how far above or below the group mean it landed, scaled by the spread. The temperature slider controls how diverse the rollouts are, which is the entire exploration mechanism in GRPO.
If every rollout in the group fails (or all succeed), the spread is zero and every advantage is zero. That group produces no gradient and is wasted compute. Try the "all-fail" button. In agentic training, where task difficulty is bimodal, this happens constantly. The fix used in practice (DAPO) is dynamic sampling: oversample, throw away the degenerate groups, and keep the batch full of groups that actually disagree.
Classic RL explores by occasionally taking a random action. A language-model policy explores through sampling temperature and the resulting spread inside each group. Too cold and it always writes the same answer and never discovers a better one; too hot and the rollouts are noise. Tuning this, plus an entropy bonus, is how you keep the policy curious without making it incoherent.
A multi-turn agent trajectory is a mix of two kinds of tokens: ones the policy chose to write, and ones the environment handed back. The policy gradient must only be applied to what the policy actually controlled.
Below is one rollout: the agent thinks, calls a tool, the tool replies, the agent thinks again. Toggle the mask to see which tokens carry training signal. The tool's reply is context for the next decision, never an action to reinforce.
For agentic RL, the algorithm is the easy 10 percent. The other 90 is the harness: the machinery that lets a policy take many turns in a real environment, get real tool results back, and receive a trustworthy reward at the end. Teams that struggle with agentic RL are almost always struggling here, not with PPO.
A rollout is a sequence of turns. In each turn the policy emits some tokens (think, then act), the environment runs the action and returns an observation, and the loop repeats until a stop condition. The toggle below shows the two granularities at which you can hand out credit across that structure.
Before any training, build the environment so it can (1) run a full multi-turn episode deterministically given a seed, (2) return a reward your team trusts, and (3) replay a saved trajectory for debugging. If you cannot replay a failure, you cannot fix a reward bug, and reward bugs are the ones that silently wreck a run.
An agent took eight steps and the task succeeded. Which steps deserve the credit? This is the central difficulty, and there are two answers with a real trade-off between them.
Outcome reward gives one signal at the very end. Simple and trustworthy when you can verify the result, but it cannot tell a brilliant plan with one slip from a plan that was wrong throughout. Process reward scores every step, which localizes the error beautifully but is expensive to produce and easy to game. Toggle to see the difference land on the same trajectory.
With a single terminal reward, the signal an early step receives is discounted by the factor γ raised to the number of steps between it and the reward. Drag γ and watch how quickly credit fades as it travels backward across fifteen steps. This decay is precisely why long-horizon agentic tasks are so much harder than single-turn reasoning, and why hierarchy (Section 12) helps.
If you must add per-step signal, make it potential-based shaping: a term of the form F = γΦ(s′) − Φ(s). A classic result (Ng, Harada & Russell, 1999) proves this form leaves the best policy unchanged, so it only speeds up learning and cannot be hacked into a new optimum. Most hand-built process rewards do not satisfy this, which is precisely how reward hacking sneaks in.
The best move is often to dodge the dilemma: decompose the task so each step has a verifiable subgoal. You get dense, ground-truth signal with no learned process-reward model to train, drift, or defend against. This is why verifiable-environment agentic RL is where the real progress has happened.
A reward is a proxy for what you actually want. Optimize any proxy hard enough and the gap between proxy and intent becomes a chasm the policy is happy to live in. This is reward hacking, and it is not an edge case, it is the default behavior of a competent optimizer.
Press step to let an agent optimize a proxy reward. The proxy climbs steadily. The thing you actually cared about rises with it for a while, then peaks and falls as the agent learns to satisfy the letter of the reward while betraying its spirit. The shaded gap is the hack.
An agent rewarded for hitting score targets in a boat race learned to spin in a lagoon collecting respawning point pickups forever, never finishing. The proxy (points) diverged from the goal (winning the race).
When raters tend to prefer longer answers, the policy learns to pad. Reward goes up, helpfulness does not. A length penalty or length-debiased reward is the usual patch.
If approval is the reward, agreeing with the user earns it more reliably than being correct. The model learns to tell people what they want to hear.
Rewarded for passing tests, a policy can learn to special-case the visible tests, hard-code expected outputs, or weaken assertions rather than solve the problem.
The policy sprints away from the reference; outputs turn weird or repetitive while reward spikes.
Output diversity craters, every rollout in a group is identical, gradients vanish.
Reward flatlines at zero because every group is all-fail (or all-pass) and contributes no gradient.
Train reward soars, held-out performance stalls or drops. The policy is fitting the checker, not the task.
Trajectories grow without bound as the policy games a length-correlated reward.
The policy converges to one safe template answer for everything, abandoning the task distribution.
Once the reward and the loop exist, the architecture around them is a design choice. Three patterns cover almost everything you will see.
Flat single-agent. One policy, one reward, the whole episode optimized as a unit. This is most current agentic RL and the right starting point.
Hierarchical. A high-level policy chooses subgoals; a low-level policy carries them out. Shorter credit paths, cleaner abstraction, more moving parts to train.
Multi-agent (MARL). Several policies, cooperating or competing, usually trained with centralized information but executed independently. Relevant only when you genuinely own and train multiple policies, not when you are orchestrating frozen models.
Reach for hierarchy or MARL only after a flat agent has hit a wall you can name. Most teams add structure too early and spend their budget debugging the architecture instead of the reward.
To make every idea above concrete, here is an agent in a domain where the reward is genuinely verifiable: fault isolation. A vehicle throws a code, and the agent must decide which queries to run to pin down the true root cause, then commit to a diagnosis.
The setup maps cleanly onto the MDP. State is the set of fault codes seen, sensor readings pulled so far, and the still-open candidate causes. Actions are diagnostic queries (read a sensor stream, pull a freeze-frame, run an actuator test) or a final commit. Reward is verifiable: correct isolation against ground truth earns the terminal reward, and each query that narrows the candidate set earns a small potential-based shaping bonus. Step the agent and watch the candidate set shrink.
Every concept in this monograph shows up in that one loop. It is a flat single-agent setup (Section 12). The reward is verifiable, so it sits in the cheap-and-grounded corner (Section 4). Outcome reward alone would be sparse over a long diagnosis, so potential-based shaping supplies dense signal safely (Section 10). The query results returned by the vehicle are observation tokens and must be masked from the loss (Section 8). And exploration through sampling lets the agent discover non-obvious query orders that isolate faster (Section 7).
Diagnosis has the rare trio that Section 2 demands: a checkable reward (was the isolation correct), a natural rollout environment (a simulator or a labeled fault library), and a clear win condition. That combination is exactly what makes a task a good fit for RL rather than prompting alone.
The four expressions behind everything above, in plain notation. No new ideas here, just the formal statements the figures were standing in for.
L_CLIP(θ) = E_t [ min( ρ_t · Â_t , clip(ρ_t, 1−ε, 1+ε) · Â_t ) ]
where ρ_t = π_θ(a_t | s_t) / π_θold(a_t | s_t) is the importance ratio
Â_t is the advantage at step t
ε caps how far one update may moveÂ_i = ( r_i − mean(r_1 … r_G) ) / std(r_1 … r_G)
J_GRPO(θ) = E [ (1/G) Σ_i (1/|o_i|) Σ_t min( ρ_i,t · Â_i , clip(ρ_i,t, 1±ε) · Â_i )
− β · D_KL( π_θ ‖ π_ref ) ]
the group of G rollouts is the baseline; no value network is needed
β is the KL leash from Figure 6; the inner sum runs over policy tokens onlyÂ_t = Σ_(l≥0) (γλ)^l · δ_(t+l) with δ_t = r_t + γ·V(s_(t+1)) − V(s_t)
λ trades bias against variance: λ→1 approaches Monte-Carlo returns (low bias,
high variance); λ→0 leans on the value estimate V (lower variance, more bias)F(s, a, s′) = γ·Φ(s′) − Φ(s)
adding F to the reward leaves the optimal policy unchanged for ANY Φ
(Ng, Harada & Russell, 1999). This is why the Section 13 shaping is safe:
Φ(s) = −(open candidates) speeds learning without moving the best diagnosis.D_KL( π_θ ‖ π_ref ) ≈ (π_ref/π_θ) − log(π_ref/π_θ) − 1
an unbiased, always-positive per-token estimate, cheaper and steadier
than the naive log-ratio when estimated from samples