Imitation Learning — Learning by Watching

01 / THE IDEAShow, don't reward

Imagine teaching someone to drive. You could hand them a rulebook and a points system — +1 for staying in the lane, −10 for hitting a curb — and let them crash their way to competence. Or you could sit them in the passenger seat, let them watch you drive for a few hours, and ask them to mimic what you did.

That second approach is imitation learning (also called learning from demonstration). Instead of discovering good behavior through trial and error against a reward signal — the way reinforcement learning does — the agent learns a policy directly from examples of an expert doing the task well.

The expert provides demonstrations: a stream of situations and the actions taken in them. The agent's job is to learn the mapping situation → action so that when it meets a new situation, it acts the way the expert would have.

The one-sentence version Imitation learning turns "an expert's behavior" into training data, and learning a skill into a prediction problem: given what I see, predict what the expert would do.

02 / WHY BOTHERWhen copying beats rewarding

Reinforcement learning is powerful, but it has two stubborn costs: you need a reward function that actually captures what you want, and you need enormous amounts of trial and error, much of it catastrophic. For a real robot or a real car, crashing ten thousand times to learn what "good" means is not an option.

Imitation learning sidesteps both problems when a competent demonstrator already exists:

No reward engineering. "Drive like this human did" is far easier to provide than a hand-crafted score that balances safety, comfort, speed, and lane discipline without loopholes.
Safe, sample-efficient. The agent learns from successful behavior instead of discovering it by failing repeatedly.
Captures the unspoken. Good demonstrations encode subtle preferences — smoothness, caution near pedestrians — that are painful to write down as rules.

The flip side: the learner can only be as good as its demonstrations, and — as we'll see — copying has a failure mode that is easy to miss and hard to fix.

03 / THE LOOPWhat the pipeline looks like

The simplest form of imitation learning, behavioral cloning, is just supervised learning. Collect a dataset of (situation, expert action) pairs, then fit a model that predicts the action from the situation — exactly like training an image classifier, where the "label" happens to be a steering angle.

Figure 1. Behavioral cloning treats imitation as plain supervised learning: the expert's demonstrations become a labelled dataset, a model is fit to it, and the model is then deployed as the policy.

It is appealingly simple. The problem is that the world the policy is deployed into doesn't sit still and wait for it — and that's where the trouble starts.

04 / THE CATCHSmall mistakes snowball

Here is the central insight of imitation learning, and the reason it's more than "just supervised learning." The expert only ever demonstrated good states — a car driven by a careful human stays near the centerline, so the training data is full of centered situations and almost empty of "drifting toward the curb" situations.

Now the learned policy makes a tiny error and drifts slightly off-center. It is now in a situation the expert rarely visited, so the policy is guessing. It makes a bigger error. It drifts further into even less familiar territory. Errors compound, and the car wanders off the road. Researchers call this covariate shift: the distribution of states the policy actually encounters drifts away from the distribution it was trained on.

Per-step noise0.40

Expert centerline Learned policy Drifted / off-distribution States the expert visited

Figure 2 — interactive. The shaded band is where the expert demonstrated. In behavioral cloning, a small steering error pushes the policy outside that band, where it has no good training data, so corrections weaken and the car spirals off the road. Switch to DAgger and the expert also labels those off-center situations, so the policy keeps correcting. Drag the noise slider to make mistakes larger.

Why the errors grow — the bound

Suppose the policy disagrees with the expert with probability ε at any single step, over an episode of length T. Because one mistake leads to an unfamiliar state that causes the next mistake, the expected extra cost does not stay proportional to ε:

behavioral cloning: cost grows like O(ε·T²)
DAgger: cost grows like O(ε·T)

The quadratic term is the whole story: double the episode length and behavioral cloning's worst-case cost roughly quadruples, while DAgger's only doubles. DAgger earns that better scaling by training on the states the policy itself visits (Ross, Gordon & Bagnell, 2011).

Seeing covariate shift directly

The same idea has a cleaner statistical picture. Training data is drawn from the states the expert visits. At deployment, the policy faces the states it visits — and the more it errs, the further that distribution slides away. Where the two stop overlapping, the policy is operating with no relevant training data at all.

Policy errorlow

Distribution overlap: —Out-of-distribution mass: —

Training states (expert) Deployment states (policy) No training data here

Figure 3 — interactive. The training distribution (teal) and the distribution the policy actually meets (blue). Raise the policy error and the deployment distribution drifts and spreads, the overlap shrinks, and the rust region — states the policy must handle but was never trained on — grows.

05 / THE TOOLKITThe classic toolkit

The field's foundational methods split into two big ideas: copy the actions directly, or infer what the expert was trying to achieve and then optimize for that.

Copy the actions

Behavioral Cloning (BC)

Fit a model to predict the expert's action from the current state. Fast, simple, offline. Suffers from the compounding-error problem in Figure 2 because it never sees its own mistakes.

Best when: demonstrations are plentiful, episodes are short, recovery is easy. → Pomerleau's ALVINN drove a van this way in 1989.

Copy + let the expert correct you

DAgger (Dataset Aggregation)

Run the current (imperfect) policy, collect the states it visits, and ask the expert what to do in those states. Add the answers to the dataset and retrain — deliberately gathering data on the policy's own mistakes.

Cost: the expert must stay "in the loop." Best when: an expert (human or planner) can be queried online.

Infer the goal

Inverse Reinforcement Learning (IRL)

Instead of copying actions, recover the reward function the expert seems to be optimizing — then use ordinary RL to find a policy for it. Generalizes to situations the expert never demonstrated.

Cost: solving an RL problem inside the loop is expensive, and the recovered reward is ambiguous. Best when: you need behavior that generalizes.

Infer the goal, adversarially

GAIL (Generative Adversarial Imitation Learning)

A discriminator learns to tell expert trajectories from the policy's, and the policy is trained to fool it. Skips the explicit reward step of IRL and scales to high-dimensional control.

Cost: adversarial training is finicky and needs many environment interactions. Best when: you have a simulator.

Figure 4. The two families. Action-copying methods (BC, DAgger) are simpler and offline-friendly; intent-inferring methods (IRL, GAIL) generalize better at the cost of more computation and environment access.

The catch inside IRL: which reward?

Recovering "the" reward function is fundamentally ambiguous. Many rewards explain the same demonstrations equally well — including the trivial zero reward, under which every behavior is optimal. The standard fix is maximum-entropy IRL: among all rewards consistent with the demonstrations, prefer the one that explains the behavior while assuming as little else as possible, breaking the tie in a principled, noise-tolerant way (Ziebart et al., 2008).

How GAIL closes the loop

GAIL turns imitation into a game. The policy proposes trajectories, a discriminator scores how "expert-like" they look, and that score is fed back to the policy as a reward to maximize. At equilibrium the policy's behavior is statistically indistinguishable from the expert's.

Figure 5. The GAIL game. The discriminator separates expert trajectories from the policy's; the policy is rewarded for being mistaken for the expert. No explicit reward function is ever written down.

More demonstrations: how far does each method get?

If I collect more demonstrations, how much better does the policy get? Behavioral cloning improves but plateaus below the expert — more centered demos still don't teach recovery. DAgger keeps climbing toward expert level, because each round adds data exactly where the policy is weak.

Demonstrations—

BC success: —DAgger success: —Expert ceiling: 95%

Figure 6 — interactive. Task success vs. number of expert demonstrations (log scale). BC (blue) saturates short of the dashed expert ceiling; DAgger (teal) keeps approaching it. More data alone does not cure covariate shift — the kind of data matters.

Extending the toolkit

Each classic method spawned refinements that are worth knowing by name:

AIRL (Adversarial Inverse RL) — like GAIL, but structures the discriminator so it recovers a reusable, transferable reward rather than just a fool-the-classifier signal. Use it when the learned reward must survive changes in dynamics.
IQ-Learn — unifies BC and IRL through a single learned Q-function, avoiding the unstable adversarial inner loop while staying sample-efficient.
The DAgger family — DART injects noise during demonstration so the expert naturally shows recovery; SafeDAgger queries the expert only when the policy is uncertain; HG-DAgger and EnsembleDAgger let a human (or an uncertainty estimate) decide when to take back control — important when the "expert" is a person who can't label thousands of states.

06 / AT SCALEModern imitation at scale

The classic methods mostly assume a small policy predicting one action at a time. The last few years pushed imitation in a different direction: richer action representations and much bigger models trained on much more data. These don't replace the ideas above — they change how the policy is parameterized.

Diffusion policies: stop averaging

Recall the mode-averaging failure — when the expert had two valid choices (go left or right around an obstacle), a model predicting a single action averages them into a disastrous middle. A diffusion policy represents the whole distribution over actions and samples from it, so it can commit to one mode instead of blending them. The figure below makes the difference concrete.

Two valid expert demos Learned path Collision

Figure 7 — interactive. The expert demonstrated going both over and under the obstacle. Behavioral cloning regresses the mean of those two demos and drives straight into it. A diffusion policy samples one coherent mode and goes cleanly around — hit Resample to watch it pick a different valid route each time.

Action chunking: predict a phrase, not a syllable

Instead of predicting the next action, predict a short sequence of upcoming actions and execute them as a unit. This damps the per-step jitter that feeds compounding error and captures temporal structure the way predicting whole words beats predicting letters. It is the core idea behind ACT, the method driving low-cost bimanual manipulation (the ALOHA system).

Figure 8. Action chunking predicts several future actions from one observation and executes them together, reducing the number of independent decisions per episode — and with it, the room for errors to compound.

Sequence models: imitation as next-token prediction

The Decision Transformer reframes control as sequence modeling: feed a transformer a history of states, actions, and desired returns, and let it predict the next action the way a language model predicts the next token. It blurs the line between imitation and offline RL — you can condition on "how well I want to do" at generation time. The closely related Trajectory Transformer takes the same sequence-modeling view.

Vision-Language-Action models: one policy, many tasks

The frontier is VLA foundation models — large networks that take a camera image and a natural-language instruction and output robot actions, trained largely by behavioral cloning on enormous, pooled demonstration datasets across many robots (RT-1, RT-2, the Open X-Embodiment collection, Octo, OpenVLA). They are the "scaling hypothesis" applied to imitation: enough diverse demonstrations and a big enough model yield a single policy that generalizes across tasks and embodiments.

Figure 9. A vision-language-action model maps perception plus a language instruction directly to actions. The "intelligence" is overwhelmingly cloned from large demonstration corpora, then sometimes fine-tuned with RL.

07 / BEYOND ACTIONSBeyond action-copying

So far the expert handed over labelled actions. Three important variants relax that assumption.

No action labels

Imitation from observation

Learn from demonstrations that contain only states — video of a task, with no recorded actions. The agent must infer the missing actions, e.g. by learning an inverse dynamics model (BCO) or matching state-transition distributions adversarially (GAIfO). This unlocks the vast supply of "watch a human do it" data.

Why it matters: most real-world demonstrations (online videos, legacy logs) have no action channel.

No demonstrations at all

Preference-based learning & RLHF

Instead of demonstrating, the human compares outcomes — "this one is better." A reward model is fit to those preferences, then optimized with RL. This is the backbone of RLHF for language models; DPO later showed you can skip the explicit reward model and optimize from preferences directly.

Why it matters: judging is often easier than demonstrating, and works where the expert can't act the task out.

Imitate, then surpass

Imitation + RL hybrids

Seed a reinforcement learner with demonstrations so it starts from competent behavior, then let reward-driven exploration push past the demonstrator — methods like DQfD and DDPGfD pre-fill the replay buffer with expert transitions. You get imitation's safety early and RL's ceiling later.

Why it matters: removes imitation's hard cap at expert performance.

08 / POSITIONINGImitation, RL, and offline RL

Three families learn control policies, and the boundaries blur. The useful way to tell them apart is by what each one requires and what it can ultimately achieve.

Paradigm	Needs a reward?	Needs to act in the world?	Needs an expert?	Can beat the data source?
Imitation (BC / DAgger)	no	BC no · DAgger queries expert	yes	no — capped at the expert
Reinforcement learning	yes	yes — lots of trial & error	no	yes
Offline RL	yes	no — fixed logged dataset	no	yes — can exceed the logged policy

The subtle line is between behavioral cloning and offline RL. Both learn from a fixed dataset without touching the environment. The difference is the label: cloning copies the actions and inherits the data's quality as a ceiling; offline RL uses a reward and can, in principle, stitch together a policy better than the one that collected the data.

The common production recipe Bootstrap with imitation to get a safe, sane policy cheaply, then refine with reinforcement learning to push past the demonstrator. Safety early, performance later — the strategy behind many real robotics and game-playing systems.

09 / IN THE WILDWhere it actually shows up

Domain	The expert	What gets imitated
Self-driving	Human drivers	Steering/throttle from sensors — ALVINN (1989) to NVIDIA's end-to-end network (2016).
Robot manipulation	Teleoperation / kinesthetic teaching	Joint trajectories for grasping, pouring, assembly; bimanual skills via ACT/ALOHA.
Dexterous & humanoid	Motion capture of people	Whole-body locomotion and acrobatics (DeepMimic, adversarial motion priors).
Surgical robotics	Expert surgeons	Fine motion patterns for suturing and knot-tying.
Drone racing / UAVs	An optimal controller (MPC)	Fast neural policies distilled from a slow expert (Deep Drone Acrobatics).
Game & sim agents	Human or scripted play	Pre-training on replays before RL fine-tuning (early AlphaGo, StarCraft agents).
LLMs & agents	Human-written responses / task demos	Supervised fine-tuning is behavioral cloning; computer-use agents clone UI demonstrations.
Clinical decision support	Clinicians	Treatment/triage policies imitating expert care pathways.

10 / CASE STUDYFrom a slow optimum to a real-time policy

One of the most useful patterns for a controls engineer barely looks like "imitation" at first glance: the expert isn't a human, it's an optimization that is too slow to run online. Imitation distills it into a policy that runs in microseconds.

Figure 10. Imitation as controller distillation: an optimal but slow controller generates (state, optimal-input) pairs offline; a small network clones the mapping and runs in real time on embedded hardware.

Approximate / explicit MPC. Model predictive control solves a constrained optimization at every step — often too slow for fast electromechanical loops. Cloning its input across the state space yields a neural controller that mimics MPC at a fraction of the compute, with active research on stability and constraint guarantees (Hertneck et al., 2018; Chen et al., 2018).
EV / hybrid energy management. The globally optimal power-split for a known drive cycle comes from dynamic programming, which needs the future and can't run live. Imitating the DP solution produces a causal, real-time supervisory strategy that approaches the offline optimum.
Calibration automation. A calibration engineer's tuning moves — how they adjust maps in response to test data — can be logged as demonstrations and cloned into an assistant that proposes the next adjustment.
VHM decision policies. A senior technician's diagnostic sequence (which codes they read, which subsystem they probe next) clones into a triage policy — reliable on the common faults they demonstrated often, and back in off-distribution territory on the rare intermittent fault, exactly like Figure 2 in diagnostic clothing.

The recurring caveat Distilled policies are trustworthy inside the operating region the expert (human or optimizer) actually covered, and unsupervised outside it. Coverage of the demonstration set — not its sheer size — is what decides whether the policy is safe to deploy.

11 / FAILURE MODESHow imitation goes wrong

Beyond compounding error, a handful of failure modes recur often enough to recognize on sight.

Spurious cues

Causal confusion

The policy latches onto a feature that correlates with the expert's action but doesn't cause it — e.g. a driving model that reads a dashboard brake indicator (which lights because the expert braked) and learns "indicator on → brake," so it never brakes on its own. Counter-intuitively, more information can make this worse (de Haan et al., 2019).

Tell-tale sign: great offline accuracy, poor closed-loop behavior.

Many right answers

Mode averaging

When the expert had multiple valid choices, a model predicting one action averages them into something neither expert would do (Figure 7). Mitigation: model the action distribution — mixtures, diffusion, or energy-based policies.

This is exactly the gap diffusion policies were built to close.

Garbage in

Imperfect demonstrators

Imitation faithfully reproduces whatever it is shown — biases, idiosyncrasies, bad habits. Inconsistent demonstrators (several people, different styles) leave the policy imitating a contradiction.

Mitigation: curate, weight by quality, or switch to a reward-based method that can surpass the data.

Slow drift

Deployment distribution shift

Even a good policy degrades when the world changes after training — new sensors, worn actuators, new road markings. The state distribution moves out from under a policy with no mechanism to notice.

Mitigation: monitor for out-of-distribution inputs, schedule re-collection, keep a human fallback.

12 / CHOOSINGWhich method when

A rough decision path for the classic four. It is not gospel — real projects mix methods — but it captures the questions that drive the choice.

Figure 11. Start from whether you can query the expert online. If yes, DAgger fixes most of BC's drift. If not, decide whether you only need to mimic (BC) or to generalize — and if generalizing, let access to a simulator pick GAIL over plain IRL.

One more axis sits orthogonal to this chart: how you represent the policy. Whichever box you land in, if the expert behaves multimodally, reach for a diffusion or sequence model; if per-step jitter is hurting you, predict action chunks; if you have a mountain of cross-task demonstrations, a VLA-style foundation policy may beat any single-task method.

13 / PRACTICAL NOTESIf you actually build one

Cover the recovery states. Deliberately include demonstrations of getting back on track, not just perfect runs. A policy trained only on flawless trajectories can't recover.
Reach for DAgger when an expert is queryable. If you can ask the expert (human, planner, or MPC) "what would you do here?" online, DAgger fixes most of BC's drift for little extra code.
Watch compounding error in long episodes. Measure performance over full episodes, not single-step accuracy; consider action chunking to cut the number of independent decisions.
Suspect causal confusion when offline accuracy and closed-loop behavior disagree. Test in the loop, not just on the dataset.
Model multimodality. If the expert had several valid choices, don't predict a single mean — use mixture, diffusion, or energy-based policies.
Exploit observation-only data. Video without action labels is still usable (BCO, GAIfO) and is often the only data you have at scale.
The policy inherits the demonstrator's flaws — and its ceiling. To exceed the expert, pair imitation with RL (DQfD-style) or learn from preferences.
Combine, don't choose. Imitate to bootstrap a sane policy, then reinforcement-learn to refine past the expert.

14 / KEY TERMSA short glossary

Policy

The function the agent learns: maps a situation (state) to an action.

Behavioral cloning

Supervised learning of a policy from (state, expert-action) pairs. The simplest form of imitation.

Covariate shift

When the states the policy meets at deployment differ from the states it was trained on.

Compounding error

Small per-step mistakes that push the agent into unfamiliar states, causing larger mistakes.

DAgger

Dataset Aggregation: iteratively label the states the current policy visits, fixing covariate shift.

Inverse RL (IRL)

Recovering the reward function the expert appears to optimize, rather than copying actions.

Max-entropy IRL

Resolves IRL's ambiguity by preferring the least-committed reward consistent with the demonstrations.

GAIL / AIRL

Adversarial imitation: a discriminator scores expert-likeness. AIRL additionally recovers a transferable reward.

Diffusion policy

A policy that models and samples from the full action distribution, handling multimodal experts without averaging.

Action chunking

Predicting and executing a short sequence of actions at once (e.g. ACT) to reduce compounding error.

Decision Transformer

Casting control as return-conditioned sequence prediction, bridging imitation and offline RL.

VLA model

Vision-Language-Action foundation model mapping image + instruction to actions, cloned from large demo sets.

Imitation from observation

Learning from state-only demonstrations (e.g. video), with actions inferred (BCO, GAIfO).

RLHF / DPO

Learning from human preferences rather than demonstrations; DPO optimizes preferences without an explicit reward model.

Causal confusion

Latching onto a feature that correlates with the expert's action without causing it.

Mode averaging

Collapsing several valid expert actions into a bad average when the policy outputs only one.

15 / REFERENCESWhere to read more

Foundations

Pomerleau, D. A. (1989). ALVINN: An Autonomous Land Vehicle in a Neural Network. NeurIPS.
Ng, A. Y., & Russell, S. (2000). Algorithms for Inverse Reinforcement Learning. ICML.
Abbeel, P., & Ng, A. Y. (2004). Apprenticeship Learning via Inverse Reinforcement Learning. ICML.
Ziebart, B. D., et al. (2008). Maximum Entropy Inverse Reinforcement Learning. AAAI.
Ross, S., Gordon, G., & Bagnell, D. (2011). A Reduction of Imitation Learning and Structured Prediction to No-Regret Online Learning. AISTATS. — DAgger and the T² vs. T bound.
Bojarski, M., et al. (2016). End to End Learning for Self-Driving Cars. arXiv:1604.07316.
Ho, J., & Ermon, S. (2016). Generative Adversarial Imitation Learning. NeurIPS.
de Haan, P., Jayaraman, D., & Levine, S. (2019). Causal Confusion in Imitation Learning. NeurIPS.

Extensions & modern methods

Fu, J., Luo, K., & Levine, S. (2018). Learning Robust Rewards with Adversarial Inverse Reinforcement Learning (AIRL). ICLR.
Garg, D., et al. (2021). IQ-Learn: Inverse Soft-Q Learning for Imitation. NeurIPS.
Laskey, M., et al. (2017). DART: Noise Injection for Robust Imitation Learning. CoRL.
Torabi, F., Warnell, G., & Stone, P. (2018). Behavioral Cloning from Observation (BCO) and Generative Adversarial Imitation from Observation (GAIfO).
Christiano, P., et al. (2017). Deep Reinforcement Learning from Human Preferences. NeurIPS.
Rafailov, R., et al. (2023). Direct Preference Optimization (DPO). NeurIPS.
Hester, T., et al. (2018). Deep Q-Learning from Demonstrations (DQfD). AAAI.
Chen, L., et al. (2021). Decision Transformer: Reinforcement Learning via Sequence Modeling. NeurIPS.
Chi, C., et al. (2023). Diffusion Policy: Visuomotor Policy Learning via Action Diffusion. RSS.
Zhao, T., et al. (2023). Learning Fine-Grained Bimanual Manipulation with Low-Cost Hardware (ACT / ALOHA). RSS.
Brohan, A., et al. (2022/2023). RT-1 and RT-2: Vision-Language-Action Models. Plus Open X-Embodiment (2023), Octo (2024), OpenVLA (2024).

Applications & control

Peng, X. B., et al. (2018). DeepMimic. SIGGRAPH; and (2021) AMP: Adversarial Motion Priors. SIGGRAPH.
Kaufmann, E., et al. (2020). Deep Drone Acrobatics. RSS. — Imitating an MPC expert.
Hertneck, M., et al. (2018). Learning an Approximate Model Predictive Controller with Guarantees. IEEE L-CSS.
Chen, S., et al. (2018). Approximating Explicit Model Predictive Control Using Constrained Neural Networks. ACC.

Surveys

Argall, B. D., et al. (2009). A Survey of Robot Learning from Demonstration. Robotics and Autonomous Systems, 57(5).
Hussein, A., et al. (2017). Imitation Learning: A Survey of Learning Methods. ACM Computing Surveys, 50(2).
Osa, T., et al. (2018). An Algorithmic Perspective on Imitation Learning. Foundations and Trends in Robotics.