← Back to Autonomy
Plain-Language Monograph · Agentic AI Series · Expanded Edition

Building a Team of AI Agents

Why one large language model is rarely enough — roles, memory, protocols, orchestration patterns, failure modes, costs, evaluation, and governance, explained like an engineering team.

By Majid Mazouchi Based on: IBM Technology — "Building a Team of AI Agents: Roles, Feedback & Teamwork Explained" Audience: Engineers new to agentic AI
Section 01

The core idea in one sentence

A single chatbot answers questions. A team of specialized agents — each with one clear job, connected by feedback and supervised by an orchestrator — can complete entire workflows.

Key idea Don't think of "an AI agent" as one magic chatbot. Think of it as a coordinated team: a planner, a doer, a tool user, a learner, a critic, and a supervisor — exactly like a human engineering team.

This is the central message of the IBM Technology video. Humans solved the problem of complex work long ago: we split it into roles, give each person a clear responsibility, and add review and management on top. Agentic AI borrows the same playbook.

Section 02

What an AI agent actually is

In simple words: a chatbot responds to a prompt and stops. An agent has a goal. It makes a plan, calls tools (search, databases, code execution, APIs), remembers context, and takes multiple steps until the goal is reached — or until it decides it cannot reach it.

IBM's working definition: an AI agent is a system that can autonomously perform tasks by designing its own workflow using the tools available to it. The keyword is autonomously — it decides the next step itself instead of waiting for you to type the next prompt.

Plain chatbotAI agent
Reacts to one prompt at a timePursues a goal over many steps
Only generates textPlans, calls tools, takes actions
No memory of the task stateTracks progress, context, and intermediate results
You drive every stepIt drives; you set the goal and the limits
Section 03

One agent vs. a team — same task, two ways

A single model asked to "do everything" tends to do everything poorly. Planning, coding, retrieving facts, and reviewing quality are different cognitive jobs. When one prompt has to cover all of them, the instructions blur and the model cuts corners.

Watch the same task — "Write a verified summary of failure rates from our fleet database" — handled both ways:

Interactive · toggle between approaches
USER ──► One LLM with one giant prompt: "Query the database, analyze the failure rates, check your work, and write a polished summary." LLM ──► Queries DB (maybe with the wrong filter) ──► Analyzes (mixes up two failure categories) ──► "Checks its own work" (agrees with itself) ──► Writes a fluent, confident, partly wrong summary USER ◄── One answer. No trace of where it went wrong.
Fast Cheap (1 model call chain) Self-review is weak — the same weights that made the error judge it No audit trail
USER ──► Supervisor Supervisor ──► Planner: "3 steps: query → analyze → summarize" Supervisor ──► Tool Agent: runs the SQL, returns raw rows ◄ evidence Supervisor ──► Doer: drafts the summary from the rows Supervisor ──► Critic: "Category B rate doesn't match row 47. Revise." ◄ caught it! Supervisor ──► Doer: fixes the number Supervisor ──► Critic: "Consistent with evidence. Accept." USER ◄── Verified answer + full trace of every step
Slower ~3–5× the tokens Independent critic with fresh context catches the error Every step logged and auditable
Practical note — what the team buys you
  • Sharper prompts. A critic prompt can say "find mistakes, never fix them." A planner prompt can say "produce steps, never execute." Narrow instructions are followed far more reliably than broad ones.
  • Right model for the right job. Strong reasoning model for planning and critique, code-tuned model for the doer, small cheap model for routing.
  • Independent review. The critic sees the draft with fresh context, so it isn't anchored to the doer's mistakes — the core weakness of single-agent "self-checking."
  • Auditability. Each role produces its own artifact (plan, draft, review, verdict), so you can trace exactly where a failure happened.
Section 04

The six roles of an agent team

The video describes six recurring roles. Each maps cleanly onto a familiar human job:

🗺️

Planner

Breaks a big goal into small, ordered steps. "Build me a mobile app" becomes: requirements → UI design → code → test → review → deploy.

Human equivalent: System architect
🔨

Doer

Produces the actual output — the code, the report, the document. It executes the plan rather than inventing one.

Human equivalent: Engineer
🔌

Tool operator

Knows how to call external systems: databases, APIs, web search, code runners, file systems, enterprise software.

Human equivalent: Tools specialist
📚

Learner / researcher

Fetches missing information from documents, knowledge bases, the web, or examples — so the doer doesn't have to guess.

Human equivalent: Researcher
🔍

Critic

Checks the output against the requirements, finds mistakes, and suggests improvements. It reviews; it never rewrites.

Human equivalent: Reviewer
🧭

Supervisor

Watches the whole workflow, routes work between agents, decides when the result is good enough, and stops the loop.

Human equivalent: Project manager
Why it works Team design matters more than raw model power. A very strong LLM with poor orchestration still fails; a set of modest, well-bounded agents with good context and a real feedback loop is far more reliable.
Section 05

Four levers that make each agent good

Once the roles exist, quality comes from four design levers:

LeverWhat it meansPractical note
1. Prompting Each agent gets a clear, narrow job description. Write prompts like job descriptions: responsibility, inputs, outputs, and explicit "do not" rules ("the critic never edits the draft").
2. Model selection Different models for different jobs. Reasoning-heavy roles (planner, critic) deserve the strongest model. Routing and formatting can run on small, cheap models.
3. Model tuning Adjusting models so they behave consistently in a domain. For enterprise work, consistency beats brilliance. Fine-tuning or strict configuration reduces day-to-day variance.
4. Context The right information at the right time. Too little context → hallucination. Too much irrelevant context → confusion. Curate what each agent sees.
Practical note — context is the usual culprit
  • When an agent team underperforms, check context routing before changing models. Most failures are "the doer never saw the relevant document," not "the model is too weak."
  • Give each agent only what its role needs. The critic needs the requirements and the draft — not the entire retrieval corpus.
Section 06

Memory: what agents remember, and for how long

"Context" in Section 05 is really three different memories with three different lifetimes. Confusing them is a common design error.

Memory typeLifetimeWhat it holdsWho needs it
Short-term (scratchpad) One task The current plan, intermediate results, tool outputs, draft versions. Lives in the context window. Every agent, but only its own slice — the doer's scratchpad shouldn't leak into the critic's.
Shared (blackboard) One workflow The common task state all agents read and write: the accepted plan, the evidence list, the current verdict. The supervisor owns it; agents get read access plus write access to their own fields.
Long-term / episodic Across tasks Past solutions, user preferences, domain facts — usually a vector store or database queried on demand. The learner/researcher primarily; others through it.
Practical notes
  • Blackboard beats broadcast. Writing shared state to one structured store, instead of stuffing every message into every agent's prompt, cuts token costs sharply and prevents context overflow on long tasks.
  • Keep the critic's memory clean. If the critic sees the doer's full reasoning trace, it tends to be persuaded by it. Give it the draft and the evidence — not the doer's justifications.
  • Long-term memory needs hygiene. Stale or wrong entries get retrieved forever. Add timestamps, source pointers, and a way to expire or correct entries from day one.
Section 07

How agents talk: MCP and A2A

Two open protocols are emerging as the standard wiring for agent systems. They solve different problems and complement each other:

MCP — Model Context Protocol (agent ↔ tools & data)

MCP, introduced by Anthropic, standardizes how an agent connects to tools, data sources, and services. Instead of writing custom glue code for every database, file system, or API, you run an MCP server that exposes them in one uniform format any MCP-capable agent can use. Think of it as USB-C for tools: one connector shape, many devices.

A2A — Agent-to-Agent Protocol (agent ↔ agent)

A2A, introduced by Google and now hosted by the Linux Foundation, standardizes how agents discover and talk to each other — across vendors, frameworks, and companies. Each agent publishes an "agent card" describing what it can do; other agents can then delegate tasks to it, track progress, and receive results, without sharing internal memory or prompts.

MCPA2A
ConnectsAn agent to tools and dataAn agent to other agents
AnalogyUSB-C port for capabilitiesA common business language between teams
Typical use"Query this database," "read this file," "call this API""Delegate this sub-task to the specialist agent and await its result"
Inside the team metaphorHow the tool operator holds its toolsHow the supervisor assigns work to teammates
Practical note
  • Within one application, most frameworks (LangGraph, AutoGen, CrewAI) handle agent-to-agent messaging internally — you don't need A2A inside a single process. A2A matters when agents from different systems or vendors must cooperate.
  • Adopt MCP early even in single-agent systems: tool definitions you write once stay reusable as the team grows.
Section 08

Orchestration patterns: four ways to arrange the team

"Multi-agent" is not one architecture. Four patterns cover most real systems:

PatternShapeStrengthsWeaknessesPick when…
Sequential pipeline A → B → C → D, fixed order Simple, predictable, easy to debug and audit Rigid; can't adapt when a step fails or the task varies The workflow is genuinely fixed (e.g., extract → draft → verify → format)
Supervisor (hub-and-spoke) One orchestrator routes all work to specialists Central control, clear stop conditions, easiest governance Supervisor is a bottleneck and single point of failure; its routing quality caps the system You need auditability and bounded behavior — the default for enterprise
Hierarchical Supervisors of supervisors; teams within teams Scales to large problems; each sub-team stays simple More layers = more latency, more cost, harder tracing The task naturally decomposes into sub-projects (e.g., per-subsystem documentation)
Peer-to-peer (swarm) Agents hand off to each other directly, no boss Flexible, resilient, no central bottleneck Hardest to bound and audit; loops and drift are common Exploratory tasks where adaptability beats predictability — rarely first choice in regulated domains
Rule of thumb Start with a sequential pipeline. Upgrade to a supervisor when tasks vary. Go hierarchical only when one supervisor visibly drowns. Treat swarms as research until you have monitoring you trust.
Section 09

Feedback loops — and their dangers

A good multi-agent system does not generate one answer and stop. It drafts, reviews, revises, and only then delivers. Agents can use feedback from other agents or from humans to improve their output — this reflection step is where most of the quality gain comes from.

But the feedback loop is also the riskiest component:

Caution Unbounded agent-to-agent conversation can loop forever, overuse tools (and burn budget), or reinforce each other's mistakes — two confident agents agreeing on a wrong answer feels like consensus but isn't. Multi-agent orchestration will malfunction without monitoring.
Practical note — bounding the loop
  • Set a maximum revision count (2–3 rounds is usually enough; quality gains flatten quickly after that).
  • Set tool-call budgets per task and per agent, and log every call.
  • Give the supervisor explicit stop conditions: "accept," "reject and escalate to a human," "out of budget." Never rely on agents to decide spontaneously that they are done.
Section 10

Failure modes catalog

Every one of these will eventually appear in a real deployment. Knowing the symptom in advance is half the cure:

Failure modeSymptom you'll seeMitigation
Infinite loop Critic and doer ping-pong revisions forever; latency and cost climb with no convergence. Hard iteration cap; supervisor escalates to a human after N rounds; require the critic to output a bounded verdict (accept / revise / escalate), never open-ended chat.
Sycophantic consensus Agents quickly "agree" — the critic rubber-stamps whatever the doer wrote. Output is fluent and wrong. Give the critic ground-truth evidence, not the doer's reasoning; use a different model for the critic; reward finding issues ("list at least the 3 weakest claims") rather than approval.
Context poisoning One early wrong "fact" propagates: every downstream agent repeats and builds on it. Tag every claim with its source; let deterministic extraction outrank LLM inference; re-verify key facts at the final gate instead of trusting upstream agents.
Tool overuse An agent calls search or a database dozens of times per task; bills spike, rate limits trip. Per-agent tool budgets; cache repeated queries; make the agent state why it needs each call; alert on outlier call counts.
Role bleed The critic starts rewriting the draft; the planner starts executing; outputs become untraceable. Explicit "do not" rules in every prompt; structured output schemas that physically can't carry the wrong content (a critic schema has no "revised draft" field).
Lost context / overflow On long tasks, early decisions vanish from the window; agents contradict the original plan. Blackboard memory for durable state; summarize-and-pin the accepted plan; don't rely on raw chat history as the system of record.
Silent partial failure A tool call fails, the agent improvises a plausible value, and nobody notices. Fail loudly: tool errors must surface to the supervisor as structured events, never be paraphrased away; verifier checks that every claimed tool result has a matching log entry.
Section 11

A practical architecture — walk through it

Put together, a typical multi-agent workflow looks like this:

User Goal │ ▼ Supervisor / Orchestrator ← owns stop conditions & budgets │ ▼ Planner Agent ──► Task decomposition │ ├──────────────┬──────────────┐ ▼ ▼ ▼ Research Agent Tool Agent Doer Agent (finds info) (calls APIs) (creates output) └──────────────┴──────────────┘ │ ▼ Critic / Verifier Agent ← checks facts, not just grammar │ ┌──────┴──────┐ ▼ ▼ Revision loop Accept (bounded!) │ ▼ Final answer / action

Now step through a live example — "Summarize last quarter's top three warranty failure causes for the fleet" — and watch the message at each hop:

Interactive · step-by-step workflow walkthrough

Notice two things. First, the supervisor sits above the loop, not inside it — it referees but does not produce content. Second, the critic sits between the doer and the final answer: nothing ships unreviewed.

Section 12

Cost & latency: the price of a team

Multi-agent systems multiply token usage. Every hop re-sends context, and every revision round repeats the doer + critic pair. A task that costs one unit as a single prompt commonly costs 3–10× as an agent team — more if context is broadcast instead of routed.

DriverWhy it multiplies costCountermeasure
Context re-sendingEach agent call resends shared background in its prompt.Blackboard memory + per-role context slices; prompt caching where the platform supports it.
Revision roundsEach loop ≈ one doer call + one critic call.Cap at 2–3 rounds; require the critic to list all issues in one pass instead of one per round.
Oversized models everywhereUsing the flagship model for routing and formatting wastes most of its capability.Tier the models: flagship for planner/critic, mid-tier for the doer, small for routing.
Latency stackingSequential hops add up — five 4-second calls is a 20-second answer.Parallelize independent agents (research ∥ tool calls); stream partial results to the user.
Rules of thumb — when a single agent wins
  • The task fits comfortably in one context window and a wrong answer is cheap to detect → one well-prompted agent is usually better value.
  • Interactive, latency-sensitive use (a user waiting on chat) tolerates at most one review hop. Deep multi-agent pipelines belong in batch/background jobs.
  • If your critic almost never finds anything, you're paying for review theater — either sharpen the critic or remove it.
Section 13

Evaluating an agent team

You cannot improve what you don't measure — and agent teams fail in ways unit tests don't catch. A practical evaluation stack has four layers:

1. Golden dataset (end-to-end)

Collect 30–100 representative tasks with known-good answers. Run the full team against them on every change — prompts, models, and orchestration logic all regress silently otherwise. Track exact-match where possible and rubric scores where not.

2. Per-agent unit evaluation

Test each role in isolation with fixed inputs: does the planner produce valid, complete plans? Does the critic catch seeded errors? Deliberately inject known mistakes into drafts and measure the critic's catch rate — this is the single most informative metric in the system.

3. LLM-as-judge — including judging the critic

Use a strong model with a written rubric to grade outputs at scale, and spot-check a sample by hand to calibrate the judge. Crucially, the judge also audits the critic: sycophantic consensus (Section 10) is invisible unless someone grades the grader.

4. Trace review

Log every message, tool call, and verdict as a structured trace. Review failed traces weekly: most systematic problems (role bleed, context poisoning, silent tool failures) are obvious in traces and invisible in aggregate metrics.

Practical note — minimum viable evaluation
  • Day one: structured trace logging. You can't debug what you didn't record.
  • Week one: a 30-task golden set wired into CI for prompts the same way tests gate code.
  • Month one: seeded-error tests for the critic, with catch-rate tracked over time.
Section 14

Do you actually need multiple agents?

Before committing to the architecture, answer honestly. Check every statement that is true for your task:

Interactive · decision checklist
Check the statements that apply — a recommendation appears here.
Section 15

Case study: an engineering documentation assistant

To make this concrete, here is how the role pattern maps onto a real engineering use case — an assistant that documents Simulink / TargetLink motor-control software from model files, ARXML interface descriptions, and A2L calibration files:

AgentJob in this system
Ring BuilderDetermines the task/ring context from the model, ARXML, and A2L files (the planner role, specialized for the domain).
Requirements AgentGenerates system and unit requirements (a doer).
Autodoc AgentWrites subsystem behavior, inputs, outputs, and purpose (a doer).
Calibration AgentFinds calibrations and signal dependencies (a researcher / tool operator).
Verifier AgentChecks every statement against MDL/SLX/ARXML evidence (a critic, mechanical).
Critic AgentFlags hallucinations, missing signals, and weak evidence (a critic, judgmental).
SupervisorDecides when the documentation is acceptable or must be revised.

Worked example: one autodoc request, end to end

Request: "Document the torque limiter subsystem." Follow the trace — note how every claim carries an evidence pointer, and how the verifier and critic act differently:

1 · Supervisor
Receives the request. Opens a task on the blackboard with budget: max 2 revision rounds, 40 tool calls. Routes to Ring Builder.
2 · Ring Builder
Parses the model and locates the subsystem and its interfaces. Pins the ring context to the blackboard.
SLX: TrqLimiter/ subsystem block ARXML: IF_TrqReq, IF_TrqMax
3 · Calibration Agent
Queries the A2L for calibrations referenced inside the ring; finds the torque ceiling map and ramp-rate limit, with their dependencies on motor temperature.
A2L: TrqMaxMap_T [Nm vs °C] A2L: TrqRampLim [Nm/s]
4 · Autodoc Agent
Drafts the documentation block: Purpose, Behavior ("limits requested torque to a temperature-dependent ceiling and bounds its rate of change"), Inputs, Outputs — citing the pinned evidence for each statement.
5 · Verifier
Mechanical check of every claim against the files. Finds one mismatch: the draft lists 3 inputs, but the subsystem port count is 4 — the motor-temperature input was omitted.
SLX: Inport count = 4 ≠ draft count = 3 → verdict: revise
6 · Autodoc Agent
Revision round 1 of 2: adds the missing motor-temperature input and its role in selecting the ceiling from the map.
7 · Critic
Judgmental pass: evidence coverage is complete, but flags one weakly supported sentence ("ensures smooth driver feel") as inference, not evidence — recommends rephrasing as a design intent note or removal. Verdict: accept with minor edit.
8 · Supervisor
Applies the minor edit policy, closes the task within budget (1 of 2 revision rounds used, 17 of 40 tool calls), and releases the documentation — with the full trace attached for human review.
The most important agent In safety-relevant engineering domains, the verifier/critic is the agent that matters most. Without it, the system writes beautiful but wrong requirements. The team must be evidence-gated: deterministic facts extracted from MDL/ARXML/A2L always outrank LLM inference — exactly what step 5 demonstrates.
Section 16

Governance: the hard part

The video's idea is correct, but the real engineering challenge is not "adding more agents." It is agent governance — the rules that keep the team honest:

Five governance rules
  • Narrow responsibility. Each agent does one thing. If a prompt needs the word "also," it probably describes two agents.
  • Evidence on every output. Each claim should carry a pointer to its source — a file, a signal, a tool result. No evidence, no claim.
  • Supervisor-enforced stop conditions. Iteration limits, budgets, and escalation paths are decided up front, not negotiated by the agents.
  • Critics check facts, not grammar. A critic that only polishes wording adds cost without adding safety. It must verify factual consistency against ground truth.
  • Bounded, logged, auditable tool calls. Every external action is rate-limited, recorded, and reviewable after the fact.

For enterprise engineering work, the winning execution order is:

1. Deterministic extraction first parse files, read signals — no LLM guessing 2. LLM reasoning second interpret, summarize, connect 3. Verifier / critic third check every claim against the evidence 4. Human approval last for any high-impact output

That pipeline is far stronger than a loose "chat between agents" architecture — it keeps the creativity of LLMs while anchoring every result to verifiable ground truth.

Section 17

Self-check quiz

Five questions covering the whole monograph. Pick an answer for each, then grade yourself:

Interactive · 5-question quiz
1. What is the key difference between a chatbot and an AI agent?
An agent's defining trait is goal-directed autonomy — planning, tool use, and multi-step action. Model size and model count are orthogonal (Section 02).
2. Why is a separate critic agent more reliable than asking the doer to "check its own work"?
Self-review is weak because the same context (and often the same flawed reasoning) that produced the error also judges it. Independent context is the fix (Sections 03, 06).
3. Which orchestration pattern is the recommended default for auditable enterprise systems?
The supervisor pattern gives central control, clear stop conditions, and the easiest governance. Swarms are hardest to bound; hierarchy is for when one supervisor drowns (Section 08).
4. "Two confident agents quickly agree on a fluent but wrong answer." Which failure mode is this?
Agreement that feels like consensus but isn't — mitigated by giving the critic independent evidence, a different model, and an incentive to find issues (Section 10).
5. In a safety-relevant engineering pipeline, what should outrank LLM inference?
Evidence-gating: deterministic extraction first, LLM reasoning second, verification third, human approval last (Sections 15–16).
Section 18

References

IBM Technology (YouTube), "Building a Team of AI Agents: Roles, Feedback & Teamwork Explained." The source video for this monograph. youtube.com/@IBMTechnology
IBM Think, "What Are AI Agents?" — IBM's definition of agents as systems that autonomously perform tasks by designing workflows with available tools. ibm.com/think/topics/ai-agents
IBM Think, "What Is AI Agent Orchestration?" — coordination patterns, supervisor architectures, and known failure modes such as runaway loops. ibm.com/think/topics/ai-agent-orchestration
IBM Think, "What Is AI Agent Memory?" — short-term, long-term, and shared memory in agent systems. ibm.com/think/topics/ai-agent-memory
Anthropic, "Introducing the Model Context Protocol (MCP)" and the MCP specification — the open standard for connecting agents to tools and data. modelcontextprotocol.io
A2A Project (Linux Foundation), Agent-to-Agent Protocol — the open standard for inter-agent discovery and task delegation, originally introduced by Google. a2a-protocol.org
Anthropic Engineering, "Building Effective Agents" — workflow patterns, when to add (and not add) agentic complexity. anthropic.com/research/building-effective-agents
Framework documentation implementing these patterns in practice: LangGraph (graph-based supervision), AutoGen (conversational multi-agent), CrewAI (role-based crews). langgraph docs · autogen docs · crewai docs
A note on sources This monograph is a plain-language study companion based on the video's topic and public summaries — not a verbatim transcript. For exact quotes and demos, watch the original IBM Technology video. Protocol and framework details reflect public documentation; verify version-specific behavior against the linked sources.