The Token Budget

The two reasons to care

Token optimization gets framed as a cost exercise, and cost is real: in a long-running agent, the same system prompt and tool catalog are re-sent on every single model call, so a 40,000 token preamble in a 30 step loop is 1.2 million billed input tokens for one task. But cost is the smaller half of the argument.

The larger half is accuracy. Liu and colleagues showed that model performance follows a U-shaped curve against the position of the relevant passage: it is highest when the needed information sits at the very start or the very end of the input, and it degrades markedly when the model has to reach into the middle of a long context. Performance also falls as the context simply gets longer, even for models explicitly built for long inputs. Anthropic's context engineering guidance frames the same finding as a design rule: treat context as a finite resource with diminishing marginal returns, and hunt for the smallest set of high-signal tokens that still gets the outcome you want.

So the tokens you cut are not only cheaper, they are often better. That reframing changes what counts as a successful optimization. A method that halves your token count and costs you two points of task accuracy is a bad trade. A method that halves your token count and raises accuracy, because it removed distractors, is the one you want, and several of the methods below do exactly that.

What a token is, concretely

Before any of the arithmetic, it helps to see the unit. A token is not a word and not a character. It is a subword unit produced by a byte-pair encoding trained mostly on English web text, which has three consequences that surprise people: common English words are usually one token while rare ones fragment, many non-Latin scripts cost several times more tokens per unit of meaning, and code pays a penalty for indentation, long identifiers and punctuation runs.

A map of the territory

There are only four things you can do. Every method in this monograph is an instance of one of them.

Which methods apply to you

The ranked table at the end of this monograph is the general answer. This is the version filtered to your workload. Answer four questions and it reorders.

1. Prompt caching

Caching is prefix-based and it is the single highest-leverage lever in any repeated-call workload, so it goes first. The provider stores the tokenized prefix of your request up to a marked breakpoint. Any later request that begins with a byte-identical prefix is served from that store. On Claude the prefix is assembled in a fixed order, tools then system then messages, and the cache references everything up to and including the block you mark with cache_control.

The economics are asymmetric in a way that decides your architecture. A cache write costs about 25 percent more than a normal input token. A cache read costs about 90 percent less. Break-even therefore arrives on the second hit, and everything after that is close to free. The design implication is blunt: put everything stable at the front and never let a variable token drift into the prefix. A timestamp injected at the top of a system prompt invalidates the entire cache on every call, and this is the most common reason a caching rollout shows no savings.

Practical rules

1. Order the prompt from most stable to least stable. Tools, then system instructions, then few-shot examples, then long reference documents, then the conversation, then the current turn.
2. Place one breakpoint at the end of the static region. The system finds the longest matching prefix, so a single well-placed marker usually beats several scattered ones.
3. Move anything volatile, current time, request ID, per-user greeting, out of the prefix and into the suffix.
4. Match the cache lifetime to the traffic. The short default window suits interactive sessions; a longer window is only worth its higher write premium when you expect enough reads to amortize it.
5. Log cache_read_input_tokens and cache_creation_input_tokens per call. A sudden spike in creation tokens means someone changed the prompt template.

2. Semantic caching

Prompt caching reuses the prefix and still runs the model. Semantic caching skips the call entirely. The incoming query is embedded, compared against a store of previous query embeddings, and if the nearest neighbour is close enough the stored response is returned without touching the model at all. GPTCache is the reference open-source implementation: queries go to the cache first, and a hit returns immediately, saving the API call and cutting response time substantially.

This is a categorically different mechanism from prefix caching and it has a categorically different failure mode. Prefix caching is exact-match and therefore cannot be wrong. Semantic caching is approximate-match and can be, in two ways. A false hit returns an answer to a question the user did not ask, and embedding models are known to score texts with opposite meanings but similar wording as near neighbours, which is tolerable for search ranking and not tolerable when the cached text is presented as an answer. A stale hit returns an answer that was correct when it was cached and is not any more. The GPTCache authors themselves note that hit rates high enough for production are hard to reach without accepting more of these.

When it is safe

• High query repetition with a narrow domain: FAQ handling, support triage, product lookups. Repetition is the whole business case, so measure it before building anything.
• Answers that do not go stale within the TTL. Anything touching live data needs either a short TTL or a freshness check, which erodes the saving.
• Never on personalized answers unless the cache key includes the user scope. A cache that crosses users is a data leak, not an optimization.
• Prefer a two-stage design: embed and retrieve top candidates, then re-rank with a cross-encoder before accepting a hit. It costs a little and removes most of the near-miss failures.

3. Context growth and compaction

In a naive agent loop the context is the full transcript, so cumulative billed input grows quadratically with turn count even though the useful information per turn is roughly constant. This is the cost curve that surprises teams in production: the tenth turn of a session costs several times what the first one did, for no additional user value.

Anthropic's guidance describes three counters. Compaction summarizes a conversation approaching the window limit and reinitializes with the summary. Structured note-taking pushes durable state out to a file the agent re-reads, so the transcript does not have to carry it. Multi-agent architectures give each subtask its own window. The lightest touch of the three, and the one to try first, is tool result clearing: once a tool has been called deep in the history, the raw result rarely needs to be visible again, so replace it with a short stub and keep the call itself.

4. Extractive prompt compression

Natural language is redundant, and a model does not need all of it. The LLMLingua line of work exploits this directly: use a small language model to score every token by perplexity, and drop the tokens whose removal barely changes the entropy of the sequence. The original method adds a budget controller that allocates different compression ratios to different regions of the prompt, instructions versus demonstrations versus the question, plus iterative token-level compression that accounts for dependence between the tokens it keeps. The reported headline is up to 20× compression with little performance loss on reasoning, summarization and dialogue benchmarks.

Two descendants matter for production. LongLLMLingua makes the compression question-aware and reorders documents, which raised benchmark performance while using substantially fewer tokens. LLMLingua-2 reframes compression as token classification, trains a small bidirectional encoder by distilling from a larger model, and is task-agnostic: it runs several times faster than its predecessors and cuts end-to-end latency at 2× to 5× compression ratios.

5. Learned compression

The methods above delete words. A second branch compresses context into learned representations instead, trading a training step for a much higher compression ratio and a representation the model reads natively.

Gisting trains a model to compress a prompt into a small set of gist tokens using nothing more than a modified attention mask during ordinary instruction finetuning, so downstream tokens can only attend to the prompt through the gists. On LLaMA-7B and FLAN-T5-XXL this reached up to 26× prompt compression with minimal quality loss, and because gists are cacheable activations, the compression is paid once and reused. AutoCompressors extend the idea to long contexts by recursively accumulating summary vectors across chunks. Cartridges take the furthest step: instead of putting a corpus in the window, train a small KV cache offline for that corpus and load it at inference. Naive next-token training on the corpus was not competitive with in-context learning, so the authors use self-study, generating synthetic conversations about the corpus and training with a context-distillation objective. The reported result is roughly 26× higher throughput at matched quality, with the training cost amortized over every query that touches the same corpus.

6. Retrieval instead of stuffing

The oldest lever and still the most misapplied. Stuffing a whole corpus into a long window is now technically possible and usually still wrong, for the accuracy reason in Part I: the relevant passage ends up in the middle, where recall is weakest, surrounded by distractors that compete for attention. Retrieval sends a small selected subset instead.

The variant worth adopting in agent systems is just-in-time retrieval: rather than pre-loading content, the agent holds lightweight identifiers, file paths, query strings, record IDs, and pulls the underlying content into context only at the moment it is needed. The identifier costs a dozen tokens; the document costs thousands.

7. Tool schemas and progressive disclosure

This is the quiet budget killer in MCP-connected agents. Most clients load every connected tool's full JSON schema into the system prompt at session start, before the agent has read a single word of the request. Anthropic's write-up on code execution with MCP reports that tool definitions and tool results can consume more than 50,000 tokens before the agent even begins, and that restructuring the same workload, presenting servers as a filesystem of code APIs the agent explores on demand, and letting intermediate data stay in the sandbox rather than round-tripping through the model, took a benchmark task from roughly 150,000 tokens to about 2,000, a 98.7 percent reduction.

The same principle ships in a lighter form as a tool search tool: keep the full catalog available, load definitions only for the tools the current step needs. Anthropic reported roughly 85 percent token reduction from that pattern while preserving access to the whole library. Both are instances of progressive disclosure: the catalog stays large, the working set stays small.

A secondary effect is worth naming. When a model must evaluate hundreds of tool options on every request, selection itself consumes capability that would otherwise go to the task. Fewer relevant options in context tends to improve tool-choice accuracy, not just cost.

8. Serialization format

The cheapest optimization on this page, because it costs one line of code. JSON repeats every key on every record. For a 200 row result set with 8 fields, that is 1,600 redundant key strings plus punctuation. CSV states the keys once. The saving is mechanical and lossless for flat tabular data.

Related input-side habits

• Reference files by path and let the agent read them, rather than inlining contents that may not be needed.
• Strip nulls, empty arrays and default-valued fields from tool results before they enter context.
• Truncate and paginate large results at the tool boundary, returning a count and a handle rather than 4,000 rows.
• Deduplicate identical schemas by reference rather than repeating them.

9. The output side

Constrain length and shape

An explicit output schema does double duty: it removes the preamble, the restatement of the question and the closing summary, and it makes the response parseable. "Return only the JSON object" is worth more than any prompt-shortening you will do on the input side of a short-prompt workload.

Emit diffs, not rewrites

When an agent edits a 900 line file, having it reproduce the whole file costs the full file in output tokens and invites transcription errors. A targeted replacement costs the changed region. This is the single largest output-side saving in coding agents.

Cap the reasoning budget

Extended reasoning is billed as output. It earns its cost on genuinely hard problems and wastes it on classification, extraction and routing. Set the budget per task class rather than globally, and measure whether the extra tokens actually move your task metric. On many production workloads they do not.

10. Model cascades and routing

Chen, Zaharia and Zou observed that LLM API prices differ by up to two orders of magnitude, and proposed three strategies to exploit that spread: prompt adaptation, LLM approximation, and the LLM cascade. The cascade sends a query to the cheapest model first and consults a learned scoring function; if the response is judged reliable it is returned, and no further model is queried. Their reported result is that this can match the strongest single model's performance at up to 98 percent lower cost, or beat it by several points of accuracy at equal cost.

The entire design rests on one component: the deferral signal. If you cannot tell when the cheap model is wrong, the cascade degrades into paying twice for every hard query. Practical signals include token-level confidence, self-consistency across samples, agreement between two cheap models, and a small trained verifier. Later work has extended this in several directions, including preference-trained routers and conformal prediction for probabilistic cost bounds.

11. Batch prompting

Distinct from a provider's asynchronous batch API, which changes latency and price but not token count. Batch prompting changes the token count: it puts several work items into a single prompt so the shared prefix, the instructions and the few-shot demonstrations, is paid once instead of once per item. Cheng, Kasai and Yu showed that under few-shot in-context learning the inference cost decreases almost inverse-linearly with the number of samples per batch, and measured up to a 5× reduction in token and time cost with six samples per batch across ten datasets, at comparable or better accuracy.

The word "almost" is doing work in that sentence, and so is the ceiling. Accuracy degrades as batch size grows, and how fast depends on task complexity: simple classification tolerates large batches, multi-step reasoning does not, because errors on one item bleed into the reasoning for the next. Follow-up work addresses this with batch permutation and ensembling, at the cost of some of the savings.

12. Distillation and fine-tuning

FrugalGPT calls this LLM approximation, and it is the only method here that removes tokens permanently. Forty few-shot examples in a prompt become weights; the prompt drops to a line of instruction. A smaller fine-tuned model can then replace a larger prompted one for a narrow task, which multiplies the saving by the price ratio.

The decision is a break-even calculation, not a preference. Fine-tuning has a fixed cost, in data preparation, training and evaluation, and pays back a fixed saving per call. Below the break-even volume it is a waste; above it, nothing else on this page comes close.

13. Context isolation and externalized state

Subagents are the architectural answer to context pollution. A lead agent delegates a bounded subtask to a subagent that has its own window, its own tools and its own instructions; the subagent does the token-heavy exploration and returns a condensed result. The detailed search context stays inside the subagent while the lead agent works only on synthesis. Anthropic's multi-agent research system found that agents with isolated contexts outperformed a single agent on complex research tasks, largely because each subagent's window could be spent entirely on one narrow question.

The related and much cheaper pattern is externalized state. Instead of carrying the plan, the findings and the open questions in the transcript, the agent writes them to a file and re-reads what it needs. A progress file plus version-control checkpoints lets an agent reconstruct where it was after a compaction event without the transcript having carried that state the whole way. And in the programmatic tool-calling pattern, sensitive or bulky intermediate data can move from one system to another without ever entering the model's window at all, which is a privacy property as much as a token one.

14. The serving layer

Everything above assumes a hosted API, where the only levers are the tokens you send. If you serve the model yourself, a second set of levers opens up. They do not reduce the token count; they reduce what each token costs you in memory and time, which for a self-hosted deployment is the same conversation.

The dominant constraint is the KV cache, whose memory grows linearly with sequence length and batch size, and which is the reason long-context serving is expensive rather than merely slow. PagedAttention, the technique behind vLLM, manages that cache in fixed-size blocks like operating-system pages instead of pre-allocating a contiguous region per request, which removes the fragmentation that otherwise wastes most of the allocation and lets many more requests share the same memory. On top of it, automatic prefix caching hashes each block by its contents plus its prefix, so identical system prompts, tool definitions and few-shot blocks are computed once and reused across users. That is the self-hosted equivalent of provider prompt caching, and it is why normalizing your prompts to be block-aligned and byte-identical matters as much here as it does against a hosted API.

Two further levers: KV cache quantization trades a little precision for roughly half the cache memory, which converts directly into concurrency; and speculative decoding uses a small draft model to propose several tokens that the large model verifies in one pass, accepting all tokens up to the first rejection. Speculative decoding does not change the token count at all, and it does not change quality, but it can substantially cut latency on single-sequence generation.

Note what the third column says throughout. Not one of these is a token optimization. They belong in this monograph because if you self-host, they are usually where the money actually is, and because teams frequently spend a quarter shaving prompt tokens when a serving-config change would have delivered more.

15. Measure before you optimize

Almost every pipeline has one or two calls that account for most of the spend, and teams routinely optimize the wrong one. Before touching anything, instrument per-stage token accounting: for every model call, record input tokens, cache read tokens, cache creation tokens, output tokens, and the stage name. Then look at the distribution rather than the total.

16. The retry budget

Rework is the largest unmeasured line item in most agent pipelines. Malformed JSON that fails to parse, a diff whose anchor does not match, a tool call with a bad argument, a schema violation, a timeout: each one throws away a completed call and pays for another. None of it appears in a per-call token dashboard, because every individual call looks normal.

The arithmetic is unforgiving. If a fraction r of calls must be repeated, effective cost multiplies by roughly 1/(1-r) for single retries, worse when retries themselves fail or when the retry prompt carries the failed attempt and the error as added context, which is the usual pattern. A 12 percent failure rate is a 14 percent cost increase before the added context, and comfortably 20 percent after it. That erases most application-level optimizations on this page.

17. Worked case study

A vehicle diagnostic agent. Thirty turns per session, eight connected MCP servers exposing 54 tools, a 4,000 token system prompt, 6,000 tokens of few-shot examples, tool results averaging 3,000 tokens, and 500 output tokens per turn. This is deliberately an ordinary pipeline, not a pathological one.

Below, the thirteen methods are applied in the recommended order. Each bar is the marginal effect of that method given everything switched on above it, which is the number that matters and the number nobody publishes. Turn steps off and watch the later bars change, because that interference is the point.

18. Anti-patterns

Every one of these has been shipped by a competent team, and every one of them reports as a success on a per-call dashboard.

19. Order of work

Ranked by expected saving per unit of engineering effort, for a typical hosted-API agentic workload. Figure 2 reorders this for your specific answers.

Two further sets sit outside this ranking. Serving-layer levers, in Section 14, apply only if you self-host, where they usually dominate everything above. Learned compression, in Section 5, needs weight access and belongs on the roadmap only when a corpus is stable and hot enough to amortize an offline training run.

All of the above assumes one agent. If your pipeline is a fleet, Part VI covers the sources that only appear at that scale, and the ranking there differs: the largest lever stops being caching and becomes duplication.

Two closing observations. First, these compose multiplicatively and interfere: progressive tool loading removes a block that cached beautifully, compaction rewrites a prefix and invalidates the cache, batching competes with caching for the same amortization, and compression on a cacheable region is strictly worse than caching it. Sequence the work and re-measure after each step rather than shipping all seventeen at once, which is exactly what Figure 17 is for. Second, an asynchronous batch API sits outside this ranking too, because it changes price and latency rather than token count, but where a workload tolerates delayed execution it stacks with caching for further savings.

20. The multiplier, and when to pay it

Start with the number everyone quotes. Anthropic's Research feature uses an orchestrator-worker architecture in which a lead agent plans, spawns three to five subagents that explore in parallel with their own context windows, and synthesizes their findings with a separate citation pass. On their internal research evaluation this beat single-agent Claude Opus 4 by 90.2 percent. It also consumes roughly fifteen times the tokens of ordinary chat, with plain agentic use sitting around four times. Reported alongside that: token usage alone explained about 80 percent of the performance variance on BrowseComp.

Read those two facts together and the design principle falls out. The multi-agent architecture is not a clever way to get more from the same tokens. It is a way to spend more tokens on a problem than one context window can hold, and the performance follows the spending. So the first question is never how to optimize a multi-agent system. It is whether the task can absorb a fifteen-fold multiplier at all.

The tasks that can share a shape: breadth-first, with independent branches, where the total information exceeds one window, and where the answer is worth dollars rather than cents. Legal due diligence, competitive intelligence, literature review, broad evidence gathering. The tasks that cannot are the tightly interdependent ones, coding foremost among them, where every branch needs to see what the other branches decided and the coordination cost swamps the parallelism.

21. Anatomy of a multi-agent session

Take a concrete system, close to what an internal agent kit looks like in practice: an orchestrator running 10 turns, four subagents at 8 turns each, 25 tools exposed, 40 skills available, tool results averaging 3,000 tokens. That is 42 model calls. Here is where the tokens go, and which of the six sources each belongs to.

The decomposition below is the centerpiece of this part. Each fix is a toggle; the stacked bar re-splits as you apply them.

22. Skills: the hidden budget

A skill is instructions, and instructions loaded eagerly are indistinguishable from a very long system prompt. The Agent Skills specification addresses this directly with three loading tiers, and understanding the tiers is the whole optimization.

1. Discovery. At startup the agent loads only the name and description from each skill's YAML frontmatter into the system prompt. One measurement across Anthropic's official skill set put the median discovery cost near 80 tokens per skill, ranging from about 55 to 235, so all 17 together came to roughly 1,700 tokens. An agent can be aware of dozens of skills for less context than a single activated one occupies.
2. Activation. When the description matches the task, the agent reads the full SKILL.md body into context. In that same measurement, bodies ranged from about 275 to 8,000 tokens with a median near 2,000.
3. Execution. Bundled reference files and assets are read only if the task touches them, and bundled scripts run without their source entering context at all, so only their output costs anything.

The description field is doing the load-bearing work, because it is what the agent matches the request against. A vague description under-triggers, which costs you the capability; an over-broad one triggers constantly, which puts you back where you started. Guidance is to say both what the skill does and when to use it.

Practical rules for skill authoring

• Write the description for the router, not for the human. It must state what the skill does and when to use it, because matching happens against that sentence alone.
• Factor aggressively. A 6,000 token SKILL.md that could be a 1,200 token body plus four reference files is paying five times over on every activation.
• Put procedures in scripts rather than prose where you can. A script's source never enters context; only its output does.
• Audit for overlap. Two skills whose descriptions both match the same request will both activate, and you pay for both.
• Count your kit. Multiply skill count by median body size by agent count by turn count, and compare that number to what you think you are spending. The gap is usually startling.

23. The harness, and the cache fragmentation trap

Every agent needs a preamble: harness rules, output conventions, safety constraints, the shared vocabulary of the system. The instinct when building a kit of twenty-five agents is to give each one a bespoke system prompt tuned to its role. That instinct is correct about quality and expensive in a way that is not obvious.

Caching is prefix-based, so a bespoke prompt per agent means a separate cache entry per agent. With a fixed number of calls in a session, splitting them across more personas means fewer calls per cache entry, which means the 1.25× write premium is amortized over fewer 0.10× reads. At the limit, one call per persona, you pay the write premium and never collect a single read: caching becomes a pure cost.

The fix is to move specialization behind the breakpoint. One identical base preamble for every agent, marked as the cached prefix, then a short role block appended after it. You keep the specialization and the whole fleet shares one cache entry. But there is a real crossing point here rather than a universal answer, and the figure finds it.

Other harness-level costs

• Framework scaffolding. Many frameworks inject their own reasoning scaffolds, output format instructions and reflection prompts on every call. Read the actual serialized request once. Teams are routinely surprised by what their framework adds.
• Re-planning. An orchestrator that re-derives its whole plan every turn pays for the plan every turn. Write the plan once to external state and reference it.
• Termination awareness. Failure to recognize task completion is one of the most prevalent single failure modes catalogued in multi-agent traces, and every extra turn past completion is a full-price call. Explicit termination conditions in the preamble are a token optimization.

24. Communication between agents

This is the cost with no single-agent analogue. When one agent hands work to another, the context does not move, only a serialization of it does. The orchestrator writes a brief that re-states what the subagent needs to know; the subagent writes a summary that re-states what it found. You pay to encode and again to decode, and information is lost at both ends.

The loss is what makes this expensive over time, because the natural response to a subagent that misunderstood its brief is to write a longer brief. That works, and it raises the per-delegation cost, and it does not fix the underlying problem, which is that prose is a lossy channel between two systems that could have shared a pointer instead.

Three topologies

• Full mesh. Every agent talks to every other. Message volume grows with the square of the agent count and it is essentially never the right choice above three agents.
• Star, or orchestrator-worker. All traffic flows through the lead agent. Volume is linear in agents, but the orchestrator accumulates every message, so its context grows with the square of the delegation count. This is the standard pattern and its weak point is the hub.
• Blackboard. Agents read and write a shared store keyed by claim or artifact. Messages carry pointers and short abstracts; payloads live in the store. Volume is linear, the orchestrator holds only what it chose to read, and you get an audit trail for free.

Handoff design rules

1. Return a pointer and an abstract, never the payload. The subagent writes its findings to the store and returns a path plus roughly 200 tokens of what is there. The orchestrator reads the full artifact only if it must.
2. Use a schema for briefs and results, not prose. Objective, scope, expected artifact type, success criteria, exclusions. Structured handoffs are shorter and, more importantly, they fail loudly instead of silently.
3. Make briefs specific. Delegation guidance from Anthropic's own system is blunt about this: a vague mandate produces duplicated work and misinterpretation, while a brief that names the sources, the scope and the exact fields to return does not.
4. Give subagents read access to the store rather than copies of context. If two agents need the same background, they should both read it, not both receive it.
5. Never re-broadcast. If the orchestrator finds itself forwarding subagent A's findings to subagent B, that is the blackboard asking to be built.

25. Coordination failure is a token cost

The most systematic study of this is MAST, a taxonomy built from over 1,600 annotated execution traces across seven popular multi-agent frameworks. It identifies 14 failure modes in three clusters: system design issues, inter-agent misalignment, and task verification failures. Across the analysed frameworks the observed failure rates ranged from roughly 41 percent to nearly 87 percent, and the authors are explicit that better base models alone will not close the gap, because these are architectural failures rather than capability failures.

Two things follow for token budgets. First, the retry unit in a multi-agent system is enormous. When a single call fails you discard one call; when a subagent misreads its brief you discard an entire session, briefing, exploration, tool calls and all. Second, the failure modes that matter most for cost are the cheap ones to fix: step repetition, unawareness of termination conditions, and misaligned briefs are all addressable with clearer specifications and explicit stop criteria rather than with better models.

26. Verification and fan-out budgets

Two things in multi-agent systems consume tokens on purpose, and both deserve an explicit exchange rate rather than a default.

Speculative fan-out means launching several branches and keeping the best. Every discarded branch was paid for in full. It buys accuracy, and the returns diminish roughly logarithmically in branch count while the cost rises linearly, so there is always a point past which it stops paying, and that point is usually lower than teams assume.

Verification passes, judges, critics, citation agents, self-consistency sampling, are frequently the second largest line item in a multi-agent bill and are almost never measured separately from the work they verify. Isolate them in your accounting. The question to answer is not whether verification helps, it is how many accuracy points per thousand tokens it is currently buying, and whether a cheaper verifier would buy the same.

27. Measuring a multi-agent system

Single-agent instrumentation does not survive contact with a fan-out architecture, because the totals hide everything interesting. Add these dimensions.

28. Order of work, multi-agent

Two closing notes specific to this part. Item 1 is not a rhetorical flourish: the largest multi-agent optimization available to most teams is discovering that their pipeline is sequential and does not need the topology. And items 2 and 3 interact with each other exactly as Part V's waterfall predicts, because tiering the skills shrinks the very block that the shared cached preamble was making cheap. Apply them in order and re-measure between, rather than reading either one's headline reduction figure and assuming it will survive contact with the other.

Appendix: the arithmetic behind each figure

Every calculator on this page uses a simplified model. Here they are, so you can check them, disagree with them, or substitute your own. Rates throughout: base input p_in = $3/M, output p_out = $15/M, cache write 1.25 p_in, cache read 0.10 p_in.

Figure 3, prompt caching

The 0.06 factor on history is a stand-in for partial history caching with incremental breakpoints, which real implementations achieve imperfectly.

Figure 4, semantic caching

Two logistic curves standing in for the overlap between the "genuinely equivalent" and "merely similar" similarity distributions. The slopes differ on purpose, 10 against 30: lowering the threshold buys hit rate gradually and buys false hits fast. Separation is the embedding model's ability to keep the two distributions apart, and it shifts the false-hit curve left rather than flattening it.

Figure 5, context growth

Figure 6, compression

A real compressor substitutes a small model's per-token perplexity for this heuristic. The visible behavior is similar; the quality is not.

Figure 8, tool schemas

Twenty-two tokens is a name plus a truncated description; 900 is the discovery instruction block.

Figure 9, serialization

Figure 11, cascade

Small model 74 percent accurate, large model 92 percent, cost relative to routing everything to the large model. Accuracy can exceed the large model alone, which is the effect FrugalGPT reported and not an artifact.

Figure 12, batch prompting

Figure 13, fine-tuning break-even

Figure 15, KV cache

The factor of two is keys plus values. Shared blocks are counted once for the pool rather than once per request, which is what content-addressed block hashing provides.

Figure 16, retries

Figure 17, waterfall

Each method is a multiplier applied to a named component of the per-session token total, evaluated in list order so that each bar shows the marginal effect given its predecessors. Caching multiplies the prefix component by 0.10 after the first turn, which is why every downstream method that touches the prefix shrinks once caching is enabled. Subagent isolation is a multiplier greater than one on total tokens.

Figure 18, topology

W is the ratio of information the task touches to one context window. The coordination penalty on fan_cost is what makes interdependent branches expensive: it grows with branch count and vanishes at full independence.

Figure 19, multi-agent anatomy

Figure 20, skill loading

Defaults follow published measurements of Anthropic's official skill set: metadata median near 80 tokens per skill, body median near 2,000.

Figure 21, harness caching

The crossing exists because bespoke caches the role block and fragments the cache, while shared keeps one cache entry and pays list price for the role block on every call. Which wins depends on the ratio of role size to base size against persona count.

Figure 22, communication topology

Figure 23, coordination failure

A caught failure costs twice: the discarded session plus its replacement. A silent one costs once and corrupts the answer.

Figure 24, fan-out and verification

Accuracy is logarithmic in branches and cost is linear, which is why an efficient branch count always exists and is usually smaller than expected.

References

1. [1] Liu, N. F., Lin, K., Hewitt, J., Paranjape, A., Bevilacqua, M., Petroni, F., Liang, P. "Lost in the Middle: How Language Models Use Long Contexts." Transactions of the ACL, 12:157-173, 2024. arXiv:2307.03172. arxiv.org/abs/2307.03172
2. [2] Hong, K., Troynikov, A., Huber, J. "Context Rot: How Increasing Input Tokens Impacts LLM Performance." Chroma Technical Report, July 2025. research.trychroma.com/context-rot
3. [3] Anthropic. "Effective context engineering for AI agents." Anthropic Engineering. anthropic.com/engineering/effective-context-engineering-for-ai-agents
4. [4] Anthropic. "Context engineering: memory, compaction, and tool clearing." Claude Cookbook. platform.claude.com/cookbook/tool-use-context-engineering-context-engineering-tools
5. [5] Anthropic. "Prompt caching." Claude Platform documentation. platform.claude.com/docs/en/build-with-claude/prompt-caching
6. [6] Anthropic. "Pricing." Claude Platform documentation. platform.claude.com/docs/en/about-claude/pricing
7. [7] Google Cloud. "Prompt caching" (Vertex AI, Claude partner models). Cache writes 25 percent above base input, cache reads 90 percent below. cloud.google.com/vertex-ai/generative-ai/docs/partner-models/claude/prompt-caching
8. [8] Bang, F. "GPTCache: An Open-Source Semantic Cache for LLM Applications Enabling Faster Answers and Cost Savings." NLP-OSS 2023, pp. 212-218. aclanthology.org/2023.nlposs-1.24
9. [9] Biton, D. D., Friedman, R. "From Exact Hits to Close Enough: Semantic Caching for LLM Embeddings." arXiv:2603.03301, 2026. arxiv.org/html/2603.03301v1
10. [10] Jiang, H., Wu, Q., Lin, C.-Y., Yang, Y., Qiu, L. "LLMLingua: Compressing Prompts for Accelerated Inference of Large Language Models." EMNLP 2023, pp. 13358-13376. arXiv:2310.05736. arxiv.org/abs/2310.05736
11. [11] Jiang, H., et al. "LongLLMLingua: Accelerating and Enhancing LLMs in Long Context Scenarios via Prompt Compression." arXiv:2310.06839. arxiv.org/pdf/2310.06839
12. [12] Pan, Z., et al. "LLMLingua-2: Data Distillation for Efficient and Faithful Task-Agnostic Prompt Compression." Findings of the ACL 2024. arXiv:2403.12968. arxiv.org/abs/2403.12968
13. [13] Microsoft Research. "LLMLingua Series" project page. microsoft.com/en-us/research/project/llmlingua
14. [14] Mu, J., Li, X. L., Goodman, N. "Learning to Compress Prompts with Gist Tokens." NeurIPS 2023, 36:19327-19352. arXiv:2304.08467. arxiv.org/abs/2304.08467
15. [15] Eyuboglu, S., Ehrlich, R., Arora, S., Guha, N., Zinsley, D., Liu, E., Tennien, W., Rudra, A., Zou, J., Mirhoseini, A., Re, C. "Cartridges: Lightweight and general-purpose long context representations via self-study." arXiv:2506.06266, 2025. arxiv.org/abs/2506.06266
16. [16] Ge, T., et al. "In-context Autoencoder for Context Compression in a Large Language Model." arXiv:2307.06945. Contains the comparison of GIST, AutoCompressors and ICAE cited in Section 5. arxiv.org/pdf/2307.06945
17. [17] Anthropic. "Code execution with MCP: Building more efficient agents." November 2025. Reports 150,000 to 2,000 tokens on a benchmark task. Summarized in: MCP.Directory, "MCP Context Bloat Fix 2026." mcp.directory/blog/mcp-context-bloat-fix-2026
18. [18] Kruczek, M. "Progressive Disclosure MCP: Token Savings Benchmark." January 2026. matthewkruczek.ai/blog/progressive-disclosure-mcp-servers.html
19. [19] Wire. "Progressive tool loading is the new MCP context pattern." May 2026. usewire.io/blog/progressive-tool-loading-mcp-context-pattern
20. [20] Chen, L., Zaharia, M., Zou, J. "FrugalGPT: How to Use Large Language Models While Reducing Cost and Improving Performance." arXiv:2305.05176, 2023. arxiv.org/abs/2305.05176
21. [21] Cheng, Z., Kasai, J., Yu, T. "Batch Prompting: Efficient Inference with Large Language Model APIs." EMNLP 2023 Industry Track. arXiv:2301.08721. arxiv.org/abs/2301.08721
22. [22] "Efficient but Vulnerable: Benchmarking and Defending LLM Batch Prompting Attack." arXiv:2503.15551, 2025. Source for the batching security caveat. arxiv.org/html/2503.15551
23. [23] Kwon, W., Li, Z., Zhuang, S., Sheng, Y., Yu, C. H., Gonzalez, J. E., Zhang, H., Stoica, I. "Efficient Memory Management for Large Language Model Serving with PagedAttention." arXiv:2309.06180, 2023. arxiv.org/abs/2309.06180
24. [24] vLLM. "Automatic Prefix Caching" design documentation. docs.vllm.ai/en/v0.9.0/design/automatic_prefix_caching.html
25. [25] vLLM. "Inside vLLM: Anatomy of a High-Throughput LLM Inference System." September 2025. Covers PagedAttention, continuous batching, prefix caching and speculative decoding. vllm.ai/blog/2025-09-05-anatomy-of-vllm
26. [26] Chen, C., Borgeaud, S., et al. "Accelerating Large Language Model Decoding with Speculative Sampling." arXiv:2302.01318. arxiv.org/abs/2302.01318
27. [27] LangChain. "Context Engineering for Agents." Covers scratchpads, summarization and subagent context isolation, including the reported 15× token multiple for multi-agent research. langchain.com/blog/context-engineering-for-agents
28. [28] Anthropic. "How we built our multi-agent research system." Anthropic Engineering, June 2025. Orchestrator-worker architecture, the 90.2 percent improvement over single-agent Opus 4, and the approximately 15× token multiple over chat. anthropic.com/engineering/multi-agent-research-system
29. [29] ZenML LLMOps Database. "Building Production Multi-Agent Research Systems with Claude." Summary of the above, including token usage explaining roughly 80 percent of performance variance. zenml.io/llmops-database/building-production-multi-agent-research-systems-with-claude
30. [30] Anthropic. "Agent Skills" overview. Claude Platform documentation. Three-tier progressive disclosure: frontmatter metadata at startup, SKILL.md body on activation, bundled files on demand. platform.claude.com/docs/en/agents-and-tools/agent-skills/overview
31. [31] SwirlAI. "Agent Skills: Progressive Disclosure as a System Design Pattern." Token measurements across Anthropic's official skill set: median discovery cost near 80 tokens per skill, body median near 2,000. newsletter.swirlai.com/p/agent-skills-progressive-disclosure
32. [32] Cemri, M., Pan, M. Z., Yang, S., Agrawal, L. A., Chopra, B., Tiwari, R., Keutzer, K., Parameswaran, A., Klein, D., Ramchandran, K., Zaharia, M., Gonzalez, J. E., Stoica, I. "Why Do Multi-Agent LLM Systems Fail?" NeurIPS 2025 Datasets and Benchmarks. MAST taxonomy, 14 failure modes in three clusters over 1,600+ traces from 7 frameworks. arXiv:2503.13657. arxiv.org/abs/2503.13657
33. [33] Sourcegraph. "Context Engineering: A Practical Guide for AI Agents." May 2026. Just-in-time retrieval and memory compaction in production systems. sourcegraph.com/blog/context-engineering