How a language model gets the right information at the right moment — starting from a fixed retrieval pipeline, then handing the steering wheel to the model itself, and finally turning context into something you deliberately design.
By Majid Mazouchi · Three architectures · plain-language walkthrough · block diagrams · practical notes
§ 01 — Orientation
The one idea behind all three
A large language model only knows two things: the patterns baked into its weights during training, and whatever text you place in its context window right now. It has no live memory of your files, your database, or yesterday's conversation. So the whole game of building useful AI systems comes down to a single question:
“How do the right facts end up inside the context window at the right time?”
The three architectures in this monograph are three increasingly capable answers to that question. They are not rivals so much as a progression — each one keeps the previous idea and adds more control:
The progression in one breath
RAG — a fixed pipeline fetches documents before the model speaks. The model never decides; retrieval just happens.
Agentic RAG — the model is given a search tool and decides for itself whether, when, and what to retrieve — and can loop.
Agentic search / context engineering — the model gets many tools reaching many sources (files, databases, the web, memory, the shell) and you deliberately curate what enters the window.
Plain-language analogy
RAG is a librarian who always hands you the same five pages before you ask your question. Agentic RAG is you walking up to the search desk and deciding what to look up. Context engineering is you with a full research library — card catalog, microfilm, the web, your own notebooks, and a terminal — choosing exactly what goes on your desk.
§ 02 — Foundation
Anatomy of a context window
Every diagram in this piece ends at the same place — a stack of text fed to the model on each turn. Reading this stack first makes the rest obvious. The richer the architecture, the more kinds of blocks appear in it.
Figure 1. The context window is just an ordered stack of text. Basic RAG fills only the top few rows; agentic systems add tool lists, skills, memory, and tool results. Every byte here competes for limited space.
Why this stack matters
The window has a hard size limit (the token budget). Everything you add pushes something else out, so what you include is a design decision.
Models attend unevenly: facts at the very start and very end are used more reliably than facts buried in the middle (the “lost in the middle” effect). Placement is not cosmetic.
Irrelevant text is not free — it dilutes attention and can actively mislead. More context is not better context.
§ 03 — Architecture I
Retrieval-Augmented Generation (RAG)
Classic RAG bolts a fixed retrieval step onto the front of generation. Before the model ever sees the question, a pipeline turns the query into a vector, runs a semantic_search() against a vector database, grabs the top few matching documents, and pastes them into the context window. Then the model answers using that pasted-in evidence.
The defining trait is that the model does not decide to retrieve. Retrieval is a hard-wired preprocessing stage that fires exactly once, the same way, every time — whether or not the question actually needs outside knowledge.
Figure 2. The query flows through a fixed semantic_search() into the database; the top documents are pasted into the window and the model answers from them. The model has no say over retrieval — it is a one-shot, deterministic step.
How it works, step by step
1. Index (offline). Your documents are chunked and each chunk is turned into an embedding vector, stored in a vector database. 2. Retrieve. At query time the user's question is embedded and compared by similarity to find the closest chunks. 3. Augment. Those chunks are inserted into the prompt next to the question. 4. Generate. The model writes an answer grounded in the retrieved text, ideally citing it.
Practical notes
Strengths: simple, cheap, fast, and predictable. One retrieval call, one generation call. Easy to debug and to ground answers in real sources to reduce hallucination.
Chunking is half the battle. Too-small chunks lose context; too-large chunks bury the answer. Overlap windows and store metadata (title, section) you can show to the model.
Hybrid retrieval wins. Pure vector search misses exact terms (part numbers, error codes). Combine semantic search with keyword/BM25, then re-rank the merged list with a cross-encoder before trimming to top-k.
Where it breaks: multi-hop questions ("compare A's spec against B's policy"), questions needing fresh data, or queries that don't actually need retrieval at all — the fixed pipeline still fires and wastes tokens.
Always evaluate retrieval separately from generation. A wrong answer is usually a retrieval miss, not a model failure. Track recall@k and answer faithfulness independently.
§ 04 — Deep dive
Under the hood: retrieval, chunking & GraphRAG
The semantic_search() box in Figure 2 hides most of what determines whether RAG works. In practice it is a small assembly line, and each station has its own failure modes.
Figure 2b. A real retrieval pipeline is two-stage: a cheap, recall-heavy first pass (vector + keyword) gathers many candidates, then an expensive, precision-heavy re-ranker reorders them so only the truly relevant few reach the window.
Chunking: the decision that quietly sets your ceiling
You cannot retrieve what you chunked badly. The split happens at index time, long before any query, so a poor strategy caps quality no matter how good the model is.
Fixed-size
Naïve
Split every N tokens. Fast, but slices sentences and tables in half. A baseline, rarely the answer.
Recursive
Structure-aware
Split on paragraphs → sentences, respecting boundaries. The sensible default for prose.
Semantic
Meaning-aware
Start a new chunk where the topic shifts (embedding distance jumps). Better coherence, higher cost.
Contextual
Augmented
Prepend a one-line summary of where the chunk sits before embedding. Large recall gains for a small token cost.
Always carry metadata with each chunk — source, title, section, timestamp — so the model can cite it and you can filter by it. And store an overlap window so an answer straddling a boundary isn't lost.
Query transformation: meeting the index halfway
User questions rarely match how documents are written. A layer of query rewriting closes that gap, and it is exactly what an agentic system does on its own:
Common rewrites
Multi-query — generate several paraphrases, retrieve for each, union the results. Cheap recall boost.
HyDE — have the model draft a hypothetical answer, then embed that to search. Answers look more like documents than questions do.
Step-back — ask a broader question first ("what governs current-control bandwidth?") to fetch foundational context before the specific one.
Decomposition — break a multi-part question into sub-questions, retrieve and answer each, then compose. This is the seed of multi-hop reasoning.
GraphRAG: when relationships are the answer
Vector search finds chunks that are similar to the query. It is weak when the answer requires connecting facts that never appear together — "which calibrations were touched by every engineer who also signed off on the saturation model?" That is a traversal, not a similarity match.
GraphRAG indexes your corpus as a knowledge graph of entities and relationships, then retrieves by walking the graph (and often summarizing communities of nodes) instead of, or alongside, embedding search. It shines on multi-hop, "global" questions over a connected corpus — and pairs naturally with the kind of structured extraction (A2L, ARXML, Simulink, RTE-C organized by functional rings) that a knowledge-graph platform already produces.
Practical notes
Don't pick one. Production systems route: vector search for "find me text about X," graph traversal for "how does X connect to Y," keyword for exact identifiers. A small router or the agent itself decides.
Graph construction is the cost. Entity/relation extraction is an upfront LLM-heavy pipeline. Worth it when relationships are queried repeatedly, overkill for flat FAQ corpora.
Re-ranking is the highest-leverage cheap win. If you add one thing to a plain RAG system, add a cross-encoder re-ranker before generation.
§ 05 — Architecture II
Agentic RAG
Agentic RAG removes the hard wiring. Instead of a forced retrieval step, search becomes a tool the model can choose to call. The model reads the question, decides whether it even needs to look anything up, issues a TOOL CALL with a query it writes itself, reads the TOOL RESULT that comes back in the window — and can then call again with a refined query if the first results were thin.
This single change — putting the model in the loop — is what makes it "agentic." Retrieval is now a decision, not a reflex, and it can repeat.
Figure 3. The model emits a tool call, the search tool hits the database, and results land back in the window as a tool result. Because the model can read that result and call again, retrieval becomes an iterative decision. Notice the richer stack: skills, tool list, memory, and history now coexist.
Practical notes
The model writes the query. It can rephrase the user's messy question into a clean search query, split a multi-part question into several searches, or skip retrieval entirely for chit-chat.
Iteration handles multi-hop. Read the first results, notice a gap, search again. This is what fixed RAG could never do.
New costs appear: variable latency and token spend (you don't know how many loops it'll take), and the risk of loops that never converge. Cap the iterations and give a clear stop condition.
Tool descriptions are now prompt-critical. A vague tool description produces vague tool use. Spell out exactly what the search tool indexes and when to use it.
Watch for over-retrieval. An eager agent will search when it shouldn't. Encourage it to answer from existing context first and only search when it genuinely lacks information.
§ 06 — Architecture III
Agentic search & context engineering
The final step generalizes the idea in two directions at once. First, the model gets a whole toolbox of retrieval tools — file search, a skill loader, database queries, web search, a memory tool, even a raw shell for grep and scripts. Second, those tools reach many different sources: local files, structured databases, the live web, and long-term memory.
"Context engineering" is the discipline layered on top: deliberately deciding what goes into the window, in what form, and when to remove it. The window stops being a passive bucket and becomes a workspace the agent actively curates — pulling a Skill only when a task needs it, writing a Plan.md to externalize its thinking, recalling a memory, and dropping stale results to free up room.
Figure 4. Many tools, many sources. The agent chooses among file search, skills, database queries, web search, memory, and the shell — drawing from local files, databases, the web, and long-term memory — and curates which results stay in the window. Color-coded arrows trace each tool to the source it reaches.
Practitioners describe the discipline as four recurring moves on the window:
Write · Select · Compress · Isolate
Write — push state out of the window into durable storage: a Plan.md scratchpad, notes, or long-term memory, so it survives across turns without hogging tokens.
Select — pull in only what this step needs: load a specific skill, fetch the relevant memories, retrieve the three documents that matter — not the whole corpus.
Compress — summarize or trim what's already there: collapse old turns, replace a giant tool result with its key findings, prune dead branches.
Isolate — split work across sub-agents or separate contexts so a noisy subtask (say, reading 40 files) doesn't pollute the main window.
Practical notes
Progressive disclosure is the core trick. Don't dump everything up front. A skill is loaded only when its YAML frontmatter matches the task; the shell lets the agent grep a huge repo and read only the matching lines. The window stays lean.
The shell tool is a force multiplier. Instead of stuffing files into context, let the agent run grep, head, or a script and read just the output. This often beats embedding-based search for code and logs.
Memory needs a write policy and a forget policy. Deciding what not to remember is as important as what to store, or memory becomes noise.
Beware context rot. As windows fill, models degrade — they re-read stale tool output, confuse old and new state, and lose the thread. Active compression and isolation are what keep long-running agents coherent.
Observability is mandatory. With many tools and loops, you need traces of every tool call and the evolving window to debug. You can no longer eyeball a single prompt.
§ 07 — Walkthrough
Worked example: a flux-table boundary trip
Abstractions land better on a concrete case. Take one engineer's question and trace it through all three architectures:
“Why did the axial-flux unit trip on drift-mode exit near 3000 RPM, and has this pattern shown up before?”
Architecture I · RAG
One pipeline, one shot
The question is embedded and semantic_search() returns the closest spec chunks — likely the field-weakening control description and the flux-map calibration note. The model answers from those. Limit: it cannot check whether this specific trip has happened before, because the fault history lives in a different system the pipeline never touches. You get a plausible textbook explanation, not a grounded diagnosis.
Architecture II · Agentic RAG
The model decides to look twice
The model reads the question and recognizes two parts. First it searches the spec corpus for the control behavior; then, seeing the words "shown up before," it issues a second tool call querying the fault database for prior drift-mode-exit trips on this platform. It composes the two results. Gain: the multi-hop structure is handled because retrieval is now an iterative decision — exactly the leap from the fixed pipeline.
Architecture III · Agentic search
Many tools, curated context
Now the agent orchestrates: shell to grep the dyno logs for trips in the 2900–3100 RPM band; database for the saturation-region inductance table; memory to recall the prior RCA that flagged PI gains tied to nominal inductance; web/file search for the relevant standard; and it writes findings to a Plan.md scratchpad so the growing evidence doesn't crowd the window. It then synthesizes the stacked root cause — sharp differential-inductance drop at the saturation boundary, gains tuned to nominal, noisy table edges, weak feed-forward at the boundary — and proposes a dyno test to confirm. This is the only level that can fuse logs, tables, prior analysis, and specs into a single grounded answer.
What the example shows
The same question exposes each architecture's ceiling: single-source, multi-hop-single-source, then multi-source synthesis.
Capability climbs, but so does what you must manage — three tools and a scratchpad means three things that can fail and a window you must actively keep clean.
Notice the agent used the shell on logs rather than embedding them. For structured, greppable data that is faster, cheaper, and more precise than vector search.
§ 08 — Synthesis
Side-by-side comparison
RAG
Agentic RAG
Agentic search / CE
Who triggers retrieval
A fixed pipeline, every time
The model decides
The model orchestrates many tools
Sources
One vector DB
Usually one DB via a tool
Files, DBs, web, memory, shell
Iteration
None — single shot
Yes — loops & refines queries
Yes — multi-step, multi-tool
Context window
Prompt + question + docs
+ tools, skills, memory, results
Actively curated workspace
Cost & latency
Low, predictable
Variable
Highest, hardest to bound
Best for
FAQ, doc Q&A, grounding
Multi-hop, conditional lookup
Open-ended tasks, coding agents, research
Main failure mode
Retrieval miss / wrong chunk
Over-searching, runaway loops
Context rot, tool sprawl
Read the table top to bottom and the through-line is clear: control over the context window shifts from the engineer's fixed pipeline to the model itself, and the cost of that flexibility is complexity you must manage deliberately.
§ 09 — Rigor
Evaluation & the failure taxonomy
The single most important habit: evaluate retrieval and generation separately. When an answer is wrong, you need to know whether the model never received the right evidence or received it and reasoned poorly — they have completely different fixes.
Retrieval metrics
Did the right docs arrive?
Recall@k — was the answer-bearing chunk in the top-k? Context precision — how much of what you retrieved was actually relevant (noise dilutes attention). Measured against a labeled set of question→source pairs.
Generation metrics
Did it use them well?
Faithfulness — is every claim grounded in the retrieved context, or did it hallucinate? Answer relevance — does it address the actual question? Often scored by an LLM-as-judge against the context.
Frameworks like RAGAS operationalize these into automatic scores. For agentic systems, add trajectory metrics: did the agent pick the right tools, in a sensible order, without redundant calls?
The classic RAG failure points
Missing content — the answer isn't in the corpus at all. Retrieval can't fix a gap; detect it and say "I don't know."
Missed the top-k — it's in the corpus but ranked too low. Fix with hybrid search + re-ranking, not a bigger model.
Lost in consolidation — retrieved but buried in the middle of a long context. Fix with placement and trimming.
Wrong format / not extracted — the model ignored or misread the evidence. Fix with prompting and output structure.
Incomplete — answered part of a multi-part question. The signal you've outgrown plain RAG and need iteration.
§ 10 — Production
Production concerns: security, cost & MCP
Security: retrieval is an attack surface
The moment a model acts on text it didn't write — a retrieved document, a web page, a tool result — that text can carry instructions. Indirect prompt injection means a poisoned document in your corpus can hijack the model's behavior. The danger sharpens as tools grow.
The "lethal trifecta"
An agent becomes genuinely dangerous when it simultaneously has access to private data, exposure to untrusted content (web pages, user-supplied files, retrieved docs), and a way to exfiltrate (send email, make web requests, write to shared stores).
Any one alone is fine. Together, a malicious instruction hidden in a retrieved page can read your secrets and ship them out. Break the triangle — sandbox the shell, allow-list outbound domains, keep untrusted retrieval away from privileged tools, and require human approval for irreversible actions.
Treat every retrieved or tool-returned string as untrusted input, never as instructions. Clearly delimit it in the prompt.
Cost & latency: the window is a budget
Each architecture trades predictability for capability, and the bill is paid in tokens. The interactive panel below makes the trade tangible — shrink the budget and watch low-priority blocks get evicted while pinned essentials survive.
Interactive · context budget eviction
Token budget: 100%all blocks fit
Cost levers that work
Prompt caching — stable prefixes (system prompt, tool list, skills) are cached so you don't pay to re-process them every turn. Order your window so the stable parts come first.
Semantic caching — if a near-identical query was answered recently, return the cached answer and skip retrieval and generation entirely.
Compression — summarize old turns and replace bulky tool results with their key findings, the "compress" move from §06.
Cap the loop — set a max number of tool calls and a token ceiling so an agentic run cannot spiral.
MCP: a standard plug for the toolbox
Figure 4's toolbox raises an integration question: every new source needs a tool, and writing bespoke glue for each one doesn't scale. The Model Context Protocol (MCP) is an open standard — a "USB-C for tools" — that lets a model connect to any compliant server (a database, a file store, a ticketing system, the shell) through one uniform interface. It turns the toolbox from a pile of custom integrations into a directory of pluggable connectors, which is what makes the agentic-search picture practical at scale.
§ 11 — Guidance
Which one should you build?
Start as simple as the problem allows and climb only when a real limitation forces you to. Over-engineering an agent for what plain RAG would solve is the most common and most expensive mistake.
A decision rule of thumb
Use plain RAG when questions map to one knowledge base, are mostly single-hop, and you value predictable cost and latency — internal doc Q&A, support FAQs, policy lookup.
Step up to Agentic RAG when questions need multiple lookups, conditional retrieval ("only search if the answer isn't already known"), or query reformulation the model should handle.
Reach for full agentic search / context engineering when the task spans heterogeneous sources, runs over many steps, or needs tools beyond search — coding assistants, research agents, ops automation, vehicle-health-style diagnostics that pull logs, specs, and history together.
And whichever you pick: evaluate retrieval and generation separately, instrument every tool call, and treat the context window as a scarce, designed resource rather than a bucket to fill.
§ 12 — Notes & references
Further reading
These are good entry points for each concept. Treat them as starting threads rather than the last word — the field moves fast.
Lewis et al., “Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks,” NeurIPS 2020 — the original RAG paper. arxiv.org/abs/2005.11401
Liu et al., “Lost in the Middle: How Language Models Use Long Contexts,” TACL 2024 — why position in the window matters. arxiv.org/abs/2307.03172
Anthropic, “Effective context engineering for AI agents” — the write/select/compress/isolate framing and progressive disclosure. anthropic.com/engineering
LangChain, “Context Engineering for Agents” (blog) — practical patterns and the four-move vocabulary in code. blog.langchain.dev
Gao et al., “Retrieval-Augmented Generation for Large Language Models: A Survey,” 2023 — a broad map of RAG variants including agentic/iterative retrieval. arxiv.org/abs/2312.10997
Edge et al., “From Local to Global: A Graph RAG Approach to Query-Focused Summarization,” 2024 — the GraphRAG method. arxiv.org/abs/2404.16130
Es et al., “RAGAS: Automated Evaluation of Retrieval-Augmented Generation,” 2023 — faithfulness, context precision/recall as automatic metrics. arxiv.org/abs/2309.15217
Barnett et al., “Seven Failure Points When Engineering a RAG System,” 2024 — the failure taxonomy used in §09. arxiv.org/abs/2401.05856
Simon Willison, “The lethal trifecta for AI agents” — private data + untrusted content + exfiltration. simonwillison.net
Anthropic, “Model Context Protocol” — the open standard for connecting models to tools and data. modelcontextprotocol.io
Reference titles and venues are provided from background knowledge for orientation; verify the exact current URL and citation before formal use.