Why running a large language model costs more than training it ever did — and how quantization, distillation, KV-cache tricks, and friends shrink the bill without (much) hurting the answers.
It's tempting to think the expensive part of an LLM is training it. In production, the opposite is usually true: training happens once, but inference happens every single time anyone uses the model. Every chatbot reply, every RAG query, every agent step, every batch summarization job burns GPU memory and compute — again and again, all day, every day.
So the practical question for anyone deploying these systems is not "how do we train a better model?" It's: how do we serve the model we have, faster and cheaper, without wrecking its quality?
An LLM is like a delivery truck. Buying the truck (training) is expensive once. Fuel and driver hours (inference) cost you forever. Compression is about getting the same deliveries done with a smaller, cheaper truck.
When engineers measure whether they've succeeded, four numbers dominate:
A model's parameters (its "weights") are stored as numbers, and the numeric format you choose decides how many bytes each one takes. That single choice dominates memory:
| Format | Bytes per parameter | Rough idea |
|---|---|---|
FP32 | 4 bytes | Full precision. Mostly used in training, rarely needed to serve. |
FP16 / BF16 | 2 bytes | The standard "half precision" most models ship in. |
INT8 / FP8 | 1 byte | Half the memory of BF16. Usually near-lossless with modern methods. |
INT4 | 0.5 bytes | A quarter of BF16. Aggressive — needs careful calibration. |
The arithmetic is brutally simple: memory ≈ parameters × bytes per parameter. A 109-billion-parameter model in BF16 is roughly 109B × 2 bytes ≈ ~220 GB just for the weights — before you add the KV cache and runtime overhead. That's already past two 80 GB GPUs and into a third.
Imagine writing 109 billion numbers in a notebook. You can write each one in long form (BF16), shorthand (INT8), or very tight shorthand (INT4). Same numbers, smaller notebook. Quantization is choosing the shorthand carefully so nothing important gets lost.
Quantization configs are often written as W<weights>A<activations>. W4A16 means 4-bit weights with 16-bit activations (weight-only quantization). W8A8 means 8-bit weights and 8-bit activations (full low-precision compute). This little code tells you both how much memory you save and which GPU math units get used.
Pick a model, a precision, a context length, a batch size, and a GPU. Watch the weight memory, the KV cache, the GPU count, and the monthly bill change. This is the whole economic argument for compression in one widget.
This extends the example from the talk: Llama 4 Scout (~109B) needs about three 80 GB GPUs in BF16, about two at INT8 — and at INT4 it can fit on one 80 GB GPU with room left over for the KV cache. Now crank the context to 128k with a batch of 16 and watch the KV cache become the problem — that's the part most back-of-envelope estimates miss.
Set Llama 8B · INT4 · RTX 4090 — a flagship-class small model running on a gaming card at home. Then set Maverick 400B · BF16 · A100 and look at the monthly cost. That spread is the entire compression business case.
Quantization means storing (and sometimes computing with) the model's numbers in a lower-precision format — BF16 → INT8 → INT4. Less memory per number means a smaller model, less data moving between GPU memory and compute units, and usually faster inference.
The catch: lower precision introduces rounding error. Naively rounding every weight degrades quality. The reason quantization works so well today is that modern methods are error-aware:
Naive quantization is like compressing a photo by deleting every other pixel. GPTQ is like a smart compressor that looks at the picture and decides where detail matters — keeping the face sharp and letting the sky get blurry.
As a rule of thumb in 2025–26: INT8/FP8 is usually near-free in quality for most models, while INT4 is usually fine but worth checking — especially for math, code, long-context, and tool-calling behavior, which degrade before casual chat does. Section 10 puts numbers on this.
Weights are only half the memory story. To generate each new token, a transformer attends over every previous token — and to avoid recomputing them, it caches each token's Key and Value vectors for every layer. That's the KV cache, and it grows with every token in every active conversation:
KV bytes ≈ 2 (K&V) × layers × KV-heads × head-dim × bytes/value × context × batch
Concrete feel: Llama 3.1 8B at FP16 stores roughly 0.125 MB per token. One 32k-token conversation ≈ 4 GB. Sixteen concurrent 32k conversations ≈ 64 GB — more than the model's own weights. For long-context RAG and agent workloads, the cache, not the weights, is what runs you out of memory.
The weights are the librarian's brain — fixed size. The KV cache is the pile of open books on the desk: one pile per reader, growing with every page. Serve enough readers with long books and the desk overflows long before the librarian forgets anything.
If you're sizing hardware for a RAG or agent system, estimate KV cache at your p95 context length × expected concurrency, not the demo's 2k-token chat. This is the single most common reason "it fit on my machine" deployments fall over in production.
Knowledge distillation trains a small "student" model on the outputs (and sometimes the internal probability distributions) of a large "teacher." The student learns not just the right answers, but the teacher's way of being uncertain — which is far richer training signal than raw text. This is a big part of how modern 7–8B models punch so far above their weight: many are distilled from much larger siblings.
Quantization shrinks the same brain. Distillation raises a smaller brain with a brilliant private tutor. The student never matches the tutor everywhere, but on the topics it was tutored in, it gets remarkably close — at a tenth of the cost.
Distillation shines when your workload is narrow (classification, extraction, a specific domain): a distilled or fine-tuned small model often beats a quantized large one on cost and latency. For broad open-ended assistants, large-but-quantized usually wins on quality.
Mixture-of-Experts models like Llama 4 Maverick have ~400B total parameters but only ~17B active per token — a router picks a few "expert" subnetworks for each token. This makes compute cheap per token. But here's the trap: all 400B weights must still sit in GPU memory, because any expert might be chosen next. MoE compresses your compute bill, not your memory bill — which is exactly why quantizing MoE models is so attractive: it attacks the one cost MoE doesn't fix.
Worth knowing as the other big inference lever, even though it changes nothing about model size:
Makes the model itself smaller/cheaper. Reduces memory, GPU count, and cost per token. Quality can change — you must evaluate.
Keeps the model exactly as is. A small "draft" model proposes several tokens ahead; the big model verifies them in one pass. Output is provably identical to the big model alone — just 2–3× faster when the draft guesses well. Costs extra memory (you host two models).
The two stack: a quantized target model with a quantized draft model is a common production combo.
There is no single "best" quantization scheme — it depends on what the system does all day.
Chatbots, RAG assistants, agents. The user is watching the screen, so time-to-first-token and smooth streaming win. These systems often run at low batch sizes and don't saturate the GPU. Here, weight-only quantization (W4A16, W8A16) is attractive: it cuts memory and weight-loading time without adding much runtime conversion overhead.
Summarizing thousands of documents, processing transcripts overnight, bulk classification. Nobody is staring at a spinner; you care about total tokens per second with the GPUs fully loaded. Here, W8A8 (INT8 or FP8) pays off, because the GPU's low-precision matrix units (tensor cores) get used at full tilt.
A taxi and a freight train are optimized differently. The taxi (chatbot) must pick you up now; the train (batch job) must move the most cargo per hour. Quantization choices follow the same split.
Decide your workload type before choosing a scheme. A common mistake is benchmarking a quantized model at batch size 1, seeing great latency, then deploying it for heavy batch traffic where a different format would have doubled throughput.
A simplified field guide — start at the top, follow your answers down.
4× weight savings, low overhead at small batch. Verify quality on math/code/tools.
Near-lossless, comfortable headroom for KV cache.
Best throughput per dollar on modern hardware; quality near-identical.
Uses INT8 tensor cores; calibrate activations carefully.
Two shortcuts that save days: (1) for popular open models, pre-quantized checkpoints (GPTQ/AWQ/FP8) are already on Hugging Face — start there; (2) whatever you pick, the serving engine matters as much as the format — the same W4A16 checkpoint can differ 2× in throughput between engines and kernel versions.
In practice, the pipeline is short and very repeatable:
# pip install llmcompressor from llmcompressor import oneshot from llmcompressor.modifiers.quantization import GPTQModifier oneshot( model="meta-llama/Llama-3.1-8B-Instruct", dataset="open_platypus", # small calibration set recipe=GPTQModifier(targets="Linear", scheme="W4A16", ignore=["lm_head"]), output_dir="Llama-3.1-8B-Instruct-W4A16", max_seq_length=2048, num_calibration_samples=512, )
# pip install vllm vllm serve ./Llama-3.1-8B-Instruct-W4A16 \ --max-model-len 32768 \ --kv-cache-dtype fp8 # quantize the KV cache too # ...then call it like any OpenAI-compatible endpoint: curl http://localhost:8000/v1/chat/completions \ -H "Content-Type: application/json" \ -d '{"model": "Llama-3.1-8B-Instruct-W4A16", "messages": [{"role":"user","content":"Hello"}]}'
# pip install lm-eval — run the same suite on base vs compressed
lm_eval --model vllm \
--model_args pretrained=./Llama-3.1-8B-Instruct-W4A16 \
--tasks gsm8k,mmlu --batch_size auto
Compression does not remove the need for testing. A model can hold its score on MMLU and still behave differently on your domain-specific tasks: long-context RAG, code generation, structured output, tool calling. Evaluate the compressed model on your own workload before trusting it. Many teams keep a small "golden set" of real prompts and diff the outputs before/after quantization.
"Near-lossless" is a claim, so here are representative numbers. A large-scale study from the vLLM/Neural Magic team (arXiv:2411.02355, ~500k evaluations across the Llama 3.1 family) measured accuracy recovery — the compressed model's score as a percentage of the original's — across academic and real-world benchmarks:
| Scheme | Memory vs BF16 | Typical accuracy recovery | Reading |
|---|---|---|---|
FP8 (W8A8) | ~50% | ~99–100% | Effectively lossless across model sizes and tasks. |
INT8 (W8A8) | ~50% | ~98–99% | Near-lossless when activations are calibrated well. |
INT4 (W4A16) | ~25% | ~96–99% | Usually fine; gaps show up first on hard reasoning, math, and code. |
Two honest caveats: recovery varies by model size (small models are more fragile under INT4 than large ones) and by task type (multi-step reasoning and strict structured output degrade before knowledge recall does). Which is precisely why the published averages are a starting point — and your golden set is the finish line.
Cutting memory in half typically costs about as much accuracy as a rounding error. Cutting it to a quarter usually costs a point or two — noticeable only where the model was already working hardest.
Fine-tune first, quantize last. Quantizing and then fine-tuning (without QLoRA-style machinery) fights the rounding at every step; quantizing the finished model preserves what you trained.
GPTQ/AWQ minimize error on the calibration data. Calibrating a code assistant on Wikipedia text quietly optimizes for the wrong distribution. Use a few hundred samples that look like production traffic.
Measuring latency at batch size 1, then deploying under concurrent load. Weight-only formats that shine at batch 1 can fall behind W8A8 at batch 32. Benchmark at your real concurrency.
"The weights fit" is not "the deployment fits." Size for p95 context × concurrency, or the first long-context burst will OOM the server.
MMLU can hold steady while tool-calling reliability or JSON validity drops. Benchmarks are necessary, not sufficient — diff a golden set of your own prompts.
FP8 needs Hopper-class GPUs or newer; on an A100 it falls back or fails. Marlin/INT4 kernels vary by engine version. Confirm the format is accelerated, not just supported, on your cards.
LLM compression — chiefly quantization, with pruning, distillation, and KV-cache management alongside — is one of the main reasons generative AI is deployable at scale. It is not about saving disk space. It directly determines how many GPUs you buy, how fast users see tokens, how much each request costs, and which models are even feasible for your application.
The short version to remember:
wIXr22QTEHg. The talk this page is based on.