LLM Inference Cost Optimization: A Systems Playbook

Inference Is Where the Money Goes

Training an LLM is a one-time cost; serving it is a bill that arrives every single day and scales with usage. For most companies running LLMs in production, inference dominates the total cost of ownership. The good news: inference cost is highly optimizable, and the biggest wins don't require touching the model. This is the playbook, ordered roughly by return on effort.

First, know your bottleneck. LLM inference has two phases: prefill (processing the prompt, compute-bound, parallel) and decode (generating tokens one at a time, memory-bandwidth-bound, sequential). Most optimizations target decode, because that's where the wall-clock time and GPU idle time hide.

Lever 1: Batching (Biggest Win, Least Effort)

A GPU serving one request at a time is mostly idle — decode can't saturate the compute. Batching processes many requests together so each expensive weight read serves multiple sequences at once, dramatically raising throughput (tokens/sec/GPU) and cutting cost-per-token.

Continuous (in-flight) batching is the modern standard: instead of waiting for a whole batch to finish, the server swaps completed sequences out and new ones in every step, keeping the GPU full. Combined with PagedAttention for efficient KV-cache memory, this is the single highest-leverage change for a serving stack. If you run raw model.generate in a loop, adopting a batching server (vLLM/TGI-style) can multiply throughput several-fold overnight. See continuous batching and PagedAttention.

Lever 2: Right-Size the Model

The cheapest token is the one a smaller model generates correctly. Teams routinely over-provision — using a frontier model for tasks a much smaller one handles well.

• Match model to task. Classification, extraction, and routing rarely need your largest model.
• Model cascades / routing. Try a small, cheap model first; escalate to a bigger one only when confidence is low or the task is hard. Most traffic never needs the expensive path.
• Distillation / fine-tuning. A small model fine-tuned on your task can match a much larger general model at a fraction of the cost.

Right-sizing often beats every other lever combined, because it attacks the per-token price directly.

Lever 3: Quantization

Serving in INT8 or INT4 shrinks the weights, so each token reads fewer bytes from memory — directly faster in the bandwidth-bound decode phase — and fits the model on cheaper GPUs. 4-bit AWQ/GPTQ typically costs only 0–2% quality for roughly 4x smaller weights. Full details in LLM Quantization Explained, with sizing in How Much GPU Memory Do You Need.

Lever 4: Caching

Don't pay to compute the same thing twice.

• Prompt / prefix caching. Many requests share a long, fixed prefix (a system prompt, a few-shot preamble, a document). Caching that prefix's KV state means you skip re-prefilling it every call — a large saving for RAG and agent workloads with heavy shared context.
• Semantic caching. For repetitive queries, cache full responses keyed by embedding similarity, so near-duplicate questions return instantly without hitting the model at all. Watch correctness — set a tight similarity threshold and scope it to stable content.

Lever 5: Speculative Decoding

A small "draft" model proposes several tokens; the big model verifies them in a single parallel pass, accepting the correct prefix. When the draft is often right, you generate multiple tokens per expensive forward pass — 2–3x faster decode with identical output quality (the big model still validates every token). Best when you have spare compute and latency matters. See Speculative Decoding.

Lever 6: Operational Efficiency

The unglamorous wins that finance notices:

• Autoscale to traffic. GPUs idling overnight are pure waste; scale down and use scale-to-zero where cold starts allow.
• Spot/preemptible capacity for batch and non-urgent workloads.
• Separate latency tiers. Route interactive traffic to low-latency serving and bulk jobs to a throughput-optimized, heavily-batched queue.
• Cap max tokens. Unbounded generations quietly inflate cost; set sane max_tokens.

The Trade-off Triangle

Cost, latency, and quality pull against each other. Bigger batches raise throughput (lower cost) but add queueing latency. Aggressive quantization cuts cost but risks quality. The job isn't to maximize one axis — it's to hit your latency and quality targets at the lowest cost. Define those targets first, or you'll optimize the wrong thing.

A Prioritized Checklist

1. Are you batching? Adopt continuous batching first — biggest win.
2. Is the model right-sized? Route/downsize before optimizing the big model.
3. Quantized? Move to INT8/INT4 if quality holds on your eval set.
4. Caching shared prefixes? Turn on prompt/prefix caching for RAG and agents.
5. Latency-bound with spare compute? Add speculative decoding.
6. Ops: autoscale, spot capacity, token caps, tiered routing.

Work top-down and re-measure cost-per-request after each change — the ordering reflects typical ROI, but your workload decides.

Key Takeaways

1. Inference, not training, is the recurring cost — and it's highly optimizable.
2. Continuous batching is the highest-leverage change for most serving stacks.
3. Right-sizing and routing attack the per-token price directly and often win biggest.
4. Quantization, prefix/semantic caching, and speculative decoding stack on top.
5. Optimize to your latency and quality targets, not for a single metric.

---

Go deeper: Continuous Batching & PagedAttention, KV Cache Optimization, and Quantization.