The Question Everyone Asks First
"Will this model fit on my GPU?" is the first practical question in LLM deployment, and getting it wrong means out-of-memory crashes or paying for hardware you don't need. GPU memory (VRAM) for inference comes from three buckets: model weights, the KV cache, and activations/overhead. Let's size each.
Bucket 1: Model Weights
The dominant term. Memory for weights is simply:
weight_memory = parameters × bytes_per_parameter
Bytes per parameter by precision:
| Precision | Bytes/param | 7B model | 70B model |
|---|---|---|---|
| FP32 | 4 | 28 GB | 280 GB |
| FP16 / BF16 | 2 | 14 GB | 140 GB |
| INT8 | 1 | 7 GB | 70 GB |
| INT4 | 0.5 | 3.5 GB | 35 GB |
The quick mental math: params (in billions) × bytes/param = GB of weights. A 13B model in FP16 ≈ 26 GB. This is why quantization is so important — INT4 turns a 70B model from "needs two 80 GB GPUs" into "fits on one."
Bucket 2: The KV Cache
Every token you've generated caches its key and value vectors so you don't recompute attention. This grows with sequence length × batch size and is often the surprise that causes OOM at long context or high concurrency. Per token:
kv_bytes_per_token = 2 × n_layers × n_kv_heads × head_dim × bytes_per_element
The leading 2 is for K and V. Note n_kv_heads — models using grouped-query attention (GQA) share KV heads across query heads, which is a deliberate design choice to shrink exactly this cache.
Worked example — a 7B-class model (32 layers, 32 KV heads, head_dim 128) in FP16:
per token = 2 × 32 × 32 × 128 × 2 bytes ≈ 524 KB/token
4096-token context = ~2.1 GB per sequence
batch of 16 = ~34 GB — more than the weights!
That is the key insight: at long context and high batch size, the KV cache can dwarf the weights. This is why techniques like PagedAttention exist — see continuous batching and PagedAttention and KV cache optimization.
Bucket 3: Activations and Overhead
Temporary buffers for the forward pass, plus CUDA context, framework overhead, and memory fragmentation. For inference this is modest — budget another 10–20% on top of weights + KV cache. Serving frameworks also pre-allocate a KV-cache pool, so real usage is "as configured," not "as needed."
Putting It Together (Inference)
total_VRAM ≈ weights + kv_cache(max_batch × max_seq_len) + ~15% overhead
Example: serve a 13B model in FP16, 4096 context, batch 8
- Weights: ~26 GB
- KV cache: ~1 GB/seq × 8 ≈ 8 GB
- Overhead: ~5 GB
- Total ≈ 39 GB → fits comfortably on a 48 GB card, tight on 40 GB.
The same model at INT4:
- Weights:
6.5 GB → total ≈ **20 GB**, fits on a 24 GB consumer card.
Training Needs Much More
Inference sizing is generous compared to training. For full fine-tuning with the Adam optimizer, budget roughly:
- Weights (FP16): 2 bytes/param
- Gradients: 2 bytes/param
- Optimizer state (Adam m + v, FP32): 8 bytes/param
- Plus FP32 master weights in mixed precision: 4 bytes/param
That's ~16 bytes/param — a 7B model needs well over 100 GB, which is why full fine-tuning is multi-GPU work. This is exactly why LoRA and QLoRA exist: they train a tiny set of adapter weights and can quantize the frozen base, cutting training memory by an order of magnitude. For scaling beyond one GPU, see Distributed Training Explained.
Quick Reference
| Model | FP16 serve | INT4 serve | Full fine-tune |
|---|---|---|---|
| 7B | ~16–20 GB | ~6–8 GB | 100+ GB (multi-GPU) |
| 13B | ~30–40 GB | ~10–14 GB | 200+ GB (multi-GPU) |
| 70B | 140+ GB (multi-GPU) | ~40–48 GB | 1 TB+ (many GPUs) |
(Serving figures assume moderate context and batch; raise both and the KV cache pushes these up.)
Practical Tips
- Leave headroom. Running at 99% VRAM invites OOM under load spikes. Target ~85%.
- Cap max sequence length and batch deliberately — they set your KV-cache ceiling.
- Prefer one bigger GPU over two smaller ones when a model almost fits; cross-GPU communication adds latency.
- Measure, don't just calculate. Frameworks pre-allocate pools; check actual utilization under realistic load.
Key Takeaways
- VRAM = weights + KV cache + ~15% overhead.
- Weights scale with params × bytes/precision; quantization is the biggest lever.
- The KV cache scales with context × batch and can exceed the weights at scale.
- Training needs ~16 bytes/param for full fine-tuning — use LoRA/QLoRA to avoid it.
- Size for peak context and batch, leave headroom, then verify under real load.
Next: cut the bill with the LLM Inference Cost Optimization playbook.