Why Quantization Matters
A 70B-parameter model in FP16 needs about 140 GB just to hold its weights — more than a single 80 GB GPU can fit. Quantization shrinks those weights from 16 bits to 8, 4, or even fewer bits per parameter, so the same model fits on smaller, cheaper hardware and streams fewer bytes from memory on every token.
Because LLM decoding is memory-bandwidth bound (see our post on KV cache optimization), reading half or a quarter as many bytes per weight directly speeds up generation. Quantization is the single highest-leverage lever for cutting inference cost after batching.
The Core Idea
A quantizer maps a range of floating-point values onto a small set of integers:
q = round(x / scale) + zero_point
x ≈ (q - zero_point) * scale
The scale and zero_point are stored per group of weights (per-tensor, per-channel, or per small block). Finer granularity — smaller groups — preserves accuracy at the cost of a little more metadata. Almost all modern 4-bit methods use group-wise quantization with group sizes of 32–128.
The Two Families
Post-Training Quantization (PTQ)
You take an already-trained model and quantize it directly, using a small calibration set (a few hundred samples) to pick good scales. No gradient updates, minutes to hours of work. GPTQ, AWQ, and GGUF all live here. This is what 95% of practitioners use.
Quantization-Aware Training (QAT)
You simulate quantization during training so the model learns weights that survive it. More accurate at very low bit-widths, but you need the training pipeline and compute. Reserved for teams shipping a model to millions of devices where every accuracy point matters.
The Methods That Actually Matter
GPTQ
GPTQ quantizes weights one column at a time, using approximate second-order (Hessian) information to compensate the remaining weights for the error introduced so far. The result: strong 4-bit accuracy with a one-time calibration pass. It's the default for many GPU-served open models and is well supported across serving stacks.
- Bits: usually 4 (3-bit possible, accuracy drops)
- Best for: GPU inference where you want maximum compression with minimal quality loss
- Watch out: calibration data should resemble your real traffic
AWQ (Activation-aware Weight Quantization)
AWQ's insight: not all weights are equally important. The ones multiplied by large activations matter most. AWQ scales those salient channels before quantizing so they keep more precision. It tends to match or beat GPTQ on instruction-tuned models and is friendlier to hardware because it avoids reordering tricks.
- Bits: 4
- Best for: chat and instruction models served on GPUs; often the best quality-per-bit on modern stacks
GGUF (llama.cpp)
GGUF is a file format plus a family of quantization schemes (Q4_K_M, Q5_K_M, Q8_0, and so on) designed for CPU and Apple Silicon inference via llama.cpp. The _K schemes use mixed precision — more bits for the parts of the network that need them. This is what powers most local, laptop, and edge LLM usage.
- Bits: 2–8, mix-and-match
- Best for: running models locally, on CPUs, or on Macs;
Q4_K_Mis the popular quality/size sweet spot
bitsandbytes (NF4 / INT8)
The go-to for loading models cheaply in PyTorch, and the quantization behind QLoRA. NF4 ("normal float 4") is a 4-bit type tuned for the roughly-normal distribution of neural network weights. It's excellent for fine-tuning and quick experiments, though dedicated serving kernels (GPTQ/AWQ) are usually faster for pure inference. See our LoRA and QLoRA guide.
INT8 vs INT4: The Trade-off
| INT8 | INT4 | |
|---|---|---|
| Memory vs FP16 | ~2x smaller | ~4x smaller |
| Quality loss | Near-zero on most models | Small but real; larger models tolerate it better |
| Typical use | Latency-sensitive, quality-critical | Cost-sensitive, fitting big models on small GPUs |
A useful rule of thumb: bigger models are more forgiving of aggressive quantization. A 70B model at 4-bit often outperforms a 13B model at FP16 while using similar memory. When you must cut precision, prefer a larger model at lower bits over a smaller model at higher bits.
What Quantization Costs You
- Accuracy: usually 0–2% on benchmarks at 4-bit; more on reasoning-heavy tasks and at 3-bit or below.
- Not always faster for prefill: prompt processing is compute-bound, so quantization helps decoding more than it helps long-prompt prefill.
- KV cache is separate: weight quantization does not shrink the KV cache. For long contexts you also need KV-cache quantization or the techniques in our continuous batching post.
A Decision Checklist
- Serving on GPUs, want best quality-per-bit? → AWQ (or GPTQ) at 4-bit.
- Running locally / on CPU / Mac? → GGUF, start with
Q4_K_M. - Fine-tuning on a budget? → bitsandbytes NF4 + QLoRA.
- Latency-critical and quality-critical? → INT8, or keep FP16 and lean on batching.
- Fitting a huge model on limited VRAM? → go 4-bit and pick the largest model that fits.
Always measure quality on your task, not just public benchmarks — a 1% MMLU drop can hide a much larger regression on your specific domain.
Key Takeaways
- Quantization is the biggest single win for inference cost after batching.
- GPTQ and AWQ dominate GPU serving; GGUF owns local/CPU; bitsandbytes owns fine-tuning.
- 4-bit is the practical default; INT8 when you can't afford any quality risk.
- Bigger-model-lower-bits usually beats smaller-model-higher-bits at the same memory budget.
- Weights and KV cache are quantized separately — plan for both at long context.
Next, pair this with How Much GPU Memory Do You Need to Serve an LLM? and our LLM Inference Cost Optimization playbook.