How much VRAM a model needs: weights, context and overhead
Work out the memory an LLM needs from its size, quantization and context length, with worked examples for 8B, 32B and 70B models and the formula behind the VRAM calculator.
The memory a model needs has three parts: the weights, the context (the KV cache), and some overhead for the runtime. If the total does not fit on the GPU, part of the model runs from system RAM and speed falls sharply.
1. Weights
Weights take parameters × bits per weight ÷ 8 bytes. Quantized formats store fewer bits per weight than the 16 of the original:
| Format | Approx. bits per weight | 8B model | 32B model | 70B model |
|---|---|---|---|---|
| FP16 | 16 | 16 GB | 64 GB | 140 GB |
| Q8_0 | 8.5 | 8.5 GB | 34 GB | 74 GB |
| Q4_K_M | about 4.8 | 4.8 GB | 19 GB | 42 GB |
Real files vary a little, because some layers are kept at higher precision. The file size of the GGUF you download is the best number to use.
2. Context (KV cache)
Every token in the context stores keys and values for every layer. Per token, at 16-bit precision:
2 × layers × KV heads × head size × 2 bytes
| Model | Layers | KV heads | Head size | Per token | 32k context |
|---|---|---|---|---|---|
| Llama 3.1 8B | 32 | 8 | 128 | 128 KB | about 4.3 GB |
| Llama 3.1 / 3.3 70B | 80 | 8 | 128 | 320 KB | about 10.7 GB |
Models that use grouped-query attention (most current ones) have far fewer KV heads than attention heads, which is what keeps these numbers manageable. Quantizing the KV cache to 8 bits roughly halves it.
3. Overhead
Allow about 0.5 to 1.5 GB for the runtime's buffers, more with large batch sizes. On a desktop, the display also takes some VRAM from the card driving it.
Worked example: 70B at 4-bit with 16k context
Weights about 42 GB, KV cache about 5.4 GB, overhead about 1 GB: roughly 48 GB. That fits on two 24 GB cards with almost nothing to spare, which is why two 3090s are the classic 70B build and why a little less context or a slightly smaller quantization often makes the difference. The VRAM calculator does this sum for other models and contexts.