Docs

How much VRAM a model needs: weights, context and overhead

Work out the memory an LLM needs from its size, quantization and context length, with worked examples for 8B, 32B and 70B models and the formula behind the VRAM calculator.

Last updated Nov 18, 2026

The memory a model needs has three parts: the weights, the context (the KV cache), and some overhead for the runtime. If the total does not fit on the GPU, part of the model runs from system RAM and speed falls sharply.

1. Weights

Weights take parameters × bits per weight ÷ 8 bytes. Quantized formats store fewer bits per weight than the 16 of the original:

FormatApprox. bits per weight8B model32B model70B model
FP161616 GB64 GB140 GB
Q8_08.58.5 GB34 GB74 GB
Q4_K_Mabout 4.84.8 GB19 GB42 GB

Real files vary a little, because some layers are kept at higher precision. The file size of the GGUF you download is the best number to use.

2. Context (KV cache)

Every token in the context stores keys and values for every layer. Per token, at 16-bit precision:

2 × layers × KV heads × head size × 2 bytes

ModelLayersKV headsHead sizePer token32k context
Llama 3.1 8B328128128 KBabout 4.3 GB
Llama 3.1 / 3.3 70B808128320 KBabout 10.7 GB

Models that use grouped-query attention (most current ones) have far fewer KV heads than attention heads, which is what keeps these numbers manageable. Quantizing the KV cache to 8 bits roughly halves it.

3. Overhead

Allow about 0.5 to 1.5 GB for the runtime's buffers, more with large batch sizes. On a desktop, the display also takes some VRAM from the card driving it.

Worked example: 70B at 4-bit with 16k context

Weights about 42 GB, KV cache about 5.4 GB, overhead about 1 GB: roughly 48 GB. That fits on two 24 GB cards with almost nothing to spare, which is why two 3090s are the classic 70B build and why a little less context or a slightly smaller quantization often makes the difference. The VRAM calculator does this sum for other models and contexts.