Docs
Sized builds from 24 GB to 96 GB, how to work out the memory a model needs, and exactly how we benchmark.
- How we benchmark GPUs for local LLMsThe exact method behind every speed on this site: llama.cpp's llama-bench, pinned versions, fixed model files and quantizations, prompt and generation tests, and what we record with each run.
- How much VRAM a model needs: weights, context and overheadWork out the memory an LLM needs from its size, quantization and context length, with worked examples for 8B, 32B and 70B models and the formula behind the VRAM calculator.
- Local AI builds at 24, 48 and 96 GB: what to buy usedThree sized builds for running LLMs at home, from a single used RTX 3090 to a four-GPU EPYC server, with the parts, the power, the trade-offs and the Mac alternative at each size.