How we benchmark GPUs for local LLMs
The exact method behind every speed on this site: llama.cpp's llama-bench, pinned versions, fixed model files and quantizations, prompt and generation tests, and what we record with each run.
Published speeds for local LLMs disagree by large margins. The usual reasons are not the hardware: different software versions, different quantizations of the "same" model, different context lengths, and different definitions of "tokens per second". We fix all of those and write them down.
The tool
We use llama-bench, the benchmark that ships with llama.cpp. It reports two numbers, and we publish both:
- Prompt processing (pp): how fast the model reads input, in tokens per second. Default test: 512 tokens (pp512). This is mostly compute-bound, so newer GPUs pull far ahead here.
- Token generation (tg): how fast it writes output. Default test: 128 tokens (tg128). This is mostly limited by memory bandwidth.
We also measure generation with a longer context already in memory, because speed drops as the context grows and short-context numbers flatter every card.
What we hold fixed
- The llama.cpp build, recorded by build number, and the backend (CUDA, ROCm or Metal) with its version.
- The model files: the same GGUF files for every card, identified by name and checksum.
- Quantization: Q4_K_M as the main comparison, Q8_0 where the card has room.
- Settings: all layers on the GPU, flash attention on where supported, default batch sizes, and llama-bench's default of five repetitions per test, reporting the average.
- The machine: driver version, power limit, and the rest of the rig, recorded with each run.
What we do not do
- We do not copy numbers from other sites, even when they look right.
- We do not publish a card we have not measured. Its page says so.
- We do not mix results from different llama.cpp versions in one comparison without saying so.
Reading our tables
Every run lists the model, quantization, context, both speeds, the backend and the date. If you want to compare with your own machine, run llama-bench with the same model file and settings; results within a few percent mean your setup is behaving normally.