GPUs for running LLMs locally: memory, bandwidth and power
The cards people actually buy for local LLMs, compared on memory, bandwidth and power. Sold prices for each card are tracked daily on GPU Sold.
| Card or part | VRAM | Type | Sold for | Sales |
|---|---|---|---|---|
| NVIDIA RTX 3090 24GB | 24 GB | GPU Sold | elsewhere | |
| NVIDIA RTX 3090 Ti 24GB | 24 GB | GPU Sold | elsewhere | |
| NVIDIA RTX 4090 24GB | 24 GB | GPU Sold | elsewhere | |
| NVIDIA RTX 5090 32GB | 32 GB | GPU Sold | elsewhere | |
| NVIDIA RTX 3060 12GB | 12 GB | GPU Sold | elsewhere | |
| NVIDIA RTX 4060 Ti 16GB | 16 GB | GPU Sold | elsewhere | |
| NVIDIA RTX 5060 Ti 16GB | 16 GB | GPU Sold | elsewhere | |
| NVIDIA RTX A6000 48GB | 48 GB | GPU Sold | elsewhere | |
| NVIDIA RTX A5000 24GB | 24 GB | GPU Sold | elsewhere | |
| NVIDIA RTX Pro 6000 Blackwell 96GB | 96 GB | GPU Sold | elsewhere | |
| NVIDIA Tesla P40 24GB | 24 GB | GPU Sold | elsewhere | |
| NVIDIA Tesla V100 32GB (PCIe) | 32 GB | GPU Sold | elsewhere | |
| AMD Instinct MI50 32GB | 32 GB | GPU Sold | elsewhere | |
| AMD Radeon RX 7900 XTX 24GB | 24 GB | GPU Sold | elsewhere | |
| NVIDIA RTX 2080 Ti 22GB (modded) | 22 GB | GPU Sold | elsewhere |
Before you buy a used GPU for AI
- Memory first. The model and its context have to fit in VRAM (or it spills to system RAM and slows to a crawl). Use the VRAM calculator.
- Bandwidth second. Token generation speed is mostly limited by memory bandwidth, not compute.
- Check memory temperatures under load on GDDR6X cards (3090, 3090 Ti). Worn thermal pads are common on ex-mining cards.
- Plan power and slot spacing before buying two or more cards. Transient spikes on high-end cards trip undersized power supplies.