Local AI Hardware

GPUs for running LLMs locally: memory, bandwidth and power

The cards people actually buy for local LLMs, compared on memory, bandwidth and power. Sold prices for each card are tracked daily on GPU Sold.

Card or partVRAMBandwidthPowerTypeSold forSales
NVIDIA RTX 3090 24GB24 GB936 GB/s350 WGPU Soldelsewhere
NVIDIA RTX 3090 Ti 24GB24 GB1008 GB/s450 WGPU Soldelsewhere
NVIDIA RTX 4090 24GB24 GB1008 GB/s450 WGPU Soldelsewhere
NVIDIA RTX 5090 32GB32 GB1792 GB/s575 WGPU Soldelsewhere
NVIDIA RTX 3060 12GB12 GB360 GB/s170 WGPU Soldelsewhere
NVIDIA RTX 4060 Ti 16GB16 GB288 GB/s165 WGPU Soldelsewhere
NVIDIA RTX 5060 Ti 16GB16 GB448 GB/s180 WGPU Soldelsewhere
NVIDIA RTX A6000 48GB48 GB768 GB/s300 WGPU Soldelsewhere
NVIDIA RTX A5000 24GB24 GB768 GB/s230 WGPU Soldelsewhere
NVIDIA RTX Pro 6000 Blackwell 96GB96 GB1792 GB/s600 WGPU Soldelsewhere
NVIDIA Tesla P40 24GB24 GB347 GB/s250 WGPU Soldelsewhere
NVIDIA Tesla V100 32GB (PCIe)32 GB900 GB/s250 WGPU Soldelsewhere
AMD Instinct MI50 32GB32 GB1024 GB/s300 WGPU Soldelsewhere
AMD Radeon RX 7900 XTX 24GB24 GB960 GB/s355 WGPU Soldelsewhere
NVIDIA RTX 2080 Ti 22GB (modded)22 GB616 GB/s250 WGPU Soldelsewhere

Before you buy a used GPU for AI

  1. Memory first. The model and its context have to fit in VRAM (or it spills to system RAM and slows to a crawl). Use the VRAM calculator.
  2. Bandwidth second. Token generation speed is mostly limited by memory bandwidth, not compute.
  3. Check memory temperatures under load on GDDR6X cards (3090, 3090 Ti). Worn thermal pads are common on ex-mining cards.
  4. Plan power and slot spacing before buying two or more cards. Transient spikes on high-end cards trip undersized power supplies.