Local AI Hardware / GPUs for local AI
NVIDIA RTX 3090 24GB for local AI
Sold prices for the NVIDIA RTX 3090 24GB are tracked every day on GPU Sold (https://gpusold.com/gpu/rtx-3090-24).
The used card most local-LLM builders start with: 24 GB for roughly the price of a new 16 GB card.
- Memory
- 24 GB
- Memory bandwidth
- 936 GB/s
- Board power
- 350 W
- Architecture
- Ampere
What matters for AI
24 GB fits a 32B model at 4-bit with a context of around 8k tokens (more with an 8-bit KV cache), which 16 GB cards cannot do. Two of them (48 GB) run 70B models at 4-bit.
Most 3090s are three slots wide and draw 350 W, with short spikes well above that. Plan the case, the PSU and the slot spacing before buying two.
Before you buy a used GPU for AI
- Memory first. The model and its context have to fit in VRAM (or it spills to system RAM and slows to a crawl). Use the VRAM calculator.
- Bandwidth second. Token generation speed is mostly limited by memory bandwidth, not compute.
- Check memory temperatures under load on GDDR6X cards (3090, 3090 Ti). Worn thermal pads are common on ex-mining cards.
- Plan power and slot spacing before buying two or more cards. Transient spikes on high-end cards trip undersized power supplies.