Local AI Hardware / GPUs for local AI
NVIDIA RTX 4090 24GB for local AI
Sold prices for the NVIDIA RTX 4090 24GB are tracked every day on GPU Sold (https://gpusold.com/gpu/rtx-4090-24).
Much faster than a 3090 at prompt processing, same 24 GB of memory.
- Memory
- 24 GB
- Memory bandwidth
- 1008 GB/s
- Board power
- 450 W
- Architecture
- Ada Lovelace
What matters for AI
For token generation, memory bandwidth matters most, and the 4090's is only about 8% higher than the 3090's. The big gains are in prompt processing and in anything compute-bound, like image generation.
Before you buy a used GPU for AI
- Memory first. The model and its context have to fit in VRAM (or it spills to system RAM and slows to a crawl). Use the VRAM calculator.
- Bandwidth second. Token generation speed is mostly limited by memory bandwidth, not compute.
- Check memory temperatures under load on GDDR6X cards (3090, 3090 Ti). Worn thermal pads are common on ex-mining cards.
- Plan power and slot spacing before buying two or more cards. Transient spikes on high-end cards trip undersized power supplies.