Local AI Hardware / GPUs for local AI

NVIDIA RTX 4090 24GB for local AI

Sold prices for the NVIDIA RTX 4090 24GB are tracked every day on GPU Sold (https://gpusold.com/gpu/rtx-4090-24).

Much faster than a 3090 at prompt processing, same 24 GB of memory.

Memory
24 GB
Memory bandwidth
1008 GB/s
Board power
450 W
Architecture
Ada Lovelace

What matters for AI

For token generation, memory bandwidth matters most, and the 4090's is only about 8% higher than the 3090's. The big gains are in prompt processing and in anything compute-bound, like image generation.

Before you buy a used GPU for AI

  1. Memory first. The model and its context have to fit in VRAM (or it spills to system RAM and slows to a crawl). Use the VRAM calculator.
  2. Bandwidth second. Token generation speed is mostly limited by memory bandwidth, not compute.
  3. Check memory temperatures under load on GDDR6X cards (3090, 3090 Ti). Worn thermal pads are common on ex-mining cards.
  4. Plan power and slot spacing before buying two or more cards. Transient spikes on high-end cards trip undersized power supplies.