GPUs and Macs for running AI models locally
Pick your hardware to see every open model that fits, the best quant, and an estimated speed ceiling.
Consumer GPUs
| Hardware | Memory | Usable | Bandwidth | Models that fit (8K) |
|---|---|---|---|---|
| RTX 3060 12GB | 12 GB | 11.4 GB | 360 GB/s | 171 |
| RTX 3060 Ti | 8 GB | 7.6 GB | 448 GB/s | 153 |
| RTX 3070 | 8 GB | 7.6 GB | 448 GB/s | 153 |
| RTX 3080 10GB | 10 GB | 9.5 GB | 760 GB/s | 161 |
| RTX 3080 12GB | 12 GB | 11.4 GB | 912 GB/s | 171 |
| RTX 3090 | 24 GB | 22.8 GB | 936 GB/s | 202 |
| RTX 3090 Ti | 24 GB | 22.8 GB | 1008 GB/s | 202 |
| RTX 4060 | 8 GB | 7.6 GB | 272 GB/s | 153 |
| RTX 4060 Ti 8GB | 8 GB | 7.6 GB | 288 GB/s | 153 |
| RTX 4060 Ti 16GB | 16 GB | 15.2 GB | 288 GB/s | 194 |
| RTX 4070 | 12 GB | 11.4 GB | 504 GB/s | 171 |
| RTX 4070 Super | 12 GB | 11.4 GB | 504 GB/s | 171 |
| RTX 4070 Ti | 12 GB | 11.4 GB | 504 GB/s | 171 |
| RTX 4070 Ti Super | 16 GB | 15.2 GB | 672 GB/s | 194 |
| RTX 4080 | 16 GB | 15.2 GB | 717 GB/s | 194 |
| RTX 4080 Super | 16 GB | 15.2 GB | 736 GB/s | 194 |
| RTX 4090 | 24 GB | 22.8 GB | 1008 GB/s | 202 |
| RTX 5060 | 8 GB | 7.6 GB | 448 GB/s | 153 |
| RTX 5060 Ti 16GB | 16 GB | 15.2 GB | 448 GB/s | 194 |
| RTX 5070 | 12 GB | 11.4 GB | 672 GB/s | 171 |
| RTX 5070 Ti | 16 GB | 15.2 GB | 896 GB/s | 194 |
| RTX 5080 | 16 GB | 15.2 GB | 960 GB/s | 194 |
| RTX 5090 | 32 GB | 30.4 GB | 1792 GB/s | 207 |
| RX 7600 | 8 GB | 7.6 GB | 288 GB/s | 153 |
| RX 7700 XT | 12 GB | 11.4 GB | 432 GB/s | 171 |
| RX 7800 XT | 16 GB | 15.2 GB | 624 GB/s | 194 |
| RX 7900 GRE | 16 GB | 15.2 GB | 576 GB/s | 194 |
| RX 7900 XT | 20 GB | 19.0 GB | 800 GB/s | 202 |
| RX 7900 XTX | 24 GB | 22.8 GB | 960 GB/s | 202 |
| RX 9070 | 16 GB | 15.2 GB | 640 GB/s | 194 |
| RX 9070 XT | 16 GB | 15.2 GB | 640 GB/s | 194 |
| Arc B580 | 12 GB | 11.4 GB | 456 GB/s | 171 |
| Arc A770 16GB | 16 GB | 15.2 GB | 560 GB/s | 194 |
Multi-GPU rigs
| Hardware | Memory | Usable | Bandwidth | Models that fit (8K) |
|---|---|---|---|---|
| 2x RTX 3090 | 2×24 GB | 45.6 GB | 936 GB/s | 214 |
| 2x RTX 4090 | 2×24 GB | 45.6 GB | 1008 GB/s | 214 |
| 2x RTX 5090 | 2×32 GB | 60.8 GB | 1792 GB/s | 217 |
Apple Silicon Macs
| Hardware | Memory | Usable | Bandwidth | Models that fit (8K) |
|---|---|---|---|---|
| Mac M1 16GB | 16 GB | 10.7 GB | 68 GB/s | 165 |
| Mac M2 24GB | 24 GB | 16.1 GB | 100 GB/s | 199 |
| Mac M3 24GB | 24 GB | 16.1 GB | 100 GB/s | 199 |
| Mac M4 16GB | 16 GB | 10.7 GB | 120 GB/s | 165 |
| Mac M4 32GB | 32 GB | 21.4 GB | 120 GB/s | 202 |
| Mac M1 Pro 32GB | 32 GB | 21.4 GB | 200 GB/s | 202 |
| Mac M3 Pro 36GB | 36 GB | 24.1 GB | 150 GB/s | 202 |
| Mac M4 Pro 24GB | 24 GB | 16.1 GB | 273 GB/s | 199 |
| Mac M4 Pro 48GB | 48 GB | 32.2 GB | 273 GB/s | 207 |
| Mac M4 Pro 64GB | 64 GB | 48.0 GB | 273 GB/s | 215 |
| Mac M1 Max 64GB | 64 GB | 48.0 GB | 400 GB/s | 215 |
| Mac M2 Max 96GB | 96 GB | 72.0 GB | 400 GB/s | 219 |
| Mac M3 Max 128GB | 128 GB | 96.0 GB | 400 GB/s | 227 |
| Mac M4 Max 64GB | 64 GB | 48.0 GB | 546 GB/s | 215 |
| Mac M4 Max 128GB | 128 GB | 96.0 GB | 546 GB/s | 227 |
| Mac M2 Ultra 192GB | 192 GB | 144 GB | 800 GB/s | 245 |
| Mac M3 Ultra 256GB | 256 GB | 192 GB | 819 GB/s | 248 |
| Mac M3 Ultra 512GB | 512 GB | 384 GB | 819 GB/s | 278 |
Unified-memory PCs
| Hardware | Memory | Usable | Bandwidth | Models that fit (8K) |
|---|---|---|---|---|
| DGX Spark | 128 GB | 109 GB | 273 GB/s | 235 |
| Ryzen AI Max+ 395 128GB | 128 GB | 109 GB | 256 GB/s | 235 |
Workstation GPUs
| Hardware | Memory | Usable | Bandwidth | Models that fit (8K) |
|---|---|---|---|---|
| RTX A6000 | 48 GB | 45.6 GB | 768 GB/s | 214 |
| RTX 6000 Ada | 48 GB | 45.6 GB | 960 GB/s | 214 |
| RTX PRO 6000 Blackwell | 96 GB | 91.2 GB | 1792 GB/s | 227 |