Can I run falcon 40b locally?
falcon 40b by tiiuae has 42B parameters. At 4-bit with an 8K context it needs about 25.2 GB of GPU memory. It fits on 33 of 65 GPUs and Macs we track.
Memory needed by quant and context length
GB of GPU memory = weights + KV cache (FP16) + runtime overhead. Weights use real file sizes from mradermacher/falcon-40b-i1-GGUF where available (marked •), otherwise parameter count × bits per weight. How we calculate.
| Quant | Quality | 4K ctx | 8K ctx | 32K ctx | 128K ctx |
|---|---|---|---|---|---|
| 2-bit (Q2_K) • | rough | 15.7 GB | 16.1 GB | 19.0 GB | 30.2 GB |
| 3-bit (Q3_K_M) • | usable | 19.7 GB | 20.2 GB | 23.0 GB | 34.2 GB |
| 4-bit (Q4_K_M) • | good, most popular | 24.7 GB | 25.2 GB | 28.0 GB | 39.3 GB |
| 5-bit (Q5_K_M) • | very good | 29.5 GB | 30.0 GB | 32.8 GB | 44.1 GB |
| 6-bit (Q6_K) • | near lossless | 33.1 GB | 33.6 GB | 36.4 GB | 47.6 GB |
| 8-bit (Q8_0) | effectively lossless | 42.4 GB | 42.9 GB | 45.7 GB | 56.9 GB |
| 16-bit (F16/BF16) | original weights | 78.9 GB | 79.4 GB | 82.2 GB | 93.4 GB |
Which GPUs and Macs run falcon 40b?
Best quality quant that fits at 8K context. "Tight" = within 10% of usable memory. Speed is a ceiling from memory bandwidth (real speed is typically 50–80% of it).
Questions
How much VRAM does falcon 40b need?
About 25.2 GB at 4-bit (Q4_K_M) with an 8K context: 23.7 GB of weights (real GGUF file size), 0.9 GB of KV cache and 0.5 GB of runtime overhead. At 8-bit it needs about 42.9 GB.
What is the smallest GPU that runs falcon 40b?
At Q4_K_M and 8K context, the smallest single consumer GPU in our table that fits it is the RTX 5090 (32 GB).
Can I run falcon 40b on a Mac?
Yes. At Q4_K_M with 8K context it fits on a Mac M4 Pro 48GB or larger, using the default macOS GPU memory limit.
Model facts
| Hugging Face repo | tiiuae/falcon-40b |
| Parameters | 41,835,970,560 (42B) |
| Layers / hidden size | 60 / 8192 |
| Attention | GQA; 128 heads, 8 KV heads, head dim 64 |
| KV cache per token (FP16) | 120.0 KiB |
| Max context | — |
| License | apache-2.0 |
| Released on HF | May 24, 2023 |
| Downloads (30d) / likes | 11,303 / 2,450 |
Sources: Hugging Face model API · config.json (tiiuae/falcon-40b) · GGUF files · GPU specs from vendor pages (list). Data fetched Nov 17, 2026. Spot an error? Report it.