llama.cpp offload calculator: -ngl and --n-cpu-moe
When a model is bigger than your GPU, llama.cpp can keep part of it in system RAM. Pick a model, your GPU and your RAM to get a starting command, the split, and an estimated speed range. For example, gpt-oss-120b on a 16 GB RTX 4060 Ti: llama-server -hf bartowski/openai_gpt-oss-120b-GGUF:Q4_K_M -c 8192 -ngl 999 --n-cpu-moe 29, on a PC with 64 GB of RAM, roughly 13–25 tokens/sec.
Loading…
8K context. Speed spans dual-channel DDR4-3200 to DDR5-6000 system RAM and is capped at what the GPU alone could do. Estimates, not benchmarks; how we estimate offload.
-ngl and --n-cpu-moe, explained
-ngl N (--n-gpu-layers) puts the first N transformer layers on the GPU and runs the rest on the CPU. Every token passes through every layer, so the CPU layers set the pace: dense models slow down sharply once even a third of the model sits in RAM.
--n-cpu-moe N is for mixture-of-experts models such as gpt-oss and Qwen3-30B-A3B. It keeps only the expert weights of the first N layers in RAM, while attention, shared weights and the KV cache stay on the GPU (use it with -ngl 999). Each token uses only a few experts, so far less data crosses from RAM per token. That is why a 120B MoE model can be usable on a 16 GB card while a 70B dense model is not. --cpu-moe keeps all expert weights on the CPU.
Recent llama.cpp builds also fit models automatically: -ngl defaults to auto and --fit is on, so a bare llama-server -hf repo:Q4_K_M picks a split for you. Setting the split yourself, as below, is still useful when you want a specific context length or to leave memory free for other apps.
Other flags that change memory: -c sets the context length (KV cache grows with it), and -ctk q8_0 -ctv q8_0 with -fa on halves the KV cache. The VRAM calculator shows the effect of KV cache quantization.
CUDA build (-DGGML_CUDA=ON) or the prebuilt CUDA releases.
Apple Silicon
Metal is on by default in macOS builds.
AMD
HIP/ROCm build, or Vulkan.
Intel Arc and iGPUs
SYCL build, or Vulkan.
CPU only
Any modern x86 or ARM CPU works; speed is limited by RAM bandwidth (roughly 5–10 tokens/sec for an 8B model at 4-bit on dual-channel DDR4 to DDR5, by the same bandwidth estimate).
Memory
GPU memory for the part of the model on the GPU, plus system RAM for the rest and for the OS. Use the calculator above.
Automated data from the Hugging Face Hub API and GGUF file listings; formulas on the methodology page. Not affiliated with the llama.cpp project. Report an error.