RunsOnMy ModelsGPUs & MacsAPI pricesVRAM calculatorNew

What AI models run on the A100 40GB?

The A100 40GB has 40 GB of VRAM (1,555 GB/s). About 38.0 GB is usable for a model. 207 of 285 open models we track fit at 8K context.

207models fit (8K ctx)
207still fit at 32K ctx
80Blargest that fits
1555 GB/smemory bandwidth

Models that fit, largest first

ModelParamsBest quant (8K)Needs32K contextMax tok/s (est.)
Hunyuan A13B Instruct80B (13B act.)Q2_K29.0 GBQ2_K327details
Kimi Dev 72B73BQ2_K28.6 GBQ2_K57details
Llama 3.3 70B Instruct71BQ2_K27.8 GBQ2_K59details
DeepSeek R1 Distill Llama 70B71BQ2_K27.8 GBQ2_K59details
Llama 3.1 70B71BQ2_K27.7 GBQ2_K59details
Kimi Linear 48B A3B Instruct49B (3.0B act.)Q5_K_M33.6 GBQ4_K_M725details
Nous Hermes 2 Mixtral 8x7B DPO47B (13B act.)Q5_K_M32.4 GBQ4_K_M170details
falcon 40b42BQ6_K33.6 GBQ5_K_M45details
flint40B (1.3B act.)Q6_K31.3 GBQ6_K1517details
Maxwell V1 Flash37B (2.3B act.)Q6_K29.2 GBQ6_K835details
Hermes 4.3 36B36BQ6_K30.3 GBQ5_K_M52details
Seed OSS 36B Instruct36BQ6_K30.3 GBQ5_K_M52details
Qwen3.6 35B A3B36B (3.0B act.)Q6_K29.0 GBQ6_K621details
Ornith 1.5 35B A3B36B (3.0B act.)Q6_K29.5 GBQ6_K610details
Qwen3.5 35B A3B36B (3.0B act.)Q6_K27.9 GBQ6_K646details
Axion 1.0 35B35B (3.9B act.)Q6_K27.6 GBQ6_K491details
Qwen AgentWorld 35B A3B35B (3.0B act.)Q6_K28.3 GBQ6_K613details
Yi 34B 200K34BQ6_K28.7 GBQ5_K_M55details
Yi 34B34BQ6_K28.7 GBQ5_K_M55details
Yi 34B Chat34BQ6_K28.7 GBQ5_K_M55details
Yi 1.5 34B Chat34BQ6_K28.7 GBQ5_K_M55details
Nous Hermes 2 Yi 34B34BQ6_K29.2 GBQ5_K_M54details
DeepSeek R1 Distill Qwen 32B33BQ6_K27.7 GBQ6_K58details
Qwen3 32B33BQ6_K27.7 GBQ6_K58details
GLM Z1 32B 041433BQ8_033.4 GBQ6_K45details
GLM 4 32B 041433BQ8_033.4 GBQ6_K45details
granite 4.0 h small32B (8.8B act.)Q8_032.6 GBQ8_0166details
gemma 4 31B it31BQ8_033.3 GBQ6_K48details
GLM 4.7 Flash31B (4.6B act.)Q8_030.8 GBQ8_0331details
Qwen3 Coder 30B A3B Instruct31B (3.0B act.)Q8_031.7 GBQ8_0487details
North Mini Code 1.030B (3.3B act.)Q8_031.6 GBQ8_0442details
Hy Memory 30B A3B30B (3.0B act.)Q8_031.1 GBQ8_0488details
Muse Glimmer 30B30BQ8_028.5 GBQ8_053details
granite 4.2 30b29BQ8_031.6 GBQ6_K50details
Qwen3.8 27B28BQ8_028.4 GBQ8_054details
Qwen3.6 27B28BQ8_028.0 GBQ8_054details
Qwen3.5 27B28BQ8_028.0 GBQ8_054details
gemma 3 27b it27BQ8_028.7 GBQ8_054details
Arete 27B27BQ8_028.1 GBQ8_054details
MiniMax H3 Image PE27BQ8_028.0 GBQ8_054details
Trinity Mini26B (2.0B act.)Q8_026.9 GBQ8_0744details
GLiDE 27B26BQ8_027.2 GBQ8_056details
diffusiongemma 26B A4B it26B (4.0B act.)Q8_026.4 GBQ8_0373details
gemma 4 26B A4B it26B (4.0B act.)Q8_026.4 GBQ8_0373details
LFM2.5 24B A2B24B (2.0B act.)Q8_024.5 GBQ8_0731details
ERNIE 4.5 21B A3B PT22BQ8_022.6 GBQ8_067details
ERNIE 4.5 21B A3B Thinking22BQ8_022.6 GBQ8_067details
gpt oss safeguard 20b22B (4.8B act.)F1613.8 GBF16506details
gpt oss 20b21B (4.2B act.)F1613.8 GBF16562details
Groot 19B A3B 26102319B (3.0B act.)Q8_019.9 GBQ8_0498details
Kimi VL A3B Thinking 250616B (3.0B act.)Q8_016.8 GBQ8_0501details
Kimi VL A3B Instruct16B (3.0B act.)F1631.5 GBF16259details
Kimi VL A3B Thinking16B (3.0B act.)F1631.5 GBF16259details
deepseek moe 16b base16B (2.8B act.)F1632.8 GBQ8_0275details
LLaDA2.1 mini16B (1.4B act.)F1631.3 GBF16543details
Moonlight 16B A3B Instruct16B (3.0B act.)F1630.7 GBF16259details
DeepSeek Coder V2 Lite Instruct16B (2.7B act.)F1630.1 GBF16292details
DeepSeek V2 Lite16B (2.7B act.)F1630.1 GBF16292details
DeepSeek V2 Lite Chat16B (2.7B act.)F1630.1 GBF16292details
Qwen2.5 Coder 14B Instruct15BF1629.7 GBQ8_053details
DeepSeek R1 Distill Qwen 14B15BF1629.7 GBQ8_053details
Qwen3 14B15BF1629.4 GBF1653details
phi 415BF1629.5 GBF1653details
gemma 3 12b it12BF1623.6 GBF1666details
Llama Guard 4 12B12BF1624.6 GBF1665details
Llama 3.2 11B Vision Instruct11BF1620.1 GBF1680details
Llama 3.2 11B Vision11BF1621.7 GBF1673details
GLM 4.6V Flash10BF1618.5 GBF1683details
GLM 4.1V 9B Thinking10BF1618.5 GBF1683details
Step3 VL 10B10BF1632.3 GBQ8_047details
SenseNova Loop 9B Preview9.7BF1617.8 GBF1687details
Qwen3.5 9B9.7BF1617.8 GBF1687details
Ornith 1.5 9B9.7BF1617.8 GBF1687details
Qwen3.5 9B Base9.7BF1617.8 GBF1687details
MiMo V2.6 Distill Qwen 9B9.4BF1617.8 GBF1687details
glm 4 9b9.4BF1623.2 GBQ8_083details
internlm3 8b instruct8.8BF1617.4 GBF1688details
granite 4.2 8b8.8BF1618.2 GBF1688details
granite 4.1 8b8.8BF1618.2 GBF1688details
Qwen3 VL 8B Instruct8.8BF1617.1 GBF1695details
Molmo2 8B8.7BF1617.1 GBF1695details
LFM2.5 8B A1B8.5B (1.0B act.)F1632.3 GBF16388details
Qwen2.5 VL 7B Instruct8.3BF1615.3 GBF16102details
UI TARS 1.5 7B8.3BF1615.3 GBF16102details
olmOCR 2 7B 10258.3BF1615.3 GBF16102details
olmOCR 7B 0225 preview8.3BF1616.6 GBF1694details
UI TARS 7B DPO8.3BF1615.3 GBF16102details
Qwen3 8B8.2BF1617.1 GBF1695details
DeepSeek R1 0528 Qwen3 8B8.2BF1617.1 GBF1695details
granite 3.3 8b instruct8.2BF1617.0 GBF1695details
granite 3.1 8b instruct8.2BF1617.0 GBF1695details
granite 3.2 8b instruct8.2BF1617.0 GBF1695details
Apertus 8B Instruct 25098.1BF1616.7 GBF1696details
Llama 3.1 8B Instruct8.0BF1616.6 GBF1697details
Llama 3.1 8B8.0BF1616.6 GBF1697details
Hermes 3 Llama 3.1 8B8.0BF1616.6 GBF1697details
DeepSeek R1 Distill Llama 8B8.0BF1631.6 GBQ8_048details
Meta Llama 3 8B Instruct8.0BF1616.6 GBF1697details
Meta Llama Guard 2 8B8.0BF1616.6 GBF1697details
Meta Llama 3.1 8B Instruct8.0BF1616.6 GBF1697details
Ling 3.0 tiny7.9B (1.4B act.)F1615.6 GBF16562details
MiMo 7B RL7.8BF1616.4 GBF1699details
internlm2 5 7b chat7.7BF1616.0 GBF16100details
internlm2 chat 7b7.7BF1616.0 GBF16100details
Qwen2.5 7B Instruct7.6BF1615.3 GBF16102details
Qwen2.5 Coder 7B Instruct7.6BF1615.3 GBF16102details
DeepSeek R1 Distill Qwen 7B7.6BF1615.3 GBF16102details
Falcon H1R 7B7.6BF1615.1 GBF16102details
Olmo 3 7B Instruct7.3BF1616.7 GBF16107details
Olmo 3 1025 7B7.3BF1616.7 GBF16107details
Olmo 3 7B Think7.3BF1616.7 GBF16107details
Mistral 7B v0.17.2BF1614.9 GBF16107details
Mistral 7B Instruct v0.27.2BF1614.9 GBF16107details
Mistral 7B Instruct v0.17.2BF1614.9 GBF16107details
falcon 7b7.2BF1618.4 GBF16108details
falcon 7b instruct7.2BF1618.4 GBF16108details
granite 4.0 h tiny6.9B (1.5B act.)F1613.6 GBF16530details
OLMoE 1B 7B 09246.9B (1.3B act.)F1614.4 GBF16606details
deepseek coder 7b instruct v1.56.9BF1617.2 GBF16113details
deepseek coder 6.7b instruct6.7BF1617.0 GBF16115details
deepseek coder 6.7b base6.7BF1617.0 GBF16115details
Llama 2 7b hf6.7BF1617.0 GBF16115details
Yi 6B 200K6.1BF1612.3 GBF16128details
Yi 6B6.1BF1612.3 GBF16128details
Molmo2 4B4.9BF1610.0 GBF16176details
WeVisOne VL 4B4.8BF1610.0 GBF16176details
Mage VL Cowork4.7BF1610.0 GBF16176details
Qianfan OCR4.7BF1610.1 GBF16176details
Qwen3.5 4B4.7BF169.0 GBF16185details
S5 nano4.7BF169.2 GBF16179details
LightOnOCR 3 4B4.5BF169.0 GBF16185details
SingVL S1 4B4.5BF169.0 GBF16185details
Intern Decision 4B4.5BF169.0 GBF16185details
Qwen3 VL 4B Instruct4.4BF169.3 GBF16193details
gemma 3 4b it4.3BF168.4 GBF16200details
medgemma 1.5 4b it4.3BF168.4 GBF16200details
medgemma 4b it4.3BF168.4 GBF16200details
gemma 3 4b pt4.3BF169.2 GBF16181details
Phi 3.5 vision instruct4.1BF1611.2 GBF16188details
Qwen3 4B4.0BF169.3 GBF16193details
Qwen3 4B Instruct 25074.0BF169.3 GBF16193details
granite vision 4.1 4b4.0BF167.6 GBF16228details
Phi 4 mini instruct3.8BF168.9 GBF16202details
Phi 3 mini 4k instruct3.8BF1610.6 GBF16203details
Phi 3.5 mini instruct3.8BF1610.6 GBF16203details
Phi 3 mini 128k instruct3.8BF1610.6 GBF16203details
Qwen2.5 VL 3B Instruct3.8BF166.7 GBF16252details
granite 4.2 3b3.7BF168.0 GBF16212details
granite 4.1 3b llmon preview3.7BF168.0 GBF16212details
Raon 1 nano3.5BF167.0 GBF16248details
granite 4.1 3b3.4BF167.6 GBF16228details
granite 4.0 micro3.4BF167.6 GBF16228details
DeepSeek OCR 23.4B (1.2B act.)F166.6 GBF16751details
deepseek vl2 tiny3.4B (1.2B act.)F167.4 GBF16661details
Unlimited OCR 23.3B (1.1B act.)F166.6 GBF16769details
Unlimited OCR3.3B (1.1B act.)F166.6 GBF16774details
DeepSeek OCR3.3B (1.1B act.)F166.6 GBF16774details
granite 3.1 3b a800m instruct3.3B (883M act.)F167.1 GBF16880details
Llama 3.2 3B Instruct3.2BF1613.5 GBF16121details
Llama 3.2 3B3.2BF167.5 GBF16242details
Hermes 3 Llama 3.2 3B3.2BF167.5 GBF16242details
d1 3B3.1BF1610.8 GBF16144details
LFM2.5 VL 3B3.1BF1610.8 GBF16144details
Qwen2.5 3B Instruct3.1BF166.7 GBF16252details
SmolLM3 3B3.1BF166.9 GBF16253details
SmolLM3 3B Base3.1BF166.9 GBF16253details
phi 22.8BF168.2 GBF16280details
LFM2.5 2.6B2.7BF1610.8 GBF16144details
granite 3.3 2b instruct2.5BF165.8 GBF16307details
MiniCPM5 2B2.5BF165.7 GBF16309details
WeVisOne VL 2B2.4BF165.4 GBF16382details
Qwen3.5 2B2.3BF164.5 GBF16412details
SmolVLM Instruct2.2BF165.4 GBF16429details
Qwen3 1.7B2.0BF164.8 GBF16451details
Qwen3.8 27B DSpine b162.0BF164.8 GBF16383details
d1 2B1.9BF167.2 GBF16224details
LFM3 VL 2B1.9BF167.2 GBF16224details
DeepSeek R1 Distill Qwen 1.5B1.8BF164.2 GBF16437details
LFM3 1.7B1.7BF167.2 GBF16224details
LFM3 1.7B Base1.7BF164.0 GBF16450details
SmolLM2 1.7B Instruct1.7BF165.2 GBF16454details
SmolLM2 1.7B1.7BF165.2 GBF16454details
SmolLM 1.7B1.7BF165.2 GBF16454details
granite 4.0 1b1.6BF164.3 GBF16476details
Qwen2.5 1.5B Instruct1.5BF163.8 GBF16504details
OLMo 2 0425 1B1.5BF164.4 GBF16523details
deepseek coder 1.3b instruct1.3BF164.5 GBF16577details
Llama 3.2 1B1.2BF163.2 GBF16629details
Llama 3.2 1B Instruct1.2BF165.5 GBF16314details
LFM2.5 1.2B Instruct1.2BF165.0 GBF16332details
HunyuanOCR1.1BF163.5 GBF16694details
PaddleOCR VL 1.6959MF162.5 GBF16811details
Qwen3.5 0.8B873MF162.4 GBF161025details
LightOnOCR 3 0.8B853MF162.4 GBF161025details
FastVLM 0.5B759MF162.2 GBF161025details
Qwen3 0.6B752MF162.7 GBF161298details
GOT OCR2 0716MF162.8 GBF161086details
Nemotron 3.5 Parse631MF161.9 GBF161190details
GOT OCR 2.0 hf561MF162.5 GBF161387details
SmolVLM2 500M Video Instruct507MF161.6 GBF161895details
SmolVLM 500M Instruct507MF161.6 GBF161895details
Qwen2.5 0.5B Instruct494MF161.7 GBF161574details
LFM2.5 VL 450M449MF161.9 GBF161093details
SmolLM2 360M Instruct362MF161.5 GBF162143details
SmolLM2 360M362MF161.5 GBF162143details
LFM2.5 350M354MF161.9 GBF161093details
LFM2 350M354MF161.4 GBF162193details

Just too big at 4-bit

These need up to 35% more memory than the A100 40GB has at Q4_K_M and 8K context. A smaller quant or shorter context may work.

ModelQ4_K_M needsShort by
Llama 3.1 70B42.7 GB4.7 GB
Llama 3.3 70B Instruct43.3 GB5.3 GB
DeepSeek R1 Distill Llama 70B43.3 GB5.3 GB
Kimi Dev 72B44.6 GB6.6 GB
Hunyuan A13B Instruct47.1 GB9.1 GB
Mistral 7B Instruct v0.247.7 GB9.7 GB

Questions

What is the biggest model that runs on the A100 40GB?

Of the models we track, the largest that fits at 8K context is Hunyuan A13B Instruct (80B) at Q2_K, needing about 29.0 GB.

How much of the A100 40GB's memory can a model use?

About 38.0 GB: we assume 95% of the 40 GB is usable after driver and display overhead.

Spec source: NVIDIA specifications. Model data: Hugging Face Hub API. Speed is a bandwidth ceiling (bandwidth ÷ bytes read per token), not a benchmark. Methodology · Report an error.