Run 100B+ MoE Models on a 16GB GPU: The 2026 CPU-Offload Hardware Guide

The Short Answer: Buy VRAM for Attention, Buy RAM for Experts

On a Mixture-of-Experts model, only the always-active parameters need to live in VRAM — the expert weights can stream from system RAM, which is why a 16GB GPU paired with 64GB of DDR5 runs 100B+ MoE models that a 32GB GPU cannot fit alone. That single sentence inverts the advice on almost every "best GPU for local LLM" page written before 2026, including some of ours. It is also, for the models people actually download today, correct.

Here is the buy recommendation before any explanation:

RAM prices below are as of August 2026 and are moving weekly in the current DRAM market — verify current pricing before you commit. GPU prices are our tracked ranges and are linked to live listings.

If you only read this far: the RTX 5060 Ti 16GB plus 64GB of fast DDR5 is the highest models-per-dollar configuration in local AI right now, and it is not close. The rest of this guide is why, with sourced numbers, and the specific condition under which the recommendation flips back to buying VRAM.

Why MoE Changed the VRAM Math

A dense model does all of its work with all of its weights. Every token that passes through a dense 70B model touches all 70 billion parameters, which is why the VRAM rule was simple: total parameters × bytes-per-parameter, and if it does not fit, you do not run it. That rule produced the entire 2024–2025 genre of GPU buying guides ranked purely by memory capacity, and our own VRAM guide still applies cleanly to dense models like Qwen 3 72B and DeepSeek R1 70B.

A Mixture-of-Experts model does not work that way. Its feed-forward layers are split into many parallel "experts," and a small router network picks a handful of them per token. The naming convention tells you the whole story: a model labelled 35B-A3B has 35 billion total parameters but only about 3 billion active per token. The other 32 billion sit there being irrelevant to that particular token.

Two consequences follow, and they point in opposite directions:

• Compute scales with active parameters. A 35B-A3B model does roughly the arithmetic of a 3B dense model per token, which is why MoE models feel dramatically faster than their parameter count suggests.
• Memory scales with total parameters. All 35 billion weights still have to exist somewhere, because the router might select any of them on the next token.

The naive reading of that second point is "so you still need 35B worth of VRAM." The insight that makes this whole guide possible is that somewhere does not have to mean VRAM. If the experts are individually cold — hit a small fraction of the time, and cheap to compute when they are hit — they can live in system RAM at a fraction of the cost per gigabyte, and only the hot path needs the fast memory.

This is not a hack. The academic framing is the same: "Efficient CPU-GPU Collaborative Inference for MoE-based LLMs on Memory-Limited Systems" (arXiv 2512.16473) formalises the split — keep the dense, always-active components on the accelerator, keep the sparsely-activated expert weights in host memory, and schedule the transfer around the router's decisions. The ik_llama.cpp hybrid inference documentation reaches the same architectural conclusion from the implementation side: attention on GPU, experts on CPU, because attention is what every token needs and experts are what only some tokens need.

To be explicit about scope: this post supersedes "just buy more VRAM" for MoE models only. If you run dense models, fine-tune, or generate images and video, the old rules are unchanged and our consumer GPU guide is still the right starting point.

What --n-cpu-moe Actually Does (and Why It Isn't -ngl)

This is the only software section, and it exists because misunderstanding this flag is the single most common reason people conclude "offload is too slow" and go buy a card they did not need.

llama.cpp has always had -ngl / --n-gpu-layers: put the first N transformer layers entirely on the GPU, leave the rest entirely on the CPU. When a model does not fit, you lower -ngl until it loads. The problem is that a transformer layer is not homogeneous — it contains attention (small, bandwidth-hungry, needed by every token) and a feed-forward block (large, and on an MoE model, mostly idle). Blanket layer offload moves both. You end up with attention computed on the CPU, which is exactly the part you could least afford to move.

--n-cpu-moe N makes the cut along the other axis. It moves only the MoE feed-forward expert tensors of the first N layers to the CPU, leaving attention, the router, the embeddings, and any shared experts resident on the GPU. The maintainers at ggml-org/llama.cpp discuss the exact tensor targeting behaviour in the project's MoE offload discussions (see discussion #22183) — worth reading directly rather than through a paraphrase, because the set of tensors matched has shifted between releases.

The practical difference is large. Same model, same card, same total bytes moved off the GPU:

Two operational notes that save an afternoon. First, tune context length before you tune the flag — the KV cache stays on the GPU, so a 128K context window can consume more VRAM than the resident weights do. Cut context first, then raise --n-cpu-moe in small steps. Second, this all assumes GGUF weights at a sane quantization level; Q4_K_M is the standard starting point and is what nearly every number quoted below was measured at.

Starting Values by GPU

Copy-pasteable starting points. These come from Lena Fischer's per-GPU testing in "llama.cpp --n-cpu-moe: Run a Big MoE Model on a Small GPU" (Aliteq, 27 July 2026), cross-checked against community configurations — including David Sanftenberg's partial-offload walkthrough for Qwen3-235B-A22B-class models, which is the best-documented example of tuning this at the very large end.

The tuning loop is: start at the table value, confirm it loads, then decrease by 2–4 at a time and re-test until you get an out-of-memory error. Step back one. That last configuration is your maximum speed for that model, quant, and context length — change any of the three and you re-tune.

Real Throughput Numbers on Cheap Hardware

This is the proof section, and it comes with a standing caveat we will repeat: every figure below is a third-party measurement, not a Compute Market benchmark, and throughput on offloaded MoE models swings roughly 2× on context length alone. A number without its context length is not a number. All figures are 4-bit-class quants unless noted.

Read the 5060 Ti rows together and the shape of the technique becomes obvious. The same card reports roughly 98 tok/s on a 35B-A3B at 262K context in one submission and roughly 44 tok/s on a similar model at 100K in another — that is not measurement noise, it is different quants, different KV cache configurations, and different offload settings on the same silicon. Anyone quoting you a single "MoE tok/s" figure for a GPU is quoting you an artifact of their config.

What survives all that variance is the important part: a $429–$479 card is producing roughly 40–100 tokens per second on models with 35 billion total parameters, at context lengths in the six figures. Human reading speed is roughly 5–8 tok/s. Even the pessimistic end of that table is more than five times faster than you can read.

The "Private LLM Inference on Consumer Blackwell GPUs" paper (arXiv 2601.09527) provides the independent consumer-GPU throughput grounding for the Blackwell figures, and is worth citing if you need something peer-reviewable rather than community-sourced.

The models these numbers are about are the ones people are actually downloading — see our per-model hardware guides for Qwen 3.6, GLM 5.2, Kimi K2.6, and DeepSeek V4 Flash, all of which are MoE architectures and all of which this technique applies to.

Storage is the constraint nobody budgets for

A 120B-parameter MoE at Q4_K_M is roughly 60–70GB on disk. Keep three of those plus a couple of 35B-class models and you have consumed 250GB before counting anything else. Worse, the model has to be read into RAM on every cold start, so a slow drive turns a 20-second load into a three-minute one — and with offload you reload more often, because you are re-tuning.

A Samsung 990 Pro 4TB NVMe ($289 – $339) is the pragmatic answer: enough capacity that you stop deleting models to try new ones, and sequential reads fast enough that load time stops being a factor in whether you bother experimenting. Our NVMe guide for local AI covers the alternatives if you want to spend less.

The 2026 Catch: RAM Isn't Cheap Anymore

Every guide written in 2025 that recommended this strategy ended with "and just buy 128GB of RAM, it's cheap." That sentence has not aged well. The DRAM market tightened hard through 2026 — the dynamics are in our DRAM shortage explainer, and both the Tom's Hardware RAM price tracking and Newegg Insider's DDR5 supply coverage document the same trend. 128GB DDR5 kits have been running roughly $1,200–$1,400 for 4800MHz, with premium 5600MHz kits sighted well above $4,000 at the worst of it. Verify current pricing before you buy; these move weekly.

That changes the arithmetic enough to be worth doing honestly. Three paths, using our tracked GPU prices and August 2026 memory pricing:

Read the middle row carefully, because it is the finding this post exists to publish: Path B lands within a few hundred dollars of Path C while being slower at everything the 5090 can actually fit. The offload strategy wins decisively at 64GB — Path A is a third of the price of Path C and runs larger total-parameter models. At 128GB the margin narrows to the point where it becomes a judgment call about which specific models you need, and if DDR5 prices climb further it inverts entirely.

Two refinements before you spec a kit:

• Bandwidth gates throughput, not capacity. Offloaded expert weights are read from system memory on every token that routes to them. Dual-channel DDR5-4800 gives roughly 76.8 GB/s theoretical; DDR5-6000 gives about 96 GB/s. That ~25% difference lands directly on the bottleneck. A 64GB DDR5-6000 kit will generally beat a 128GB DDR5-4800 kit on any model that fits in both.
• Check your CPU's real memory controller limits before buying four sticks. Many consumer platforms drop their supported speed when all four DIMM slots are populated, which means a 4×32GB configuration can end up slower than 2×32GB. Two large sticks beats four small ones.

Our how much RAM local AI needs guide is the general-capacity companion to this section; treat this post as its MoE-specific sequel, because MoE offload is the one workload where the answer is meaningfully larger than the general one.

When You Should Not Offload

The reason to trust the rest of this guide is that this section exists. Four cases where --n-cpu-moe is the wrong tool:

• Dense models. There is no sparsity to exploit. Every offloaded weight in a dense model is read for every single token, so you are running the entire model at DDR5 bandwidth. The penalty is brutal and there is no configuration that fixes it — this is the case where the old "buy more VRAM" advice is still exactly right. Our cheapest 32GB GPU guide is written for you.
• Training and fine-tuning. Backpropagation touches all parameters including the experts, gradients and optimizer states need to be resident, and the access pattern has none of the sparsity that makes inference offload work. See our fine-tuning GPU guide — this is a VRAM problem, full stop.
• Long-context serving and batching. With many concurrent requests, the KV cache — which stays on the GPU — becomes the dominant memory consumer, and expert offload does nothing to relieve it. Batching also destroys the routing sparsity assumption: with enough concurrent tokens in flight, most experts get activated on most batches, and you are back to reading everything.
• Your model already fits. If a 35B-A3B model loads on your 24GB card at your working context, set --n-cpu-moe 0 and stop reading. Offloading a model that fits is pure loss.

If you land in the first three buckets, the alternative to a single big card is usually several smaller ones — our multi-GPU local LLM setup guide covers the wiring, and RTX 4090 vs RTX 3090 covers the used-card value math that makes those builds work.

Apple Silicon: Unified Memory Does This for Free

On a Mac the entire question dissolves. Unified memory is simultaneously system RAM and GPU memory — there is no PCIe bus between the experts and the accelerator, no offload decision to tune, and no --n-cpu-moe value to guess at. A 128GB Mac Studio M4 Max ($1,999 – $5,999) simply loads a 120B MoE model and runs it.

This is why high-memory Macs punch far above their apparent price on large MoE workloads specifically. Apple's memory bandwidth sits between desktop DDR5 and discrete GDDR7, which is a mediocre place to be for a dense model that fits in a 5090 — and an excellent place to be for a 120B MoE that does not fit in anything else on your desk. Our MLX vs llama.cpp on Apple Silicon comparison covers which runtime to use once you are there, and the Apple Silicon for AI hub collects the rest.

The trade-off is unchanged from every other Apple-vs-NVIDIA discussion on this site: no CUDA, weaker fine-tuning story, and a price that starts where a complete Path A build ends. But if your requirement is "run the biggest MoE models with the least fiddling," it is the least-effort answer that exists.

What to Buy: Three Builds

Budget — the "does this even work" build

An Intel Arc B580 12GB ($249 – $289) plus 32–64GB of DDR5. Twelve gigabytes of VRAM and the cheapest entry into this technique that we would recommend — --n-cpu-moe 32 as a starting value, Vulkan or SYCL build of llama.cpp, and 30–40B-class MoE models run fine. The catch is ecosystem friction: Intel's stack works but gets upstream attention later than CUDA, so expect occasional build gymnastics. Our Arc B580 for local AI review has the detail, and RTX 5060 Ti vs Arc B580 is the direct comparison.

The RTX 4060 Ti 16GB ($399 – $449) is the alternative budget pick and, honestly, the better one if you can stretch: four extra gigabytes of VRAM and CUDA out of the box.

Its weakness is memory bandwidth — 288 GB/s against the 5060 Ti's 448 GB/s — which matters on the resident attention layers. The head-to-head works through it. Buy the 4060 Ti if you find one meaningfully under the 5060 Ti's street price; otherwise pay the difference.

Sweet spot — the build this post is arguing for

RTX 5060 Ti 16GB ($429 – $479) + 64GB DDR5-6000 + Samsung 990 Pro 4TB ($289 – $339). Sixteen gigabytes of GDDR7 at 448 GB/s holds attention and KV cache for a six-figure context, 64GB of fast system RAM holds the experts for anything up to roughly 120B total parameters at 4-bit, and 4TB of NVMe means you stop deleting models to make room. Around $970–$1,220 for the core three components — roughly half of Path C, and about $700–$900 if you already have a drive you can live with.

Before you commit, RTX 5090 vs RTX 5060 Ti lays out what the four-times-price card actually buys you, and RTX 5060 Ti vs 5070 Ti covers the one-step-up question. The short version of both: for MoE inference specifically, the extra money buys speed on models you can already run, not access to models you cannot.

No compromise — when offload is the wrong answer

An RTX 5090 32GB ($1,999 – $2,199), or two used RTX 3090s ($699 – $999 each) for 48GB of aggregate VRAM at a similar total. Buy this tier if you fine-tune, run dense models, serve concurrent users, or generate video — all the workloads listed in the "don't offload" section. The RTX 4090 ($1,599 – $1,999) sits between the two and remains a strong used-market buy.

Whatever you pick, budget for the rest of the build: the accessory bundle below covers the PSU, cooling, and cabling, and our PSU sizing guide explains why an AI box needs different power planning than a gaming rig.

The Verdict

• Running MoE models? A 16GB card plus 64GB of fast DDR5 beats a 32GB card plus stock RAM, at roughly a third of the price.
• Running dense models, training, or serving? Buy VRAM. Nothing in this post applies to you.
• Already own a 12GB card? A RAM upgrade to 64GB unlocks 100B+ MoE models today for a few hundred dollars. Do that before you replace the GPU.
• Considering 128GB of DDR5? Price it against an RTX 5090 first. At August 2026 memory prices the gap has narrowed enough that the answer is no longer automatic.
• Want none of this fiddling? Unified memory on Apple Silicon does it without a flag.

The reason this recommendation is unusual is that it points down-market, and most hardware content does not. But the models changed underneath the advice: when the frontier open-weight releases are 35B-A3B and 120B-A5B rather than dense 70B, the binding constraint moved from "how much fast memory can I afford" to "how much total memory can I address." Those are very different purchases.

Start at the local LLM guide hub if you are new to running models locally, the Ollama setup guide if you want the easy on-ramp before touching llama.cpp flags, or AI on a budget for more of this genre — every recommendation on that hub is chosen the same way this one was.