dynamic VRAM offloading

Dynamic VRAM offloading is a way of temporarily moving part of a model out to the computer's regular memory when the graphics card's memory isn't enough. It's like a desk too small to lay out an entire book at once, so you keep only the page you're reading on the desk and stash the rest in a nearby drawer, swapping pages in and out as needed.

Graphics card memory (VRAM) is fast but limited in size, while a computer's regular memory is slower but much larger. If a model is bigger than the VRAM, it can't all be loaded at once — so only the part needed for the current calculation is sent to the graphics card while the rest waits elsewhere. This lets even large models run on smaller graphics cards.

But it isn't free. Just as opening and closing a drawer takes time, moving memory back and forth slows down computation. That's why results run this way are generally slower than running on a dedicated, high-end graphics card.

Articles describe this with phrases like "a 33B-parameter model running on an RTX 4070 laptop." A common misreading is that laptop GPU performance has simply gotten that good — but in reality, it's more likely that only parts of the model were loaded and swapped in as needed, rather than the whole model fitting in memory. In other words, it's often not a leap in performance but a memory-saving technique at work.

When using a tool to run large models locally, check for an offload or CPU offload setting. Turning this option on lets you directly compare how VRAM usage drops while generation speed slows down.