GPU VRAM & Deployment Fit Planner

A GPU VRAM planner calculates whether a language model will actually fit on your hardware in production, which is a different and much harder question than whether it fits at all. Most VRAM estimates stop at model weights: multiply parameters by bytes per parameter and compare against the card. That number tells you whether the model loads. It does not tell you whether it serves traffic, and teams discover the difference when a deployment that worked in testing runs out of memory the moment concurrent users arrive.

The calculation has two parts. Weights are straightforward and fixed: parameter count multiplied by bytes per parameter, where the multiplier depends on quantization — two bytes for half precision, one for int8, half a byte for int4. This is why quantization is such a powerful lever, turning a model that needs multiple cards into one that fits on a single card. The second part is the KV cache, and it is where deployments fail. Every token in every active request stores key and value tensors across every layer, so the cache scales with layer count, hidden size, context length, and — critically — the number of requests in flight simultaneously. A cache that consumes a couple of gigabytes for one request at short context can consume more than the model weights themselves at long context with meaningful concurrency. Runtime overhead for activations, fragmentation, and the inference framework is added on top, and the total is divided by per-GPU VRAM to give the card count.

The planner also reports what fraction of total memory the KV cache represents and how many concurrent requests a single GPU could hold once weights and overhead are paid, because that is the number that governs real throughput. Two design consequences follow. First, context length is expensive in a way that is easy to underestimate: quadrupling the supported context roughly quadruples cache consumption per request, so advertising a very long context window has a direct and substantial hardware cost. Second, concurrency and context trade against each other on a fixed card — you can serve many short requests or a few long ones, and the serving framework's batching strategy determines which. Techniques like paged attention, grouped-query attention, and KV cache quantization materially change these numbers, so treat this as the first-order model and validate against your actual serving stack. The source research is right that a useful planner must add hardware availability, context and batch tradeoffs, quantization, and cost per throughput target rather than reporting a single fit-or-not answer.