Explained · Infrastructure
KV cache
While writing a reply, the model keeps a running set of intermediate numbers for every token so far, so each new token does not require re-reading the whole conversation. That store is the KV cache. It is what makes generation fast, and it is also what fills up the GPU.
Where it breaksThe cache grows with every token and lives in the accelerator's fast memory, so on long conversations it can occupy more space than the model itself. When it does not fit, the server either evicts and recomputes — a visible stall — or refuses new users, which is why concurrency on a long-context product collapses long before the chips run out of arithmetic.
up to 24× throughput gain vLLM reports over HuggingFace Transformers by managing KV cache memory in pagesKwon et al., "Efficient Memory Management for LLM Serving with PagedAttention" (SOSP 2023) · 2023-09-12
What the cache holds
To produce the next token, the model has each position in the text look back at every earlier position and decide how much attention to pay it. Doing that requires two derived vectors per earlier token — a key and a value — computed from the model's weights. Those do not change once a token is fixed: the key and value for the tenth word are the same when writing the eleventh word as when writing the five-hundredth. So the server computes them once and keeps them. That store is the KV cache. Without it, every new token would mean recomputing the entire conversation from the beginning, and generation would slow down quadratically as the reply grew. With it, each new token costs roughly the same as the last. It is the single optimisation that makes conversational AI feel like typing rather than like waiting.
Why it decides how many users fit
The cache lives in the accelerator's own high-bandwidth memory, alongside the model weights, because it is read on every single token. Its size is the product of four things: how many tokens are in the conversation, how many layers the model has, how wide each layer is, and how many bytes each number takes. That product grows linearly with conversation length and it grows per user — every concurrent session has its own. The consequence is the central fact of serving economics. A card holding, say, eighty gigabytes spends most of it on weights, and what is left divided by the per-user cache size is the number of people that card can serve at once. Long contexts shrink that number directly, which is why long-context features are priced the way they are and why architectures that shrink the cache are pursued so hard.
Where it breaks
Classic implementations reserved a contiguous block of memory per request sized for the longest reply it might produce, and most replies are far shorter, so a large share of the most expensive memory in the building sat reserved and unused. Paged management fixed that by allocating the cache in small blocks on demand, the way an operating system handles virtual memory, and the throughput gains were dramatic — which tells you how much was being wasted. The remaining failure is eviction. When memory runs out mid-conversation the server must drop someone's cache and recompute it later, and that recomputation is the stall users experience as a request that suddenly takes ten seconds to start. Under load, a serving system does not degrade smoothly; it degrades at the moment the cache stops fitting.
What follows from it
Almost every serving trick you will hear named is an attack on this cache. Grouped-query attention has several attention heads share one set of keys and values, cutting the cache by a large factor for a small quality cost. Quantising the cache to eight bits halves it again. Prefix caching keeps the shared preamble of many requests in memory once instead of per user. Splitting prefill from decode onto different machines exists because reading a prompt and writing a reply stress memory in opposite ways. If you are choosing between serving stacks or trying to understand why your GPU bill is what it is, the useful question is not how fast the chip computes — it is how much cache each of your users needs, and how many of them therefore fit.