Explained · Hardware & Power
The memory wall
Processors got much faster at arithmetic than memory got at delivering numbers to work on, so modern accelerators spend most of their time waiting rather than computing. Almost every optimisation in AI serving is an attempt to move fewer bytes, not to do less maths.
Where it breaksIt makes intuition about performance wrong. Buying a chip with more arithmetic performance changes nothing if the workload is starved for memory bandwidth, and a model that looks twice as expensive by parameter count can run at nearly the same speed. Judge serving hardware on bytes per second and capacity before teraflops.
~300 FLOPs per byte ratio of BF16 arithmetic (989 TFLOPS) to memory bandwidth (3.35 TB/s) on an H100 SXMNVIDIA H100 Tensor Core GPU datasheet · 2023-03-21
The gap that opened
For decades, arithmetic got cheaper and faster much more quickly than memory got wider. Both improved; they improved at different rates, and compounding did the rest. The result on a current accelerator is stark: the chip can perform a few hundred arithmetic operations in the time it takes to read a single byte from its own attached memory. So the useful question about any workload is not how much maths it requires but how many operations it performs per byte it must fetch — its arithmetic intensity. If that ratio is above the chip's crossover point, the arithmetic units stay busy and you are limited by compute. If it is below, the units idle while data arrives, and the chip runs at a fraction of the performance printed on the box no matter how the code is written.
Why token generation sits on the wrong side
Writing one token of a reply means every weight in the model is read from memory and used for roughly one multiply-accumulate. That is about two operations per byte read — two, against a chip that could manage hundreds. Single-stream generation is therefore almost entirely a memory-bandwidth problem, and its speed can be estimated with arithmetic a reader can do in their head: divide the memory bandwidth by the bytes of model weights and you get roughly the tokens per second. That single calculation explains why a smaller model is faster in near-exact proportion to its size, why quantization speeds things up despite doing the same number of multiplications, and why two chips with very different headline compute figures generate at similar speeds when their bandwidth is similar.
Where it breaks — and how the industry works around it
The escape is to reuse each fetched byte more times. Batching does exactly this: fetch the weights once and apply them to twenty users' tokens, raising arithmetic intensity twentyfold and moving the workload back towards being compute-bound. That is the real reason batched serving is cheap and single-user serving is not, and it is why the same model costs a provider far less per token than it costs you to run alone at home. The workarounds have limits, though. Batching needs memory for every user's cache, so the wall reappears as a capacity limit; and reading the prompt is compute-bound while writing the reply is memory-bound, which is why serving systems increasingly split those two phases onto different machines rather than making one machine bad at both.
What to do with this
Use it as a filter on claims. When a chip is advertised on peak arithmetic performance, ask for the memory bandwidth and the capacity, because for inference those two decide throughput and how many users fit. When someone reports a speedup, ask whether it moved fewer bytes or did less maths — the first is usually real, the second usually costs quality. And when sizing your own hardware, compute the crude bandwidth-over-weights figure before buying anything; it is accurate enough to rule out most bad purchases, and it will tell you immediately whether the money should go on a faster chip or simply on more memory.