Explained · Hardware & Power
HBM (high-bandwidth memory)
HBM is memory stacked vertically and mounted right next to the processor, giving it a far wider road to the chip than ordinary memory. It is what lets an accelerator feed itself fast enough to be useful, and it is one of the scarcest, most expensive parts of an AI machine.
Where it breaksIt is the supply bottleneck and the capacity ceiling at the same time. Only a few firms can make it, packaging capacity is booked far ahead, and once a model plus its caches exceed what is on the card you are splitting the job across chips and paying in network hops.
141 GB at 4.8 TB/s HBM3E capacity and bandwidth on the NVIDIA H200NVIDIA H200 Tensor Core GPU datasheet · 2023-11-13
Why memory had to be rebuilt
Ordinary computer memory sits in slots some centimetres from the processor and talks to it over a relatively narrow set of wires. That was fine when chips were slow enough to keep up. AI accelerators are not: they can perform arithmetic far faster than any conventional memory can deliver numbers to work on, so the memory road, not the engine, sets the speed. HBM is the response. Instead of laying chips out flat and far away, the memory dies are stacked in a tower, connected vertically by thousands of tiny holes drilled straight through the silicon, and the whole stack is mounted on the same package as the processor, millimetres away. That geometry buys an enormously wide road — thousands of wires instead of dozens — at a lower clock speed, which is also why it is more energy-efficient per byte moved.
Why it dominates the cost of an AI chip
Two things about HBM make it expensive out of proportion to its size. It has to be manufactured almost perfectly, because a stack is only good if every die in it is good — one bad layer discards the whole tower, so yields are harder than for flat memory. And it cannot simply be soldered next to the processor: it must be attached with advanced packaging, on a silicon interposer that carries the thousands of connections, and the world's capacity to do that packaging is limited and booked far in advance. The result is that a single component few consumers have heard of is a substantial share of an accelerator's bill of materials, that the memory makers capable of supplying it can be counted on one hand, and that the queue for packaging, rather than for transistors, is often what decides how many accelerators exist in a given quarter.
Where it breaks
Capacity is the wall people hit first. An accelerator has a fixed amount of HBM, and it must hold the model weights plus every concurrent user's KV cache. Exceed it and the job has to be split across several chips, which means every token now involves traffic between chips — fast interconnect, but far slower than local memory — so cost per token rises and latency becomes uneven. Capacity is also why quantization matters so much: halving the bytes per weight can be the difference between one chip and four. Meanwhile bandwidth caps the speed of generation itself, because producing each token requires streaming the weights through the processor once. That is the number to look at when comparing accelerators for serving, and it is a very different number from headline arithmetic performance.
What it means for the money
HBM is where several separate stories converge. It explains why accelerator supply moves on packaging and memory schedules rather than on logic fabrication; why memory manufacturers became AI stocks; and why serving costs track gigabytes rather than raw compute. For anyone buying capacity, the practical rule is to size by memory first — model weights plus expected concurrent context — and treat arithmetic throughput as the second question, because a chip you cannot fit the workload into is not a cheaper chip. For anyone reading the industry, watch memory capacity per accelerator generation and packaging capacity announcements; they forecast what inference will cost more reliably than model releases do.