Explained · Infrastructure
Inference
Training builds the model; inference is using it. Every time you ask a chatbot a question, a data centre runs your words through the model and generates the reply, one word-piece at a time. Training happens once — inference happens billions of times a day, and every run costs money.
What actually happens when you hit send
Your message is chopped into tokens — word-pieces, roughly three-quarters of a word each — and turned into lists of numbers. The model reads the whole prompt in one parallel pass, then writes its reply one token at a time: billions of multiplications produce a score for every token it could say next, one is picked, bolted onto the text, and the whole run happens again for the token after that. That loop is why replies stream in word by word — the model genuinely does not know the end of its own sentence until it gets there.
Why it costs what it costs
Producing each token of the reply means streaming the model's billions of weights through a chip's memory, and while a GPU is doing that for you it is not doing it for anyone else. Reading your prompt is comparatively cheap, because the whole thing can be crunched in one parallel pass; writing the reply is the expensive part, because each new token has to wait for the one before it. That is why providers meter usage in tokens, and why output tokens typically cost several times more than input tokens. Notice the business model hiding in the mechanics: training is paid once, like building a factory, but inference is paid on every use — so the more successful an AI product becomes, the bigger its compute bill grows.
Where it breaks
The failure modes fall straight out of the mechanism. Long answers are slow because generation is one-token-at-a-time — token twenty cannot start until token nineteen is finished, which is why a two-paragraph reply takes visibly longer than a two-word one. Long conversations get expensive because the chip has to keep the whole exchange in fast memory, and that footprint grows with every turn. And the model cannot look anything up mid-answer: at inference time it knows only what training baked into its weights plus whatever is in your prompt, which is why it can be confidently wrong about anything recent — and why techniques like RAG exist to hand it the missing pages.
The economics in one line
Training is a capital cost; inference is the cost of goods sold. A model is built once but used billions of times, so nearly every serious efficiency trick in AI — quantization to shrink the model, routing to keep easy questions off expensive models, caching to avoid recomputing what was already computed — is an attack on the inference bill. When a lab raises enormous sums for compute, some of it buys smarter models; a great deal of it simply keeps the answers coming.