AI cheat sheet

Fine-tuning Fine-tuning takes a trained model and gives it a short second education on your own examples — your tone, your formats, your field — so the behaviour sticks without teaching anything from scratch. Think of hiring a brilliant generalist, then spending a week training them on your house style.

10,000× fewer trainable parameters · reduction LoRA reports when adapting GPT-3 175B instead of retraining every weight

Context windows The context window is the model's working memory — how much text it can hold in mind at once, counting your question, the conversation so far, and anything you pasted in. When it fills up, the oldest material falls out, which is why very long chats start to drift. Bigger windows exist; they cost more to run.

128,000 tokens · context window of GPT-4o, roughly a 300-page book

Quantization A model's knowledge is stored as billions of numbers. Quantization rounds those numbers to a coarser precision — like compressing a photo — so the model takes far less memory and runs faster and cheaper. Done carefully, the quality loss is barely measurable, which is why almost every model you use is quantized.

4× smaller · memory cut from storing weights at 4 bits instead of 16

Tokens Models do not read letters or words. Text is first cut into tokens — common word-pieces from a fixed vocabulary — and every token is billed, counted against the context window, and produced one at a time. A rough rule for English: a token is about four characters, or three-quarters of a word.

~4 characters · per token, for ordinary English text

Transformers and attention The transformer is the design nearly every modern AI model uses. Its key move is attention: while processing each word, the model looks at every other word at once and weighs which ones matter. That parallel look is what made models trainable on enormous amounts of text.

8 attention heads · parallel attention comparisons per layer in the original transformer

Mixture of experts (MoE) Instead of one dense model where every part works on every word, a mixture of experts holds many specialised sub-networks and uses a small router to wake only a couple of them per token. The model is enormous on paper and cheap per answer, because most of it stays asleep.

46.7B total, 12.9B active per token · parameter split in Mixtral 8x7B

Distillation Distillation trains a small model to copy a large one. The big model answers a pile of questions, and the small model learns to reproduce those answers, ending up far cheaper to run while keeping most of the ability on the work you care about.

40% smaller, 97% of performance · DistilBERT's size cut and retained GLUE score against BERT

Hallucination A model that does not know something does not stop — it produces the most plausible continuation, which can be a fluent, well-formatted, entirely invented fact. It is not lying or malfunctioning; generating likely text is the only thing it does, and truth is not a separate check inside it.

58% truthful · best-performing model's score on TruthfulQA against 94% for humans

RLHF (learning from human feedback) A raw model predicts likely text, which is not the same as being helpful. RLHF shows people pairs of answers, records which they prefer, trains a scoring model on those judgements, and then tunes the model to score well. It is the step that turns a text predictor into an assistant.

1.3B beat 175B · labelers preferred the small RLHF-tuned InstructGPT over the 100× larger base GPT-3

Inference Training builds the model; inference is using it. Every time you ask a chatbot a question, a data centre runs your words through the model and generates the reply, one word-piece at a time. Training happens once — inference happens billions of times a day, and every run costs money.

$0.15 / $0.60 per million tokens · listed input / output price of GPT-4o mini at launch

KV cache While writing a reply, the model keeps a running set of intermediate numbers for every token so far, so each new token does not require re-reading the whole conversation. That store is the KV cache. It is what makes generation fast, and it is also what fills up the GPU.

up to 24× throughput · gain vLLM reports over HuggingFace Transformers by managing KV cache memory in pages

Latency vs throughput Latency is how long one person waits; throughput is how many people the machine serves per second. They pull against each other — the main way to raise throughput is to make each request wait a little so it can be processed alongside others.

TTFT 2,000 ms, per-token 200 ms · latency constraints MLPerf Inference imposes on the Llama 2 70B server scenario

Batching Running one request through a GPU wastes most of the chip, because the weights have to be fetched either way. Batching processes many requests together against the same fetched weights, so serving twenty people costs barely more than serving one — which is where the margin in inference comes from.

up to 36.9× throughput · gain from continuous batching at the same latency, versus fixed-batch serving

Speculative decoding A small fast model guesses the next few tokens, and the big model checks them all in one pass. Guesses it agrees with are kept, the first disagreement is corrected. The output is identical to running the big model alone — it just arrives sooner.

2–3× faster · speedup reported with identical output distribution

GPUs vs TPUs A GPU is a general-purpose parallel processor that happens to be excellent at AI; a TPU is a chip built for one job and nothing else. Custom silicon can be cheaper per unit of work, and it locks you to one vendor's software and one company's supply.

80 GB at 3.35 TB/s · memory capacity and bandwidth of the NVIDIA H100 SXM

HBM (high-bandwidth memory) HBM is memory stacked vertically and mounted right next to the processor, giving it a far wider road to the chip than ordinary memory. It is what lets an accelerator feed itself fast enough to be useful, and it is one of the scarcest, most expensive parts of an AI machine.

141 GB at 4.8 TB/s · HBM3E capacity and bandwidth on the NVIDIA H200

The memory wall Processors got much faster at arithmetic than memory got at delivering numbers to work on, so modern accelerators spend most of their time waiting rather than computing. Almost every optimisation in AI serving is an attempt to move fewer bytes, not to do less maths.

~300 FLOPs per byte · ratio of BF16 arithmetic (989 TFLOPS) to memory bandwidth (3.35 TB/s) on an H100 SXM

Data centre power AI capacity is limited less by chips than by electricity and the ability to connect to a grid. A modern AI hall draws as much power as a small town, and the wait for a grid connection is now measured in years, which is why builders chase places with spare power.

460 TWh in 2022, 620–1,050 TWh by 2026 · global data centre electricity consumption and forecast range

PUE and cooling PUE is the ratio of everything a data centre draws to the part that reaches the computers; 1.5 means half again is spent on cooling and losses. AI racks run so hot that air cooling is running out of room, which is pushing the industry to plumb liquid directly to the chips.

1.10 · Google fleet-wide trailing twelve-month power usage effectiveness

RAG (retrieval-augmented generation) Instead of hoping the model memorised the right facts, you hand it the right documents at question time. The system searches your files, pastes the best passages into the prompt, and the model answers from those. It is why a chatbot can cite your company wiki without being retrained on it.

21 million passages · size of the Wikipedia index the original retrieval systems searched per question

AI agents A chatbot answers you; an agent acts for you. Give it a goal and it runs a loop — pick a step, use a tool like search or code, check the result, adjust, repeat until the job is done. The bet is delegation: you review finished work instead of doing the work yourself.

500 tasks · size of SWE-bench Verified, the human-validated set of real GitHub issues agents are graded on

AI evaluations (evals) Evals are the test suite for an AI system: a fixed set of questions with known good answers, scored automatically every time something changes. Without them, you are guessing whether a new model or prompt actually made things better. Teams with sharp evals ship improvements; teams without them ship vibes.

57 subjects · breadth of MMLU, the benchmark most model launches still quote

Embeddings An embedding turns a piece of text into a long list of numbers that stands for its meaning, positioned so that passages about similar things end up near each other. Search then becomes geometry: find the nearest points. It is the machinery behind semantic search, recommendations and the retrieval half of RAG.

1,536 dimensions · default vector length of OpenAI text-embedding-3-small

Guardrails Guardrails are the checks placed around a model rather than inside it: filters on what goes in, validators on what comes out, and limits on what the system is allowed to do. They exist because you cannot make a model refuse reliably, but you can refuse on its behalf.

free · price of the OpenAI moderation endpoint, the standard input filter

Prompt injection A model cannot tell your instructions apart from the text it is reading — both arrive as the same stream of words. So a web page, email or document can carry instructions of its own, and a system that reads it may follow them. It is the defining security problem of AI that touches the outside world.

ranked LLM01 · prompt injection's position at number one in the OWASP Top 10 for LLM applications

MCP (Model Context Protocol) MCP is a common plug format between AI assistants and the tools and data they use. Write one server for your system and any assistant that speaks the protocol can use it, instead of every product building a bespoke connector for every service.

open-sourced 25 Nov 2024 · date Anthropic published the Model Context Protocol specification

Benchmarks Benchmarks are the standard exams models are graded on, and they are how launches get compared. They are useful for tracking the field over years and close to useless for choosing a model for your particular job, which your own test cases decide.

34% · accuracy of skilled non-experts with unrestricted web access on GPQA questions, against 65% for domain PhDs

Model routing Not every question needs the biggest, most expensive model. A router reads each request and sends easy ones to a small, cheap model and hard ones to the heavyweight. Users rarely notice the difference — the bill notices, which is why most serious AI products quietly do this.

~17× cheaper input · gap between GPT-4o at $2.50/M and GPT-4o mini at $0.15/M input tokens

Prompt caching Most requests to an AI system start with the same long preamble — instructions, examples, a document. Prompt caching stores the model's half-finished reading of that fixed part so the next request skips re-reading it, cutting both the price and the wait before the first word appears.

10% of base input price · what Anthropic charges for a cache read against a written prompt cache

Open weights An open-weights model is one whose trained numbers you can download and run on your own hardware. That is not the same as open source — the training data and code usually stay private, and the licence often carries conditions — but it does mean nobody can take the model away from you.

700 million monthly users · threshold above which the Llama 3.1 licence requires a separate agreement