Explained · Engineering
Embeddings
An embedding turns a piece of text into a long list of numbers that stands for its meaning, positioned so that passages about similar things end up near each other. Search then becomes geometry: find the nearest points. It is the machinery behind semantic search, recommendations and the retrieval half of RAG.
Where it breaks"Near" means similar-sounding, not correct or relevant. A policy and its exact opposite sit close together because they discuss the same subject, so retrieval happily returns the wrong one. Embeddings also carry no sense of time or authority, so last year's superseded document scores exactly as well as this week's replacement.
1,536 dimensions default vector length of OpenAI text-embedding-3-smallOpenAI, "New embedding models and API updates" · 2024-01-25
Meaning as a location
Feed a sentence to an embedding model and you get back a list of numbers — commonly a few hundred to a few thousand of them. Treat that list as coordinates and every piece of text becomes a point in a space with that many directions. The model was trained so that texts people treat as related land close together and unrelated ones land far apart, which means the geometry carries meaning: "how do I reset my password" sits near "I am locked out of my account" even though they share almost no words. Distance is usually measured as the angle between two points rather than the straight-line gap, because what matters is direction — what the text is about — rather than magnitude. That is the whole idea. Everything else is engineering built on top of it.
What it replaces
Traditional search matches words. It is fast, exact and completely defeated by synonyms, paraphrase and jargon — the user asks about "termination notice" and your document says "ending the agreement", and keyword search returns nothing. Embedding search matches meaning, so paraphrase stops mattering. The cost is that you must convert every document in advance, store the resulting vectors, and search them with an index designed for nearest-neighbour lookup rather than for exact matches. The conversion is cheap per document and permanent until the text changes; the index is the new infrastructure you take on. In practice serious systems run both and merge the results, because keyword search remains unbeatable at the thing embeddings are worst at: finding an exact product code, error string or surname.
Where it breaks
Similar is not the same as relevant, and this catches everyone. A document stating a rule and a document repealing it sit almost on top of each other, because they discuss identical subject matter — so a question about current policy can retrieve the superseded version with a high similarity score. Embeddings encode no notion of recency, authority or correctness at all; those have to be added as separate filters over the results. The second problem is chunking: you embed pieces, not documents, and a piece that begins "This does not apply to contractors" is meaningless without the paragraph above it. The third is drift — change your embedding model and every stored vector becomes incomparable with the new ones, so an upgrade means re-converting the entire corpus. That is a migration, not a config change.
How to tell if yours is working
Judge the retrieval, not the answer. Take fifty real questions, write down which passage should have been returned, and measure how often it appears in the top few. That single number tells you more than any amount of prompt tuning downstream, because a generator handed the right paragraph rarely fails and a generator handed the wrong one always does. When the number is poor, the fixes are ordinary: chunk on structure rather than character count, prepend the document title to each chunk so context is not lost, retrieve widely and then re-rank with a slower and more accurate model, and blend keyword results in for exact identifiers. Almost every disappointing RAG system is a retrieval problem wearing a model problem's clothes.