Retrieval-augmented generation, explained properly

RAG is usually introduced as: search your documents, put the results in the prompt, generate an answer. That is correct and it conceals every decision that matters.

What the pipeline really is

Ingestion. Documents are parsed, split into chunks, embedded into vectors and indexed with metadata.

Query time. The question is embedded, similar chunks are retrieved, usually re-ranked, assembled into context, and passed to the model with instructions to answer from that context.

Five decisions inside that determine whether it works.

Decision 1: how you chunk

The default — fixed-size windows with overlap — is wrong for most enterprise documents, because enterprise documents have structure that carries meaning.

A contract clause is a unit. A policy section has a scope statement at the top that governs everything below it. A clinical note has headings that change how content should be read. Slicing through those produces chunks that say things the document does not.

Chunk on the document's own structure. Carry the section path as metadata, so a retrieved paragraph arrives labelled "Section 4.2 — Exclusions" rather than floating free.

Decision 2: what you retrieve against

Pure vector similarity is good at meaning and bad at exactness. It will happily miss a document that contains the exact policy number you asked for, because number strings embed poorly.

Hybrid retrieval — dense vectors plus keyword search, fused — is the practical default. Enterprise questions contain identifiers, and identifiers need lexical matching.

Decision 3: scope

This is the one that separates a demo from a production system.

Retrieval must be restricted to what this question is about and what this user may see: the resolved entity, the permitted documents, the current versions, the right jurisdiction.

Unscoped retrieval produces the failure that ends enterprise programmes — an answer that blends two customers, fluently. Scope is enforced in the query, not hoped for in the prompt.

Decision 4: re-ranking

Initial retrieval optimises for recall: get the candidates. Re-ranking optimises for precision: order them by actual relevance to this question, using a model that sees query and document together.

Retrieve twenty, re-rank, pass four. Passing twenty because the context window allows it costs money and measurably degrades answers — models attend worse across long, diluted contexts.

Decision 5: whether it can abstain

A retrieval system that always returns its top results will always produce an answer, including when the corpus contains nothing relevant.

Score retrieval quality independently of generation. If the best candidates are weak, say so and stop. Measure correct refusal as a first-class outcome alongside correct answer.

Where RAG is the wrong tool

RAG grounds an answer in retrieved text. It is poor at anything requiring computation, aggregation or exhaustive coverage.

"How many claims over 50,000 did we settle last quarter" is a query, not a retrieval. "Summarise every contract with an auto-renewal clause" needs exhaustive filtering, not top-k similarity.

Route those to a deterministic path. The most common enterprise RAG failure after unscoped retrieval is asking it to do arithmetic over a corpus by finding documents that mention numbers.

The honest summary

RAG works well when the question is answerable from a small number of specific documents, the corpus is governed, retrieval is scoped, and the system is allowed to decline.

It works badly when any of those are missing — and three of the four are properties of your data layer, not your model.