There is a version of the same meeting that happens in every organisation trying this.
The pilot demoed beautifully in March. It is July, it is not in production, and the room is discussing whether to try a different model. It is almost never the model.
Here is what tends to have happened, in the order it usually happens.
The pilot had a curated corpus
Someone picked the documents. Possibly they did not think of it as picking — they pointed at a SharePoint site that happened to be well-maintained, or exported the sixty PDFs the team actually uses.
That corpus was internally consistent, current, and about one thing. Production is none of those. Production has the 2019 version of the policy sitting next to the 2024 version with no metadata distinguishing them, and retrieval has no reason to prefer either.
Fix: version and date every document at ingest, and make recency a retrieval signal rather than a hope. Then go find the superseded documents and mark them superseded. This is boring and it is most of the win.
Retrieval had no entity scope
The pilot asked questions about one customer at a time and a human typed the customer's name. In production the system has to work out which customer, from an identifier that exists differently in four systems.
Vector search returns similar content. Similar is not "about the same entity". The failure mode is an answer that blends two customers, and it is fluent, and it is the failure that ends programmes because it is the one an executive notices personally.
Fix: resolve entities before you retrieve. Scope retrieval to the resolved entity. This is the single highest-leverage change available and it is a data project, not a model project.
Chunking ignored the document's structure
Fixed-size chunking with overlap is the default in every tutorial and it is wrong for almost every enterprise document.
Contracts have clauses. Policies have sections with scope statements at the top. Clinical notes have headings that determine how the content should be read. A 512-token window sliced through the middle of a clause produces a chunk that says something the clause does not.
Fix: chunk on the document's own structure, and carry the section path as metadata so the model knows a retrieved paragraph came from "Section 4.2 — Exclusions" rather than from nowhere.
Nothing could say "I don't know"
The pilot was measured on whether answers were good. Nobody measured how often it answered when it should have declined.
A retrieval system that always returns its top five results will always produce an answer, including when the corpus contains nothing relevant. The model will then write something plausible from those five irrelevant chunks, because that is what it is for.
Fix: score retrieval quality independently and make an explicit abstain decision before generation. Then measure correct refusals as a first-class outcome. Systems that can decline get trusted; systems that always answer get audited.
Evaluation was three people saying it seemed good
The pilot's quality bar was vibes in a demo. That works for a demo and provides no way to tell whether a change helped.
Fix: a hundred real questions with graded rubrics beats ten thousand synthetic ones. Build it from things that have actually gone wrong.
None of these are model problems. All of them are visible in the first week of production and invisible in a pilot, which is why pilots that go well are weak evidence.
If your pilot is stalling, the useful question is not "which model should we try". It is "what did the pilot's setup do for us that production does not".
- RAG
- Data
- Production