Entity resolution is the whole game

Summary

Enterprise AI programmes stall in a place the organisation rarely expects. It is seldom the model, seldom the infrastructure, and seldom the use case.

It is the point where the system has to answer a question about an entity that exists differently in four systems, and cannot.

This paper is about that layer: what a governed record is, why survivorship rules are a business decision rather than a technical one, why quarantine is a feature, and why lineage has to be at record level to be worth anything.

The shape of the failure

A typical pilot works. It works because it was scoped to one system, usually the one with the cleanest data, and because a human picked the documents.

The move to production is where the estate arrives. Now the customer is in the CRM with one identifier, in the billing platform with another, in the document store as a name on a PDF, and in the support system as an email address that belongs to their assistant.

The retrieval layer does what it is designed to do: it returns the most similar content. Similar is not the same as about the same entity. The system produces an answer that blends two customers, and it produces it fluently, and somebody notices in a meeting.

At that point the programme has a credibility problem that is much more expensive than the technical one underneath it.

What a governed record requires

Resolution, not deduplication

Deduplication removes copies of the same row. Resolution decides that four different rows, with different identifiers, different spellings and different completeness, refer to one real-world entity — and produces a single record with a durable identity that survives future arrivals.

The difference matters because the entity is the unit everything downstream reasons about. Retrieval scoped to an entity is a fundamentally different operation from retrieval that hopes similarity implies identity.

Survivorship rules owned by the business

When two sources disagree about an attribute, something must decide which wins. That decision is not technical.

Which system is authoritative for a customer's legal name? For their address? For their risk classification? The answers differ by attribute, by jurisdiction, and sometimes by product line, and they are the business's answers to give.

Letting engineering pick survivorship rules — usually "most recently updated wins" — produces records that are internally consistent and wrong in ways nobody detects for months.

A quality bar with real rejects

A record that cannot be resolved with adequate confidence must not enter the decision path. It goes to quarantine: held, flagged, and routed for remediation.

Organisations resist this. Any visible quarantine rate reads as a defect in the pipeline. It is the opposite: a pipeline with no rejects is one that has silently decided to guess, and the guesses do not announce themselves.

Quarantine needs three things to work: a remediation path with an owner, an expiry policy for records nobody will ever fix, and visible metrics — because an unwatched quarantine becomes a landfill within a quarter.

Lineage at record level

"This answer came from the customer corpus" is not lineage. It is a citation of a filing cabinet.

Useful lineage identifies the specific records that contributed to a specific output, their source systems, their versions at the time of use, and the survivorship decisions applied. It has to be captured at write time — reconstructing it later is not possible once the sources have moved on.

The test is a regulator, an auditor or an angry customer asking why a particular decision came out the way it did, eighteen months after the fact, about a record that has since changed four times.

Why this layer is undervalued

It does not demo. Nobody has ever been promoted for a survivorship rule.

The demo is the agent answering a question in natural language. The reason the agent can answer correctly at estate scale is four months of unglamorous work underneath it, and that work is invisible precisely when it is done well.

The sequencing consequence is real. Programmes that start with the model and reach the data layer in year two spend year one producing outputs they cannot defend, and then have to explain why the numbers changed when the foundation was fixed.

A practical order of work

1. Pick the entity that matters most. Customer, supplier, asset, patient, policy. One. The instinct to resolve everything at once is how this becomes a three-year programme with no output.
2. Enumerate its systems of record honestly. Including the shared inbox and the spreadsheet. Especially those — they exist because a real gap exists.
3. Get survivorship rules written down and signed by an owner. Per attribute. This will surface disagreements that predate the AI programme by a decade.
4. Set the quality bar, and instrument quarantine before go-live. With an owner and an expiry policy.
5. Capture lineage from the first record. Retrofitting is not possible.
6. Only then build on top of it.

Every step here is deferrable and every deferral compounds. The order matters more than the pace: the work deferred here is the work that determines whether anything above it can be defended.