Watching the corpus instead of waiting for accuracy to move

The problem with accuracy monitoring

Accuracy monitoring is retrospective by construction. By the time a metric moves enough to trigger an alert, the model has been making worse decisions for as long as it took the signal to accumulate.

For a claims model where outcomes settle over ninety days, accuracy-based alerting can be a quarter behind reality. That is not a tuning problem. Ground truth simply does not exist yet.

So the question is whether anything available earlier carries signal.

The candidates, and why each might work

Quarantine reason mix. Not the rate — the rate is usually stable and uninformative — but the distribution of reasons. The mechanism: an upstream system changing its behaviour shows up first as a different mix of why records fail the quality bar, before any of it reaches a model. This is the signal we weight most heavily, and it is the one with the clearest causal story.

Retrieval abstention rate. Rising abstention means the corpus no longer covers what is being asked. It should precede accuracy loss because a system's first response to a coverage gap is to decline, and only later to answer badly from weak evidence.

Vocabulary novelty. The proportion of terms absent from the training corpus. Should work in domains with real terminology churn — new products, new regulations — and be flat and noisy in stable ones. Expect it to be useless more often than not.

Composition shift. Distribution over source systems, document types and entity classes. Genuinely hard to separate from seasonality without a long baseline, so expect false alarms.

Entity churn. The rate of new entities and re-resolutions. We expect this to be weak as a leading indicator and excellent for diagnosis after the fact.

What we do not know

We have not established how much lead time these actually buy. That requires a history of degradation events with corpus signals reconstructed alongside them, and it has to be done per estate — the answer will differ by domain, by how fast the upstream systems change, and by how long ground truth takes to arrive.

Any estate running this platform can produce that history, and it is worth doing deliberately rather than discovering after an incident.

How to validate it on your own estate

1. Record all five signals weekly from go-live. They are cheap and you cannot reconstruct them later.
2. Log every degradation event with a date, including the ones caught by complaint rather than by monitoring.
3. After a handful of events, look backwards: which signals moved first, and by how long.
4. Weight your alerting on what actually led, not on this list.

What we ship, and why it is conservative

The platform tracks all five, weighted toward quarantine reason mix and abstention, and routes them as a review trigger rather than an incident.

The recommended response is deliberately boring: run the evaluation suite. If it clears, log the signal and continue. If it does not, you have found a degradation before the accuracy metric would have shown it.

We do not recommend automated retraining on these signals, and we do not ship it. Composition shift and vocabulary novelty are noisy enough that automated intervention would fire on seasonality — and a retrain triggered by noise is a worse outcome than the drift it was chasing.