Why we require an evidence quorum rather than a confidence score
Self-reported confidence is a property of the model's output distribution. It cannot encode what was never retrieved.
The choice
Two ways to decide whether an automated system may act without a human.
Single-lane confidence. One reasoning model over assembled context, with an automation threshold on its self-reported confidence.
Evidence quorum. Retrieval grounding, deterministic constraint checks and reasoning, each producing an independent assessment. Automation requires agreement; disagreement routes to a human.
The platform defaults to the second for consequential decisions. This note is why, and what would have to be true for that to be wrong.
Why single-lane confidence fails the way it does
Self-reported confidence is a property of the model's output distribution, not of the evidence. A model given a coherent but incomplete context will produce a confident, coherent, incomplete answer. Nothing in the confidence signal encodes what was not retrieved.
That is not a calibration problem to be tuned away. It is structural: the signal is computed over the tokens the model produced, and the missing evidence never became tokens.
The mechanism quorum relies on
Quorum works — if it works — because the lanes fail differently:
- Retrieval failure shows up as weak grounding scores.
- A constraint violation is deterministic and does not care how fluent the reasoning is.
- Reasoning failure shows up as disagreement with the other two.
A confident error then requires all three to fail simultaneously. The whole bet rests on those failure modes being close to independent, and that is the assumption most worth attacking. If retrieval and reasoning share an upstream cause — the same stale corpus, say — they are correlated and the quorum buys less than it appears to.
What we expect, and what would falsify it
We expect quorum to automate slightly less and escalate more, in exchange for a large reduction in the errors that get made autonomously and are caught late.
That trade only looks good if the second effect dominates. The measurement that decides it, which any estate can run in shadow mode:
| Outcome class | What it costs |
|---|---|
| Correct, automated | The win |
| Correct, escalated | Cheap — a human's time |
| Incorrect, escalated | Cheap — the human catches it |
| Incorrect, automated | Expensive, and found late |
Run both configurations on the same traffic. If the reduction in the fourth row does not outweigh the loss in the first row on the estate's own cost weighting, single-lane is the right answer there and we would say so.
The unglamorous part
Our expectation is that the deterministic lane does most of the work — limits, entitlements, coverage, eligibility, hours-of-service. Cheap, uninteresting checks that are not learned and not negotiable.
That is not a satisfying claim for an AI platform to make, which is one reason to state it plainly. The ablation that would test it: remove each lane in turn and watch the confident-error rate. If removing the deterministic checks moves it most, the boring lane is the load-bearing one.
What this means in practice
Quorum is the default for consequential decisions and single-lane operation is available where the decision is genuinely low-consequence and the latency budget is tight.
Expect the escalation rate to rise when you switch. Two teams we have talked this through with assumed that was a regression to be tuned away. It is the mechanism working — and the population that lands in the escalation queue is the thing to go and read before deciding otherwise.
- Decisioning
- Evaluation
- Confidence