Governing agentic systems in regulated enterprises

Summary

The governance conversation about generative AI has been dominated by output risk: what the model says, whether it is biased, whether it made something up. That framing was adequate while models only produced text.

An agent with tool access is a different object. It changes state in systems of record. It moves money, alters entitlements, files tickets, cancels services and books capacity. The relevant discipline is not content moderation — it is the same discipline that governs any other automated actor in a regulated firm: scoped authority, segregation of duties, approval thresholds, evidence retention and the ability to stop it.

This paper sets out six controls. None of them are novel. What is novel is applying them to a component that is probabilistic at its core.

Control 1: Typed, scoped tool contracts

Every action an agent can take must be a declared tool with a typed contract, a permission scope and an owner. No general-purpose execution surfaces — no arbitrary shell, no unconstrained database access, no "call any internal API" adapter.

The test: someone in second line should be able to read the tool register and enumerate, exhaustively, everything the agent is capable of doing. If the answer is "it depends what it decides to do", the system is not governable.

Scopes should be per-tool, not per-agent. An agent that can read the billing platform and write to the ticketing system holds two different scopes, and they should be revocable independently.

Control 2: Policy gates written by the risk owner

For each tool, the conditions under which it may be invoked autonomously must be written by whoever owns the risk — not by the engineering team, and not inferred from a prompt.

In practice these are threshold rules and they should be boring: value bands, customer flags, jurisdiction, time of day, cumulative exposure within a window. Boring is the goal, because a gate that requires interpretation is a gate that will be interpreted differently under pressure.

Two failure patterns to avoid:

• Prompt-as-policy. Putting the rule in the system prompt makes compliance a function of model behaviour. Gates must be enforced outside the model, in the runtime, where they cannot be argued with.
• Confidence-as-policy. "Act when confidence exceeds 0.9" conflates two different questions. Confidence is an input to eligibility. Eligibility is a policy decision.

Control 3: Evidence quorum before consequential action

Consequential actions should require agreement across independent evidence lanes rather than a single model's judgement: retrieved evidence, deterministic constraint checks, and reasoning over both.

The deterministic lane is the one that carries the non-negotiables — limits, entitlements, hours-of-service, sanctions, coverage. These are not things to be learned. They are things to be checked, and an option that fails them should never be surfaced, let alone executed.

Control 4: A trace for every action, retained like a transaction record

Every agent action produces a trace containing: the trigger, the evidence retrieved with record-level lineage, the tools called with their arguments, the policy gate that applied, the model versions in play, the computed confidence per lane, and the outcome.

Retention should match the underlying business record, not the shorter retention that log infrastructure defaults to. If a payment record is held for seven years, the trace of the agent that authorised it should be too.

The practical test is a regulator asking for a sample of 200 decisions from eighteen months ago. If producing that pack is a project, the control is not in place.

Control 5: Evaluation gates before promotion, on the risk owner's suite

No agent version reaches production without clearing an evaluation suite owned by the risk function. The suite should be weighted toward the failure modes that matter to the firm rather than toward general capability benchmarks.

Two properties matter more than suite size:

• It must be able to fail a release. A suite that has never blocked a promotion is not a gate, it is a report.
• It must be versioned and owned outside engineering. Otherwise the bar moves quietly whenever it is inconvenient.

Control 6: A kill switch that has been used

Every agent needs a documented, tested mechanism to stop it — per-tool, per-agent and estate-wide — that does not require an engineer to deploy anything.

Test it on a schedule, in production, during business hours. An untested kill switch is an assumption.

Segregation of duties

The uncomfortable question in most implementations is who can change the policy gates. If the same team can write the agent and relax the threshold it operates under, the control is nominal.

Gate definitions should live under change control with the risk owner as approver, versioned separately from the agent code, with an audit trail on every change. This is standard practice for limits systems and it should be standard here.

Where human review actually belongs

The reflexive answer — a human approves everything — produces a queue, and queues degrade. Within six months the reviewer is approving in batches without reading, which is worse than no control because it manufactures evidence of oversight that did not happen.

Put humans where judgement changes the outcome: cases at the boundary of policy, cases with vulnerable parties, cases where the evidence lanes disagree, and a continuous random sample of the autonomous population, re-worked blind.

The blind sample is the control most often missing and the one that tells you whether any of the others are working.

Applying this

None of this requires a new regulatory framework. Firms already know how to govern automated actors with limited authority, because they have been doing it with trading systems, payment engines and workflow automation for decades.

The mistake is treating an agent as an AI problem and routing it to a model risk process designed for scorecards. It is an operational risk problem with a probabilistic component, and the existing operational controls apply with one addition: the evidence trail has to capture what the system considered, not only what it did.