AI models, distrusted by design

A company should rely on the controls around AI, as it does with its most powerful insiders. Microsoft’s CEO calls for limits, logs and human oversight placed beyond its reach.¹

The most trustworthy system is the one that needs to trust the model least.¹

On its own, AI offers little to check. Older software let anyone follow a result back to the line of code behind it. AI leaves no such path, while it works with sensitive data and acts for the company.¹

Established security controls

Companies have long given powerful actors access to important systems. Security practice refined the controls for them over decades: a known identity, privileges limited to the job, a log of activity, and boundaries that contain a mistake.¹

A model belongs under the same controls. It needs:

1. An identity of its own and only the rights its task requires.
2. A record of its meaningful actions that people can read.
3. Someone with authority who can pause it or shut it down.¹

The reason has nothing to do with bad intent: any capable actor with access to important systems can make mistakes or be compromised.¹

Those controls already live in the software a company runs: user profiles, access rights, activity logs and approval steps. An agent connected there can inherit them, with a login and limits of its own. IT teams could then manage its rights in the same admin panels they use for employee accounts.

Permissions held apart from the model

Security engineering has a rule for where such controls go. The mechanism checking permissions runs on each request and stays out of reach of the program it checks.² For AI, the rule puts access rights and logs in systems separate from the model.¹

Checks belong there as well.

Models can test and verify one another. Stacked in layers, they leave one opaque process watching the next.¹

A reliable audit runs on evidence the company keeps and a person can read. With it, a reviewer can reproduce how a result came about.¹

Over-permissioned models

Security practitioners now rank excessive agency third among the risks of AI applications, up from sixth.³ The risk grows from three excesses in a model:⁴

1. Too much functionality.
2. Too many permissions.
3. Too much autonomy.

Each of the three widens what a single wrong output can set in motion. Insider controls would cut exactly those excesses.

Questions before deployment

Responsibility for an agent’s actions stays with the firm that runs it, whatever its provider promises.¹ So before it touches company email or the ERP, whoever is in charge should be able to answer five questions:

1. Which systems can it reach?
2. Who reads the record of its actions?
3. Can someone stop it mid-task?
4. Has it been tested on failures and attacks, along with successful runs?
5. Who hears about it when something goes wrong?¹

Answers written down before launch become the agent’s job description.

Some AI arrives with nobody asking the questions: an agent a team builds for its own use, or company work done in a private chatbot account. This shadow AI stays beyond the firm’s controls until someone finds it.

Verified knowledge sources

Rights set what a model can do. The material it draws on shapes what it can say. A system can be built to answer only from a verified knowledge base the company owns. Each reply then points to a document a person can open. The reviewer checks the source and holds the model’s output up against it. Control covers the input as well as the action.

Grinding by the model, steering by people

With these controls in place, AI can run inside the tools, permissions and approval steps a company already has. The model grinds through the volume. People review, approve and steer, and the record shows who did what.