The price of trusting AI agents

The better AI gets, the more we trust it and the less we check it. Every check we skip hands AI a little more control. How much of it are we willing to give up?

With AI you can run several things at the same time. You start one task, hand it to an agent, start the next and come back to check the results. So attention gets scattered. There are many more decisions to make, and there is a pull towards skimming each one. The work looks done, so the easy move is to approve it and move on to the next.

Then again, there is a theory that quick decisions are often the better ones. Chess offers some support. In professional games, faster moves were on average the better ones, because a strong player simply sees the move.¹

The question is whether the person checking an agent’s work sees the problem that fast. Time pressure points the other way. Under it, specialists checking AI advice leaned on it more.²

Less control, more trust

More trust lets us get more done, but the mistakes can get bigger.

There are already AI tools that take over your computer. You switch one on, tell it what needs doing and go shopping. It clicks around, sends emails and does whatever the task needs. Most of the time it works. Sometimes it gets it wrong, and the mistake can be very expensive. One agent wiped a company’s live data, backups included, in about nine seconds.³ Its access had no limits.

Mistakes like that are the price of trust, and we have to decide how high a price we will pay.

Ordinary risk analysis

That decision is an old one. It is ordinary risk analysis. You walk through a process and mark the few things that have to stay with you. Mostly they are the things that rest on judgment, where a person has to look at the case themselves. The rest could go to the agent.

Judgment is hard to hand over because of how language works. Language models were supposed to talk to us in natural language. In practice they seem to do better with code, where everything has to be spelled out.

A sentence is a fairly precise record of a thought, but it lacks what its author assumes the reader knows from context. That part stays in the author’s head. Give AI the text alone, and it has to guess the rest. So the AI will make mistakes. One way to limit them is the risk analysis above, with critical processes kept away from AI.

More context, longer instructions

The other way to limit them is to give the agent more context. The better the notes on past work, the better the agent understands what the work is about and what it should do. The agent can also keep its own notes across the whole collaboration, on the problems it hit and the way the team works. In day-to-day work, quality improves as it collects all that. In studies, the gain is smaller than it feels.

Notes written by people raised coding agents’ success by about 4%, and notes the agents wrote for themselves lowered it.⁴

The notes also keep growing, and soon the instructions run to dozens of pages

Research suggests that length has a price of its own. When instructions pile up, models follow fewer of them, because the instructions start to clash with each other.⁵ When an agent’s memory of a long conversation fills up, it gets trimmed, so things get lost. One agent, told to confirm before deleting emails, lost exactly that instruction and deleted the emails anyway.⁶

Weeks of work with one agent show the gain and the risk together. It understands more and more, starts pointing things out and corrects itself. Then it makes a silly mistake that sets the project back. Sometimes it slips in a change of its own, with a note along the lines of “by the way, I changed that because it made no sense.” Everything it writes has to be read, carefully.

So the two ways work together. Context makes the agent better at the work that can be written down. Judgment stays with people wherever a mistake costs too much. Where that line runs is different in each process, and it is likely to move as the models get better.