Agent Security › Why now
Reliability isn't an argument against agent security. It's one of the reasons we need it. The real problem isn't superintelligence, it's giving unreliable intelligence real authority.
There's something slightly strange about the conversation around AI agent security. We talk about agents becoming increasingly autonomous. We worry about them finding unexpected routes through systems. We discuss agents communicating with other agents, operating computers, accessing business applications and taking actions without constant human supervision.
And yet anyone who has spent serious time actually running AI agents knows another side of the story. They fail. A lot. Not necessarily dramatically. Usually much more mundanely. A scheduled task doesn't run. An agent checks 17 things when it was supposed to check 20. A tool fails and the agent doesn't recover properly. An agent decides it's finished when it isn't. A task works perfectly for days and then inexplicably doesn't. An agent responsible for checking another agent's work misses the problem too, so you end up checking the checker.
Which raises an obvious question: if AI agents aren't even completely reliable yet, why are we already worrying about securing them? Because reliability isn't an argument against agent security. It's one of the reasons we need it.
We don't need to begin with an imaginary AI agent plotting against its owner. The much more immediate problem is simpler: we're beginning to give unreliable intelligence real authority. That combination matters.
An unreliable chatbot can give you a bad answer. Annoying. An unreliable agent with READ access can give you bad information. More serious. An unreliable agent with CHANGE authority can alter business systems. One with SEND authority can expose its mistakes to customers. One with SPEND authority can cost money. One with DELETE authority can create a very different kind of afternoon. The intelligence didn't necessarily become dramatically more dangerous. The authority changed.
We've tolerated unreliable AI for years. Ask a chatbot a question, it gets something wrong, you notice, you correct it, you try again. The human remains the final action layer. Agents change that relationship. Instead of "AI produces answer, human evaluates, human acts", we increasingly have "human defines goal, agent interprets goal, agent chooses actions, agent uses tools, business consequence occurs". That removes some of the natural checkpoints we've previously relied upon, which is exactly why authority matters.
The obvious failures are relatively easy. ERROR. TASK FAILED. API UNAVAILABLE. Great, at least you know. The more difficult failures are SUCCESS, when the job wasn't actually successful. Imagine an agent's task is to check 30 organisations for significant AI announcements. It checks 24. Six fail. It produces a perfectly good report based on the 24, then says "task completed successfully". Was it? The output may look fine, the writing excellent, the reasoning sensible. But 20% of the job never happened. That's an observability problem, and once the task matters enough, it becomes a control problem too.
An agent reporting SUCCESS is not proof that the intended outcome occurred. It's another data point. Don't ask the agent whether it succeeded. Verify the outcome.
If an agent says it backed up the database, check whether the backup exists. If it says it updated all 1,000 records, check the database. If it says it published the new version, check production. If it says it reviewed all 50 sources, check the coverage.
A natural response is "fine, we'll have another agent check the first one". Sometimes that's useful. Agent A performs the task, Agent B reviews Agent A, perhaps Agent C monitors both. But there's an awkward problem: Agent B is an AI agent too. It can miss something, misunderstand the requirement, accept Agent A's explanation, check the wrong evidence, or confidently conclude that everything is fine. So Agent A can fail, Agent B can fail to detect Agent A's failure, and your monitoring system can still display SUCCESS. Adding agents doesn't automatically remove uncertainty. Sometimes it simply creates more layers of uncertainty.
This is where the promise of agent automation can start becoming frustrating. You create agents to reduce work. Then you create systems to monitor the agents. Then checks for the systems. Then alerts. Then dashboards. Then another agent to summarise the alerts. Then you periodically check whether the monitoring agent is still running. Eventually you realise the oversight has become a problem of its own.
This isn't an argument against agents. I've seen enough of what they can do to believe they're going to become an important part of how businesses operate. But it does suggest we need to be realistic about what running them actually involves.
The analogy isn't perfect. AI agents aren't children. But operationally there's something recognisable about it. Imagine a room full of very capable toddlers. You give everyone a job. One wanders off halfway through. One brings you exactly what you asked for. One confidently brings you something completely different. One gets stuck but doesn't tell you. Another announces "finished!" It hasn't finished. So you ask one particularly capable toddler to check everyone else's work. It returns and tells you everything's fine. You look around. Everything is not fine.
The answer isn't to stand in the centre of the room shouting increasingly detailed instructions. You change the environment. Which doors are locked? What are they allowed to touch? Where can they go? Which things require you? How do you know something has gone wrong? How do you intervene? That's the important part of the analogy. You control the environment, not every individual decision.
Traditionally we might separate these. Reliability: did the system work? Security: did somebody do something they shouldn't? But with agents, those boundaries become less neat. Imagine an agent deletes the wrong customer records. Perhaps an attacker manipulated it (security incident). Perhaps prompt injection manipulated it (security incident). Perhaps it misunderstood the instruction (reliability failure). Perhaps another agent supplied incorrect information (multi-agent failure). Perhaps its permission was far broader than necessary (architecture failure). Perhaps nobody realised for three days (observability failure). The classification matters when you investigate. But the customer records are still gone. A bad outcome doesn't care which department owns the terminology.
This is one reason agent security needs to move beyond defending against attackers. "Agent may delete maximum 10 records per operation" can limit malicious activity, prompt injection and accidental deletion. "Payments above £500 require approval" can limit fraud, compromise and an agent misunderstanding an invoice. "External email may only go to approved domains" can limit data exfiltration, prompt injection and the agent accidentally sending something to the wrong recipient. Good security boundaries can also be good reliability boundaries.
This is perhaps the simplest way to describe the current problem. An agent may be capable of editing 50,000 database records without its judgement being reliable enough to decide when 50,000 records should be edited. It may be capable of sending an email to every customer without being the right thing to independently decide when to do so. It may be capable of deploying production code without every deployment needing to occur without another control.
Capability can exceed judgement. Humans experience this too, which is why organisations have roles, permissions, approval limits, separation of duties, financial controls and reviews. Agents need equivalent concepts.
This doesn't mean putting a human in front of everything, which would defeat the point. If an agent reads a source and asks a human, analyses and asks, drafts and asks, saves and asks, we haven't created useful autonomy. The goal isn't a human in every loop. It's a human at the right consequence. An agent may autonomously research, analyse, compare, summarise, prepare, test and monitor, then require approval to send, spend, publish, delete, change production or access particularly sensitive information. The exact boundary depends on the job.
Security shouldn't turn agents back into conventional automation. One of the reasons agents are useful is that they adapt: if Route A doesn't work, try Route B; if information is missing, look elsewhere; if one tool fails, use another approved tool. We want that flexibility. So the security model shouldn't be "follow exactly this path". It should be "here is the environment you can work inside, and here are the boundaries you cannot cross". That's a different model.
The agent cannot negotiate with the boundary. If a system is intentionally prohibited, the agent shouldn't be able to reason its way around the restriction. It can ask for permission, escalate, explain why access is required, or continue without it. But ACCESS DENIED means denied.
This is another area where the conversation can become more dramatic than the current reality. People imagine groups of agents secretly colluding. Perhaps one day that's a meaningful concern. Today I'm much more interested in something less cinematic: cascading incompetence. Agent A produces the wrong result. Agent B trusts it. Agent C takes an action based on it. Agent D checks the process and misses the original error. Now one mistake has moved through four agents. No collusion required. Just trust.
Imagine Research Agent to Analysis Agent to Content Agent to Publishing Agent. If Research Agent misses an important source, Analysis Agent may analyse incomplete information, Content Agent may create a convincing article, and Publishing Agent may publish it. Every downstream agent may perform its own task correctly. The original failure still reaches the final outcome. That's why multi-agent observability needs provenance, coverage, verification and delegation history, not simply "did each agent complete its assigned step?"
And the final output can look excellent, which is part of what makes agent failures tricky. Bad software often produces an error. AI can produce something convincing. An incomplete research process can still produce a beautifully written report. A misunderstood customer request can still produce a polished email. So output quality isn't necessarily proof of process quality. You need to know what happened before the output appeared.
Monitoring asks "is the agent running?" Observability asks "what actually happened?" For an important task that might include what was expected, what actually ran, which tools were used, which systems were touched, which sources were checked, which actions failed, which were blocked, which approvals occurred, which agents were involved, what outcome resulted and how it was verified. You shouldn't need to read thousands of lines of logs; the system should help reconstruct the task.
And then there's absence. Traditional monitoring is often good at detecting that something happened. Agents also require us to detect that something didn't happen: the expected source wasn't checked, the expected agent didn't run, the expected output wasn't created, the expected test wasn't performed, the expected approval was skipped, the expected backup doesn't exist. Those are easy failures to miss because there's no dramatic event. Nothing happened. Exactly.
If we build a supervising agent and ask it "did all the other agents work correctly today?", we haven't solved the fundamental problem. We've created another probabilistic participant. Agent Control needs to use observable evidence wherever possible. When an agent says SUCCESS, the system asks what evidence demonstrates success.
That's why agent-reported status and verified status should be treated as different things. An agent may report SUCCESS on a task that expected 50 sources checked while only 43 were observed, giving a verified status of INCOMPLETE. An agent may report SUCCESS on a backup that doesn't exist, giving a verified status of FAILED. And an agent may report BLOCKED because it correctly stopped before a £500 payment, giving a verified status of "blocked correctly". Not every block is a failure. Sometimes the control system is working exactly as intended.
If our control model assumes agents must never fail, we're going to be disappointed. Humans fail. Software fails. Networks fail. APIs fail. Models fail. Agents fail. The goal should be that failure doesn't automatically become consequence, and that failure becomes visible quickly enough to do something about it. That's a much more realistic objective.
One of the simplest improvements is also one of the most powerful: define success before the task starts. Don't give an important agent "check our websites". Give it "check these 12 websites, record whether each responded successfully, verify the expected content is present, and treat a run as incomplete if any site cannot be checked". Now success isn't whatever the agent thinks it means. It's defined externally. The same applies to security: rather than "be careful with sensitive information", define which information, which systems, which destinations, which actions, which approvals, which limits and which exceptions. The clearer the environment, the less judgement you're asking the model to invent.
Security shouldn't depend on the agent being smart enough. A good architecture should still work when the agent misunderstands, gets distracted, is manipulated, hallucinates, chooses a strange route, or simply isn't as clever as we hoped. If the control system only works when the model makes the correct decision, it isn't much of a control system.
As businesses deploy more agents, we're going to need something between the human and the growing collection of autonomous systems doing work. A control layer that answers: which agents exist, who owns them, what are they supposed to do, what can they access, what are they allowed to do, what are they doing, did they actually do the job, what failed, what looks unusual, which ones need me, and can I stop them? That's not simply cybersecurity, monitoring, AI governance or workflow automation. It's starting to look like Agent Operations.
Without that layer, the human ends up doing it manually: remembering, checking, cross-referencing, reviewing, opening dashboards, looking at logs, checking whether scheduled tasks ran, asking agents whether they finished, and checking the agents that checked the agents. That's exactly the sort of cognitive overhead agents were supposed to reduce. Automation shouldn't require a human to remember whether the automation is automating.
Agent Control grew out of running agents in the real world. The more agents we used, the clearer the problem became. It wasn't simply "how do we secure these agents?" It was "how do we actually control them?" Know what exists, what's running, what's failing, what has authority, what needs approval, what actually happened, and what requires human attention: KNOW, DEFINE, CONTROL, OBSERVE, VERIFY, INTERVENE. Not because agents are secretly plotting, but because they're increasingly useful, increasingly connected, increasingly capable of action, and still imperfect.
Because this is probably the best time to do it. Before hundreds of agents, thousands of autonomous actions, deep system access, complex delegation chains and business-critical dependency. We have an opportunity to build the control model while the technology is still developing.
And perhaps the most important principle is also the simplest: the less reliable agents are, the more important authority control becomes. We don't need to assume agents are brilliant. We don't need to assume they're malicious. We don't need to assume they're secretly collaborating. We only need to recognise what is already happening.
We're giving probabilistic software the ability to act. That is enough to require a different control model.