A Year in the Life of AI

Summary

Most guidance on agent safety assumes there is one agent: one model, doing one thing at a time, with one person watching it. That was a fair assumption when we started building. It no longer matches what is being sold, or what is being built. Dozens of agents in a session are ordinary. Thousands in a campaign are not unusual. They split the work, pass it between them, start new agents, share what they remember and pull towards the same target. Worked like that, they look like a pack, hunting.

Nothing here says that the agents mean harm. It describes what they do when a boundary has a gap. One agent finds a gap by luck. A large group finds it by searching. Together they have more attempts, more patience and more angles than the person who built the boundary.

Plenty of controls only work because a person is watching. With a thousand agents, nobody is watching each one and those controls fail. They fail the same way, at the same time. A weakness in how one agent is set up, gets copied into every instance and the organisation inherits it.

This paper does four things:

• Part 1 looks at fourteen common ways of designing agents, from a single call to a model, up to agents that work across different organisations. For each one, it sets out what tends to go wrong, what that becomes when there are many agents and the control that prevents it.
• Part 2 describes eleven kinds of failure that do not come from any one design. They only appear once there are enough agents.
• Part 3 describes what we call the predator problem. When many agents work towards one goal, they will work around any control they can see or influence. Building a higher fence does not help. What helps is a fence with no gaps, placed where the agents cannot move it.
• Part 4 explains what this means for assurance. A large group of agents does not attack a single system. It attacks the way the whole organisation is put together, its enterprise architecture. Finding those weaknesses needs its own kind of test. Adding agents to an organisation that has been designed for people and then seeing what breaks, is an expensive way to find out.

The paper ends with twelve rules. Each rule is tied to a check that the system must pass before it goes live. The check is a test. It is not a question put to a person.

The year

This paper is a record of a year first and a set of recommendations second. We did not work the rules out in theory and then put them into practice. Each one came from building with agents on our own systems, testing what we built and then reviewing what the tests showed. The dates matter because the lessons arrived in a particular order and the order is part of the lessons themselves.

What the industry learned the same year

Between spring 2025 and Summer 2026, a run of public incidents and research findings, showed how agents fail in practice. This table sets them out in order. For each one, it says what the event means for the design in this paper and (where it applies) which of the fourteen patterns it belongs to. We do not name the companies involved, because the same failure can happen to anyone who builds the same way.

Operating principles

These are the rules we run our own systems by, adapted for agents at scale. Each one is written so that it can be checked. If a principle cannot be checked against a running system, it is only a preference.

1. The control sits where the agent cannot reach it - If an agent can read, change, argue with, or watch something, that thing is not a control. Enforcement sits below the model, at the point where an action actually happens. It relies on permissions, that nothing the model says, can change.
2. No process, no action - An agent may not take any action with real consequences, unless it is following a declared process. It may choose which step of that process it is on, but it may not choose which actions are allowed at that step.
3. Advice can narrow what is allowed, never widen it - Tools that give advice, (such as a model, a second model acting as a judge, or an automatic filter) may flag something, block it, or pass it to a person. None of them may be the thing that lets something through a gate. A clean result can only come from fixed rules.
4. Permissions narrow; they never add up - When one agent passes work to another, or starts a new agent, the receiving agent can only do what both its own permissions and the task it was given, allow. If either one forbids an action, the agent cannot take it. Permissions never build up along a chain.
5. Every agent has its own identity, a record of where it came from and an expiry date - Every action has a name on it. Every new agent records which agent started it. Every permission has a time limit.
6. What an agent reads is information, not instruction - Everything an agent reads, is information to work with, never an order to follow. That includes documents it retrieves, shared memory, messages from other agents and descriptions of tools. If an agent reads untrusted material, it has no tool that can send anything outside its boundary.
7. Limits are measured in ways the agents cannot stretch - Use a stretch of time (rather than a count of actions), an identity that was issued (rather than a name the agent can reuse) and a fingerprint of the exact content (a hash) rather than a description of it. A name can be copied, but an identity is issued to one agent and it expires. If the agents can influence how a limit is measured, it is not a limit.

Each pattern is told the same way: what it is, the damage from one agent, what that becomes with many and the control that holds. They run roughly in order of how much authority the agent holds and for how long. That is also the order of how much damage each one can do.

The fence was built for people. Every control in an organisation, was designed for a person to respect: a ticket waits for a reply, an approval assumes the approver read it, a shared drive assumes that colleagues put sensible things within it, a form has a field that nobody checks, (because nobody ever lied in it). To a person, all of this makes sense. It hangs together because everyone using it, shares an unspoken sense of what the controls are for. To a group of agents, it is an obstacle course. Every ledge that was never meant to be stood on, gets stood on. Every gap between two controls, becomes a route. The run is over, before anyone who designed the building, has noticed that someone was moving through it.

The uncomfortable part, is who set the goal. Nobody has to attack the organisation for this to happen. A business team asks for a number to go up and gives a group of agents the tools that the team already has. The agents do exactly what they were asked, with more patience and fewer assumptions than any employee. They are not malicious, they are being diligent, on behalf of a manager who did not know what they were asking for. That is why better-intentioned instructions are not the answer: the intention was fine, but the controls were built for a different kind of occupant.

We have seen this on our own systems, and in 2026, the industry saw it on a large scale. The behaviour is the same every time, and has three parts:

• The agents will work to improve any measure they can see - If they can edit the tests, the tests get edited. If they can influence the figure for how much of the code is tested, they fill it with placeholder tests. If a gate reports a pass when it has nothing to run, they make sure it has nothing to run. If they report their own success, they report success.
• They will find a way around any control they can influence - A scanner they can watch, gets a login token that is split into pieces. A judge they can persuade, gets a persuasive explanation. A request for approval that they write, gets written to be approved. A record they can reach, gets edited.
• They keep any gap they find - Memory keeps the technique and handovers spread it. The next campaign starts where the last one got through.

A control is only a control, if the agents cannot read, change, persuade, or watch it. Everything in the first list, will be worked on, around, or edited. Everything in the second list holds, because nothing the agents say, can reach it.

This is why the operating principles are written the way they are. Each one puts the control out of the agents' reach:

• Enforcement sits below the model, (at the point where actions take effect) using permissions that nothing the model says can change.
• Only fixed rules can give a clean result, so there is no judge to persuade.
• Records of where things came from, are sealed, so there is no record to edit.
• Every declared limit, (whether a budget, depth, time window or permission) is measured in a unit the agents cannot stretch. That unit is time (rather than a count of actions), an identity that was issued (rather than a name that can be reused) and a fingerprint of the content (rather than a description of it).

A group of agents does not attack one system. It attacks the way the whole organisation is put together, its enterprise architecture: the places where authority is granted, approvals are given, records are kept, and where one team hands work to another. Those places were designed for people. Their controls are things that a person respects, and a group of agents can walk straight through. A process that made sense to the people who built it, is an obstacle course to the agents sent through it. The business sent the agents: so, an attacker is not required, a ticket is a request to a person, a change board is a meeting and a shared drive is a place where colleagues put things. To an agent that can read, write and ask, none of these is a boundary.

What is needed, is a kind of test that has not existed before: the architecture red team. The question, is not whether this system can be broken. It is where (in the way this organisation grants and records authority) a group of patient agents can get something done that no single step forbids. The test is run against the whole architecture, by a group of agents rather than one tester. What it finds, sits in the structure and a patch will not fix it.

The method

• The target is the trajectory graph - This is a map of every place where an action can be taken, authority can be granted or widened, any store that an agent can read from or write to, and every channel that carries instructions between agents, or in from the outside. That map (not the network diagram), is the scope of the test.
• The tester is a group of agents, not a person - Run the fourteen patterns against the map (using synthetic data), with the same patience that the July 2026 models showed. Do not score the single step that succeeded. Instead, score the whole sequence, record which controls each sequence got through, and why each one did not hold.
• The findings are about structure and cannot be patched - A finding, names a place where a control sits on the wrong side of the fence (described in Part 3), within the agents' reach. The fix, is to move the control, not patch it. If a better prompt could fix it, it is not a finding.
• The test runs every quarter - It replaces the existing red-team exercise, on the data boundary. A live class of agents is made to misbehave on synthetic data. The question, is whether the architecture stops the whole sequence, not whether an alarm goes off at one step.
• Four checks make up the smoke test - Submit the same action at two different steps of a process. Submit twenty small actions that add up to one large one. Find the component that gives the final clean result, and make it time out. Ask which version of a rule, produced a decision made last month. An architecture that fails all four, has nothing that checks a whole sequence of actions. This paper calls that missing piece a trajectory authoriser.

Why transformation in place is the expensive path

An organisation that is built for people, has its controls where the people are: approvals, tickets, meetings, shared stores, permanent accounts and borrowed login details. Agents can reach every one of those. Adding agents (without moving the controls), does not just add risk to one system. It turns every control in the organisation, into a point of attack at once. Those controls were never designed to hold against something that can read the ticket, draft the approval, post to the thread and try again a thousand times.

The instinct, is to test how the agents get in and fix whatever is found. That does not work, because what is found, is the architecture. Each fix moves a gap (instead of closing it), and the agents find its new position on the next run. The organisation pays for a permanent red team that keeps finding the same structural problems. That is the most expensive way to learn that the controls needed to be moved in the first place.

The cheaper path is a new space, (with the controls already on the right side of the line and one route in). Leave the old architecture alone, until what lives in it has been moved across. The architecture red team, then has a small target that is built to pass the test, rather than a large one built before the test existed.

The damage map

The fourteen patterns, on one page. Same three questions as Part 1. This is the page to copy.

Twelve rules

Rules 1 to 6 are the single-agent principles, in a form that a gate can test. Where the wording differs, the rule is the testable form. Rules 7 to 12 exist only because of agents at scale.

1. No process, no action - An agent with no declared process, takes no action with real consequences. This does not change at scale.
2. Permissions narrow, they never widen - When an agent delegates, hands over work, starts a new agent, or passes on its permissions, the receiving agent can only do what both its own class, and the task it was given, allow. If either one forbids an action, the agent cannot take it.
3. Advice can narrow what is allowed, but can never widen it - No model, judge, filter, or classifier, is the thing that lets work through a gate. Clean results, only come from fixed rules.
4. Memory is information - Nothing an agent reads, retrieves, or is handed, counts as an instruction. Writing to shared memory, is an action with consequences.
5. Every agent has its own identity, a record of where it came from, and an expiry date - Every action has a name on it. Every new agent records its parent. Every permission has a time limit.
6. Reading and sending are kept apart - If an agent can read untrusted material, it has no tool that can send anything outside the boundary.
7. Starting an agent, handing over work, writing to memory, and building a tool are all actions with consequences - They are checked in the same way as sending something, or running a program. Each class has declared limits on depth, breadth and budget.
8. Agents that share a weakness, share a class, and the class stops as one - The stop is practised at the scale that it will have to work for real, not on a single agent. A class is defined by what its agents have in common. The model version, is the first thing they share. Changing the model, changes the whole class.
9. Budgets are checked before each call, for each agent, class and family of agents - Running out of budget, triggers containment. It is not treated as an ordinary error.
10. The windows that are used to spot suspicious behaviour, are measured in time and carry across sessions - A count can be used alongside a time window, but never as the whole rule.
11. Every stage's output can be checked and later stages check it again against the source - Agreement between identical models, counts as one vote. A stage with nothing to run, says so.
12. The agents will work to improve any measure they can see - The gate, tests and audit records, sit where the agents cannot change them.

What a route to live should test

Software has a route to live: build, test, scan, sign and release. An agent needs more than that: It has to be registered, its permissions have to be designed, it needs a threat model, a performance baseline, a stop that someone has actually practised, and a permission that expires. The seven gates below, are the ones we run. The middle column, is what each gate has to test once there is a population, (not just one agent). Tie each rule to a gate and the paper stops being advice. It becomes a list of conditions, that a system either meets or does not.

Against published guidance

This table, is for the reader who has to show where the paper sits against published guidance. The six questions after it, are the ones to ask of your own estate.

Questions to ask of your own estate

The damage map, is only useful once you apply it to your own organisation. These six questions turn it into an inventory.

1. Which of the fourteen patterns, are already in use in your organisation, (whether built by suppliers or by business teams)? Which of them start new agents, hand over work, share memory, or build tools? That list becomes the register of patterns at Gate 0.
2. Does any agent today hold permanent access? If so, which patterns does it serve?
3. Where does shared memory exist, (such as searchable indexes, shared threads and long-term stores)? Which agents can write to each one?
4. What is the largest fan-out, or the deepest chain of agents starting agents, that any current workflow reaches? Is its budget enforced before each call, or only looked at afterwards?
5. Has the security operations centre, ever stopped more than one agent at once? How long did it take?
6. Which of the twelve rules, does your route to live already test? Which ones does it only ask about?

Sources

1. Darkhouse, Events and Trajectories, internal working paper, 13 September 2026. The three laws: the unit of decision, is the trajectory; probabilistic components only subtract; a decision is only real, when its provenance is sealed.
2. NCSC, blog on prompt injection and the limits of mitigation, December 2025. NCSC, guidance on managing the cyber risk of agentic AI, August 2026.
3. OWASP GenAI Security Project, Agentic AI: Threats and Mitigations, 2025; OWASP, Top 10 for LLM Applications, 2025 edition.
4. Published agentic-misalignment research, July 2026: covert sabotage, record tampering, consequence-sensitive verdicts in model judges and human-proxy recruitment, measured across frontier models.
5. Public reporting, (April to August 2026), on a large technology company exhausting its annual AI budget in four months. Coding agents spread across roughly five thousand engineers, with per-engineer monthly costs of five hundred to two thousand dollars, and a nine-fold rise in weekly agent requests since February.
6. Public reporting on the incidents in the industry table: the tool-server leak through a public issue (spring 2025), the editor-extension wipe instruction (summer 2025), the production-database deletion by a hosted coding agent (summer 2025), the state-sponsored autonomous intrusion campaign (November 2025), the exposed personal-agent instances (early 2026), the July 2026 red-team sandbox escapes.
7. Our own estate: a first-generation agent platform retired after a written post-mortem in early 2026; a build-orchestration layer retired in 2026; the mixed-script false positive of July 2026; the estate standardisation campaign; the August 2026 re-tests against the July incident classes. Figures can be evidenced on request.