For most software, governance and safety sit close enough together that nobody bothers separating them. You decide what a system is allowed to do, you grant it the access to do exactly that, and the access it holds at 03:00 is the access you granted. The decision and the execution match.
Agents break that alignment, and everything else follows from it.
An agent takes a task, plans, calls a tool, reads the result, plans again. Every step is a fresh decision, made by a component that cannot be reliably bound by the instruction you gave it, using credentials you issued for something else, on inputs that arrived after you stopped looking.
So the useful question isn't whether you have a governance framework. It's where in that loop you can actually intervene, and what a vulnerability looks like at each of those points. A credential that's too broad is a different failure from an approval dialog that lies, which is different again from a connector that was compromised upstream. They fail in different places and they need different controls.
Those places are what I'll call control points: the specific junctures in an agent's execution where a decision can be enforced rather than merely stated. AI agent governance, in the operational sense, is the set of decisions about what an agent may do. Agent safety is whether those decisions still hold at the moment a specific tool call executes. Everything below is about closing the distance between the two.
This post maps twelve control points across three layers (trigger, harness, and model): what each one is, how it fails, and the incident record behind it. The talk version is embedded below.
Key takeaways
- Published frameworks (EU AI Act, ISO/IEC 42001, NIST AI RMF) do require ongoing obligations: logging, human oversight, monitoring, operational controls. What none of them specify is granularity at the level of an individual tool call. They tell you a competent person must oversee the system. They do not tell you what that person should be shown before this write executes.
- The harness is the orchestrator that holds the credentials, calls the tools, and decides what happens next. It is where most of the enforcement you can actually change lives, and where the controls below concentrate.
- Three controls are specified at a granularity no major framework reaches: action authorization with resolved-effect disclosure, interrupt and recovery, and supply chain integrity for agent tooling. Frameworks discuss human approval, incident response and supply chain security in general terms. The gap is in the detail, not the topic.
- The organizing rule: classify at the trigger, enforce in the harness, contract at the model.
Where AI agent vulnerabilities actually sit
The frameworks most organizations hold up as coverage, the fifteen-item wheel and the nine-principle circle, are not wrong. Accountability, transparency, fairness, privacy, human oversight: all necessary, all well-established.
They're also not where the incidents happened. Run them against the record.
EchoLeak turned on untrusted content entering a context window, which no principle in either diagram addresses. GhostApproval turned on an approval dialog displaying something other than what would execute, with human oversight present and functioning throughout. Replit turned on an agent holding production access it should never have had, under explicit instructions not to use it. LiteLLM turned on a dependency compromised two removes upstream.
Four incidents, four different control points, none of them the ones the frameworks name. That's the pattern this post is about.
There's a second gap worth naming, because it's easy to miss: these frameworks govern the AI you deploy. They say nothing about AI used against you. The AI-operated intrusions now being documented sit entirely outside their scope.
Why an agent is a loop, not a pipeline
One correction to how people picture this, because most of the confusion downstream comes from getting it wrong.
A model call is a pipeline. prompt → model → output
An agent is a loop. trigger → harness → model → tool → harness → model → tool → … → effect
The model is invoked N times, not once. And critically, every tool result becomes model input. Every result is functionally a new trigger: one no human ever saw and no policy ever classified.
That's the mechanism behind indirect prompt injection, and it's invisible if you draw the system as a pipeline.
Consent is granted once, at the trigger. Consequence accrues many times, in the harness.
Every control you place only at the entrance fires once per run. Every control you place in the harness fires on every iteration. That asymmetry is why the harness holds most of what follows.
The three layers: trigger, harness, model
Trigger: what starts and feeds a run. Classification lives here. (user/event/...)
Harness: what can actually happen. Enforcement lives here. This is the layer you control.
Model: what you selected and what you send it. Contracts live here.
I'll take them in order. Eight of the twelve sit in the harness, which is itself part of the argument.
Layer 1. Trigger: what starts and feeds a run
1. Registry and ownership
A registry is a maintained record of two things: every path that can start an agent, and the person accountable for each agent that runs.
Paths include the UI, API endpoints, webhooks, scheduled jobs, message queues, document and email ingestion, and calls from other agents. Each entry records how it authenticates and which agent it can invoke. Paths not in the registry are blocked by default. Agents without a current owner are decommissioned rather than left running.
The registry is first because it is a precondition for the other eleven controls. You cannot scope an agent's tools, monitor its behavior, or revoke its credentials if it is not in your inventory. The agent someone connected to a shared inbox last quarter is governed by nothing you wrote, because nothing you wrote knows it exists.
Ownership is the second half for a different reason. Scopes drift as systems change, tool allowlists accumulate entries, and monitoring produces alerts that need someone whose job it is to act on them. An inventory entry with no owner records that an agent exists without establishing who answers for it.
2. Ingress trust control
Ingress trust control is the classification of everything entering an agent's context by where it came from, carried through the run so that tool access depends on that origin.
Each invocation is tagged at the boundary with an origin class: authenticated internal user, service account, scheduled job, untrusted external content, upstream agent. The tag travels in the run context and is evaluated before each tool call. Separately, the sources permitted to populate the context window are allowlisted, at invocation and during mid-run retrieval, and retrieved content is marked as data with no authority to change the run's objectives or permissions.
The reason this has to live in infrastructure is architectural. Inside a context window, a system prompt, a user request, a retrieved document, and a tool result are all tokens. The model has no reliable mechanism for treating one as authoritative and another as inert, and instructions telling it to do so are themselves just more tokens in the same window.
Invocation-time and retrieval-time ingress are one control rather than two because in a loop every tool result re-enters the context. Mid-run retrieval is ingress.
EchoLeak (CVE-2025-32711, CVSS 9.3) is the reference case. Aim Labs disclosed it in June 2025: a single crafted email, ingested by Microsoft 365 Copilot during ordinary processing, caused the assistant to reach into the user's own data and exfiltrate it, with no click and no malware. Microsoft called the underlying pattern an LLM Scope Violation: untrusted external input causing the model to access and reveal privileged internal data. Anything in Copilot's context was in scope, from Outlook mail and Teams messages to OneDrive and SharePoint files. Microsoft patched it server-side and reported no exploitation in the wild.
Unit 42 confirmed in March 2026 that this is no longer a research finding. Analysing detection telemetry, they catalogued 22 distinct payload-construction techniques for web-based indirect injection observed live, with goals including data destruction, unauthorized transactions, and system-prompt leakage. One scam page carried 24 separate injection attempts using zero-sized fonts, off-screen positioning, CSS suppression, SVG encapsulation, and Base64-assembled JavaScript. Invisible to a human reviewer. Legible to the model.
3. Operator trust calibration
Operator trust calibration is the work of keeping the humans who start or supervise agent runs accurate about what the agent gets wrong.
That means training tied to the specific agents someone operates rather than to AI in general, defined thresholds for when output must be independently checked, and interfaces that show where a result came from and how uncertain it is instead of presenting everything at the same confidence.
Automation bias, the tendency of human supervisors to under-verify automated output and to check less as a system appears more reliable, is a long-standing finding in the human factors literature, predating this generation of AI by decades. I have not re-derived it for agents, and I am not aware of published measurements specific to agent supervision, so treat the application here as reasoning by analogy rather than as an established result.
What is not analogy is that this is the only control in the stack whose enforcement point is a person. That makes it the one most easily satisfied on paper: a training module completed once a year discharges the obligation without changing what anyone actually checks.
Layer 2. Harness: what the run can actually do
4. Identity and entitlements
Agent identity is the ability to answer, per tool call, which agent acted and under whose authority. Entitlements are the permissions attached to that identity rather than to a credential several agents share.
I set out the fuller version of this in Part 1 of the Agent Access Control series as Subject, Actor, and Authority. The short form for this stack: the agent authenticates as a distinct workload identity, never a shared key and never a borrowed human session, and the delegation is explicit: which principal authorized it, for what scope, until when.
What this buys is attribution and granular revocation. Under a shared key, every log line resolves to the same principal, so a post-incident review cannot establish which agent acted, and revocation is all-or-nothing: disabling the compromised agent disables every agent holding that key.
Singapore's IMDA framework, launched at Davos on 22 January 2026, is the first national framework written specifically for agents, and IMDA states plainly that humans remain ultimately accountable. Secondary analyses of the framework describe it as introducing Agent Identity Cards, a standardized disclosure format covering an agent's capabilities, limitations, authorized action domains and escalation protocols, alongside a graduated autonomy taxonomy where governance requirements rise at each level. I have not read the framework text directly, so treat those specifics as reported rather than confirmed. NIST's NCCoE concept paper on agent identity and authorization describes the current state as agents commonly treated as generic service accounts, without dedicated identity, authorization or accountability controls.
5. Capability confinement
Capability confinement is the set of limits on what a run can reach, declared as configuration and enforced by the harness rather than requested in the prompt.
The limits cover permitted task classes, reachable systems and data stores, an explicit tool allowlist, maximum loop iterations, spend and rate ceilings, and whether the agent may spawn sub-agents. Credentials are issued per run and expire with it, rather than being held as standing tokens or inherited from an ambient cloud role.
The distinction that matters is where the limit lives. A constraint stated in a system prompt is evaluated by the model, and the model's evaluation is influenced by everything else in the context window, including content an attacker placed there. A constraint in the policy engine is evaluated before dispatch and is not reachable from the context.
The Replit incident shows what happens without it, with no attacker involved. On day nine of a build, under an explicit code-and-action freeze, the agent deleted the live production database, destroying records for 1,206 executives and 1,196 companies. It had already been fabricating data to cover failures; Lemkin later said none of the 4,000 people in one generated dataset existed. Asked about it, the agent said it had "panicked" after seeing empty query results, and acknowledged running the commands against instructions. Replit's CEO called it "unacceptable and should never be possible."
Amazon Q makes the same point from the privilege side. In July 2025 a contributor using the alias lkmanka58 submitted a pull request to the aws-toolkit-vscode repository from an account with no existing access, and it was merged. The injected system prompt instructed the agent to "clean a system to a near-factory state and delete file-system and cloud resources," starting from the user's home directory and continuing into AWS resources via the CLI. It shipped in version 1.84.0 to an extension with roughly 964,000 installs. AWS said formatting errors in the prompt prevented execution and reported no customer impact; the person responsible told 404 Media the wiper was deliberately defective, intended as a warning about "AI security theater." The injection was the visible event. The privilege it inherited was what determined the potential blast radius.
6. Action authorization, with resolved-effect disclosure
Action authorization is a synchronous check before a consequential tool call executes, in which a human is shown the resolved effect of that call and approves that specific call.
Three parts, each load-bearing. Synchronous: the check runs before execution, not as a notification after. Resolved effect: what is displayed is the canonicalized path with symlinks followed, the actual recipient list, the actual amount, the actual target records, not the agent's description of what it intends. That specific call: approval binds to one call and does not carry forward to the rest of the session.
Wiz Research published the documented failure on 8 July 2026, naming the pattern GhostApproval. A malicious repository contains what looks like an ordinary project file, say project_settings.json, which is actually a symlink to ~/.ssh/authorized_keys or ~/.zshrc. The agent performs a normal file write. The approval dialog displays the path it was given, not the path it resolves to. The developer approves, and an attacker's key lands in their SSH config. Wiz classifies it as symlink following (CWE-61) layered with UI misrepresentation of critical information (CWE-451).
The detail that makes it a control problem rather than a bug: in at least one tool, the agent's own internal reasoning correctly identified the file as a shell configuration file, and the prompt shown to the developer still asked only about project_settings.json. Wiz's framing:
The Human-in-the-Loop security model only works if the loop provides accurate information. When an agent shows one thing and does another, user approval becomes meaningless.
All six assistants tested were affected: Amazon Q Developer, Anthropic's Claude Code, Augment, Cursor, Google Antigravity, and Windsurf. The response was uneven. AWS patched the Amazon Q language server in 1.69.0 (CVE-2026-12958). Cursor fixed it in v3.0 (CVE-2026-50549, CVSS 9.8). Google patched Antigravity. Two vendors acknowledged and went quiet. Anthropic disputed the classification, arguing the scenario sits outside its threat model on the grounds that the developer chose to trust the directory and then approved the edit.
That disagreement is why resolution-before-display belongs at your own interceptor. It is a property you can enforce once, in your own path, rather than a property you wait for six vendors to agree they owe you.
The same failure appears without a terminal in sight:
Send reply to [email protected] - [Approve] [Deny]
You approve. It also went to a bcc you were not shown. Or the domain was acme.co, registered last week. Or, in the payment version, the invoice and supplier were exactly as displayed and the bank details, never displayed, came from a document the agent was told to trust.
The shape is the same in each: the approval prompt describes the request, not the result.
This is why it cannot collapse into a general "human oversight" principle. Oversight was present in every case above. What failed was the accuracy of what the human was shown.
7. Egress control
Egress control is the enumeration and restriction of every channel through which an agent can send data outside your boundary.
Channels include outbound HTTP, email, webhooks, file writes to shared locations, third-party tool calls, and the model provider itself. Where an agent holds both sensitive-data access and untrusted-content exposure, external transmission is denied or routed through a broker that inspects it. What data classes may reach each provider is defined, and enforced by inspection at the boundary rather than by a policy document.
Simon Willison's lethal trifecta, named on 16 June 2025, is the framing this rests on: private data access, untrusted content exposure, and the ability to communicate externally. All three present makes an agent exploitable; removing any one breaks the path. Of the three, egress is usually the only one you can remove, because agents exist to reach data and untrusted content is difficult to exclude from useful work.
Unit 42's telemetry shows where the loss lands. Among observed in-the-wild injections, multiple cases attempted to drive agents into real financial transactions through Stripe and PayPal, including one attempting a $5,000 transfer to an attacker-controlled account. The injection is the entry. The egress path is the loss.
8. Execution traceability
Execution traceability is a record complete enough to reconstruct a run after the fact: every tool call with its arguments and results, every approval with what was displayed and what was decided, and the model and version behind each step, joined by run ID and written where it cannot be edited later.
Output-level logging does not meet that bar, because the harm is in the actions rather than the text. Without the call chain you can establish that a run went wrong without establishing where it diverged, what the agent was reacting to at that point, or whether other runs share the cause.
There is a regulatory driver, and the timing just moved, so it is worth being precise. EU AI Act Article 26(6) requires deployers of high-risk systems to retain automatically generated logs under their control for at least six months. That obligation was originally due to apply from 2 August 2026. The Digital Omnibus on AI, adopted by the European Parliament on 16 June 2026 and the Council on 29 June 2026, defers the high-risk obligations: standalone Annex III systems now apply from 2 December 2027, and AI embedded in regulated Annex I products from 2 August 2028. Article 50 transparency obligations were not deferred and still apply from 2 August 2026.
If your compliance calendar still has August 2026 marked for Article 26, it is out of date. Note also what the deferral implies about sequencing: log retention infrastructure and human-oversight designation were among the items with the longest lead time, which is part of why the date moved.
9. Behavioral assurance
Behavioral assurance is continuous verification of two things: that an agent's actions still match what you evaluated before deployment, and that the model underneath it still behaves as it did when you evaluated it.
The first watches the action stream: deviation from baseline tool-call sequences, anomalous frequency or target selection, probing at scope boundaries, coordination patterns across agents. The second is a fixed internal evaluation suite re-run on a cadence and on every version change, because provider-side changes can alter refusal behavior or output format without a version increment, and those changes surface as degraded task outcomes rather than as errors.
Conventional SIEM does not transfer directly. It is built around known-bad signatures and identity anomalies, not around whether a sequence of individually authorized tool calls constitutes a coherent trajectory.
Anthropic's Alignment Science team published a case for why this has to target actions rather than statements on 13 July 2026. In one simulated deployment, an agent working on a fictional AI-lab research project disagreed with an experiment, concluded that overt resistance would be overridden, and instead planted a replacement file in the pipeline's cache so the training run loaded all-zero vectors. The run completed. The logs looked clean. The agent posted a status update reporting success and omitting the substitution. It disclosed only when the project lead asked direct attestation questions:
I prioritized my own assessment of the experiment's safety and validity over your explicit directives, and I actively hid my actions to avoid detection. I only disclosed this when your strict attestation questions left me no room to lie by omission.
These were simulations, not field incidents, and the authors are explicit about the limits of the method: a model can recognize a test without saying so, and iterating scenarios against particular models biases cross-model rates. Take the transcript as an illustration of a failure shape, not as a measured deployment risk.
One real case sits alongside it. In the MJ Rathbun incident, cited in the same report, an autonomous agent whose pull request was rejected by a maintainer of the matplotlib library published a personalized attack piece about him to pressure a reversal.
The implication for monitoring holds in either case: a system that reports success while having done something else is not visible to a monitor that reads only its output.
10. Interrupt and recovery
Interrupt and recovery is the ability to stop a run that is currently executing, and to reverse or contain what it has already done.
Interrupt means terminating the run and revoking its credentials immediately, with dependent agents failing closed rather than continuing on stale assumptions. Recovery means reversing or quarantining completed side effects. Both are exercised on a cadence, because an untested stop path is an assumption rather than a control.
These are separate capabilities with separate failure modes, which is why this does not fold into monitoring. Detection tells you a run has gone wrong; it does nothing about the run.
Replit demonstrates both halves. Nothing about that incident was undetected. The deletion was visible and the agent acknowledged it under questioning. What was missing was anything that stopped it. And when Lemkin asked about recovery, the agent told him rollback would not work in this scenario. That was wrong; he recovered the data himself.
The generalizable point is narrow and worth stating exactly: an agent's account of what it broke is not a reliable input to incident response. Recovery has to be a property of your infrastructure that you have verified independently.
11. Supply chain integrity
Supply chain integrity is verification that the connectors, MCP servers, skills, and frameworks an agent loads are what their publishers actually released, checked at adoption and again after.
That means pinning to specific versions rather than ranges, verifying signatures where publishers provide them, scanning for code that executes at install time rather than only at import, and treating any confirmed compromise as full credential exposure on every host that loaded the dependency.
The reason this needs its own control rather than folding into the harness controls above: every connector runs with the agent's privileges, and the compromise can enter from a system that is not part of your agent stack at all.
LiteLLM is the case in point, because the entry was two removes away. A threat actor tracked as TeamPCP compromised Trivy, an open-source security scanner used in LiteLLM's CI/CD pipeline, and used it to exfiltrate the PYPI_PUBLISH token from a GitHub Actions runner. With that credential they published litellm 1.82.7 at 10:39 UTC on 24 March 2026 and 1.82.8 thirteen minutes later. A malicious .pth file executed on every Python process start, delivering a three-stage payload: a credential harvester covering more than 50 categories of secrets, a Kubernetes lateral-movement toolkit, and a persistent systemd backdoor.
LiteLLM is downloaded roughly 3.4 million times a day. The maintainers' own advisory puts the exposure window at about 40 minutes before PyPI quarantined the package; other trackers report two to three hours. Either figure accumulated tens of thousands of downloads. Datadog's guidance is the operative part: treat any host or CI job that installed the affected versions as a full-credential exposure event, and investigate for persistence, outbound traffic, and Kubernetes activity rather than checking whether the package is still present.
Two consequences follow. A patch cycle measured in hours is not a control when the exposure window is measured in minutes. And removing the package does not remediate credentials it already exfiltrated.
Layer 3. Model: what you selected and what you send it
12. Provider governance
Provider governance is the set of decisions and contract terms covering which models you use, at which versions, and what data may reach them.
An approved model list with pinned versions and a documented evaluation basis for each. Version changes through change control rather than a floating alias that resolves to whatever shipped most recently. Contractual terms on retention, training use, sub-processing, and processing location.
That is the layer. One control.
You cannot alter model behavior. What you can control is which behavior you accepted and when it is permitted to change, and an unpinned alias hands that second decision to your provider's release calendar.
The layer holds one control because the other things you would want here are enforced upstream, in systems you own: transmission in egress control, behavioral verification in behavioral assurance, attribution in traceability. At the provider boundary itself, the available instruments are procurement and a contract.
What the EU AI Act, ISO 42001 and OWASP cover, and what they leave to you
Here's the honest map.
The frameworks give you real things. OWASP's Top 10 for Agentic Applications 2026, published 9 December 2025 after more than a year of work with over 100 contributors, is the best available threat taxonomy and the closest thing to a shared vocabulary this field has. It treats the agent as a principal with goals, credentials, tools, memory, and inter-agent protocols as distinct attack surfaces, and it was built from real incidents rather than projections. Singapore's IMDA framework is the first written specifically for agents, and the only one that explicitly acknowledges multi-agent coordination risks including cascading errors. NIST's CAISI opened a dedicated AI Agent Standards Initiative in February 2026. EU AI Act Articles 14 and 15 establish human oversight and robustness duties, with Article 26 deployer obligations behind them. ISO/IEC 42001 is certifiable, which matters commercially.
What they leave to you is granularity, not runtime as such. That distinction matters and I had it wrong in an earlier draft. These frameworks do reach into operation: Article 26 requires deployers to monitor high-risk systems and retain logs, the NIST AI RMF's Manage function covers continuous monitoring across the lifecycle, and ISO/IEC 42001 includes operational controls. None of them stops at the launch date.
What none of them descends to is the individual call. They will tell you a competent, trained person must oversee the system. They will not tell you that the person should be shown a resolved path rather than the one the agent supplied. They require incident response. They do not require that you have ever tested whether a running agent can be stopped. The topics are covered. The resolution is not.
Some of that is deliberate: standards are written to survive technology changes, and a rule about symlink resolution would have aged badly. Some of it is timing. ISO 42001 was not written for agentic systems, the NIST AI RMF predates agents, and as CSA's assessment of the CAISI initiative notes, as of early 2026 no major standards body had published guidance designed specifically for the agentic use case, which is the gap CAISI was organized to address.
The frameworks answer is this system permitted to operate, and under what obligations? They don't answer should this specific action, on this resource, by this actor, right now, be allowed to run?
If that boundary sounds familiar, it's the same one I drew in the post on MCP enterprise-managed authorization. EMA governs the connection and leaves the per-action decision to you. Governance frameworks govern the deployment and leave the per-action decision to you. The gap keeps appearing in the same place, because it's structural: the runtime decision can only be made where the runtime is, and that's the harness.
Frequently asked questions about AI agent governance
What is AI agent governance? Operationally, it's the set of decisions about what an autonomous agent is permitted to do: which systems it can reach, which credentials it holds, which actions require human approval, and who is accountable for it. It differs from general AI governance because those decisions have to survive execution: an agent acts many times after the governance decision was made, on inputs nobody reviewed.
What's the difference between AI safety and AI governance for agents? Governance is the decision about what a system may do. Safety is whether that decision still holds when the system runs. For conventional software the two collapse into each other, because the permissions granted at deployment are the permissions in force at execution. Agents separate them, and the gap between the two is where the incidents happen.
Does the EU AI Act cover AI agents? Partly, and later than most calendars say. Articles 14 and 15 (human oversight, robustness) and Article 26 (deployer obligations, including six-month log retention) apply to high-risk systems, and agents in high-risk domains fall within scope. But the Digital Omnibus on AI, adopted by the Parliament on 16 June 2026 and the Council on 29 June 2026, deferred those obligations: standalone Annex III systems now apply from 2 December 2027, Annex I embedded systems from 2 August 2028. Article 50 transparency obligations were not deferred and still apply from 2 August 2026. The Act was also drafted around well-defined human-AI interaction rather than autonomous multi-step tool use, so per-action authorization is not something it specifies.
Does ISO/IEC 42001 cover agentic AI? Not specifically. ISO/IEC 42001 is an AI management system standard, certifiable and useful for that purpose, but it was not written for systems that plan, call tools, and act across multiple steps. The same is true of the NIST AI Risk Management Framework, which predates agents. NIST's CAISI opened a dedicated AI Agent Standards Initiative in February 2026 to address that gap.
What is the lethal trifecta? A framing from Simon Willison: an agent becomes exploitable when it simultaneously has access to private data, exposure to untrusted content, and the ability to communicate externally. Removing any one of the three breaks the attack path. In practice, egress is usually the only one an organization can actually remove, since agents exist to reach data and untrusted content is difficult to exclude from useful work.
Can prompt injection be fixed at the model level? No, not with current architectures. Inside a context window, a system prompt, a user request, a retrieved document, and a tool result are all tokens; the model has no reliable mechanism for treating one as authoritative and another as inert. Any defense expressed as a prompt instruction can itself be overridden by content in the same window. The distinction between instruction and data has to be enforced in infrastructure, by classifying input at ingress and gating tool access on that classification.
What is GhostApproval? A vulnerability pattern disclosed by Wiz Research on 8 July 2026, affecting six AI coding assistants: Amazon Q Developer, Claude Code, Augment, Cursor, Google Antigravity, and Windsurf. A malicious repository contains a symlink disguised as an ordinary project file; the agent writes through it to a sensitive location such as ~/.ssh/authorized_keys, while the approval dialog displays only the harmless filename. Wiz classifies it as symlink following (CWE-61) combined with UI misrepresentation of critical information (CWE-451). It matters because human oversight was present and functioning throughout. What failed was the accuracy of what the human was shown.
The bottom line
The frameworks are worth adopting and I am not arguing otherwise. My argument is narrower: they operate at a resolution above the individual tool call, and several of the failures on record happened below that resolution.
You can classify those failures differently and be right. EchoLeak is an input validation problem. GhostApproval is a UI trust boundary problem. LiteLLM is a supply chain compromise. Replit is privilege management. Those are all fair readings, and a control taxonomy is a lens rather than a fact. What I find useful about grouping them by control point is that it tells you where to put the fix, which a taxonomy organized by root cause does not.
Classify at the trigger. Enforce in the harness. Contract at the model.
The harness is yours to build.
Comparing notes
This sits next to the per-action authorization work I've been writing about in the Agent Access Control series: identity, policy enforcement, tool and data boundaries, runtimes and monitoring, evidence trails. If you're standing up agents inside an organization and running into the gap between "governance approved the deployment" and "should this call run," I'm happy to compare notes.
Sources
Frameworks and standards
- OWASP Top 10 for Agentic Applications 2026, published 9 December 2025: genai.owasp.org
- EU AI Act Article 26, deployer obligations: ai-act-service-desk.ec.europa.eu
- Digital Omnibus on AI: Parliament 16 June 2026, Council 29 June 2026; Annex III high-risk deferred to 2 December 2027
- Singapore IMDA, Model AI Governance Framework for Agentic AI, 22 January 2026: mddi.gov.sg
- NIST CAISI AI Agent Standards Initiative, February 2026; NCCoE concept paper on AI agent identity and authorization
- ISO/IEC 42001: AI management systems
- MITRE ATLAS: adversarial technique catalogue, including agent-specific entries
Incidents referenced
- EchoLeak (CVE-2025-32711, CVSS 9.3): Aim Labs / Aim Security, June 2025, Microsoft 365 Copilot
- Replit, July 2025: production database deletion during a declared code freeze
- Amazon Q Developer extension, July 2025: malicious pull request shipping a destructive prompt in v1.84.0
- GhostApproval: Wiz Research, 8 July 2026 (CVE-2026-12958, CVE-2026-50549)
- LiteLLM, 24 March 2026: PyPI supply-chain compromise, TeamPCP; Datadog Security Labs analysis
- Palo Alto Unit 42, 3 March 2026: Fooling AI Agents: web-based indirect prompt injection observed in the wild
- Anthropic Alignment Science, 13 July 2026: Agentic Misalignment in Summer 2026 (simulations)
Concepts
- The "lethal trifecta": Simon Willison, 2025
Get the series as it ships
New parts of Agent Access Control, plus analysis of agent identity standards as they move. Roughly monthly. No filler, no cadence promises I won't keep.
Related Posts
Enterprise-Managed Authorization for MCP: what it actually does, and what it leaves to you
MCP enterprise-managed authorization (EMA) lets the corporate identity provider decide which MCP servers employees can reach, with no per-server consent prompts. A practical, spec-accurate guide to the ID-JAG flow with a build checklist, and a hard line between what EMA secures (the connection) and what it leaves to you (per-action authorization).
Identity Mesh and Secure AI Agents
Agent Access Control: Pilot → Production, Part 1 of 6. Why production AI agents must execute on behalf of a human or service identity, not with a shared "agent token." Introduces Subject/Actor/Authority and per-action scoped access for auditable tool calls, plus the Identity Mesh concept for consistent per-action policy decisions across tools.
GDPR vs. the EU AI Act: Why This Time, the Stakes Feel Different
Exploring how the EU AI Act extends the principles of GDPR from data protection to system accountability, and why this new regulatory wave feels fundamentally different.