Recommendation: treat autonomous agents as controlled software operators, not digital employees. Start with bounded, reversible workflows where tool access, identity, approval points and recovery actions can be specified before deployment. Keep high-impact decisions and irreversible changes outside the agent’s independent authority until testing demonstrates that the controls work under adversarial conditions.
Choose processes for bounded autonomy
An agent is a good candidate for a process when the objective is clear, the required tools are known, the permitted data is limited, and incorrect actions can be detected and reversed. Useful early candidates include assembling a case file, reconciling records into a proposed update, or preparing a change for human approval.
Be cautious when a process combines ambiguous objectives, sensitive data, irreversible actions and weak observability. “Resolve the issue” is not a sufficient operating specification if the agent can alter customer records or approve payments.
Use these decision criteria:
- Bounded objective: define what success and failure look like.
- Constrained action space: prefer specific business functions over general shell, browser or API access.
- Reversibility: design compensating transactions, rollback and expiry.
- Observable state: record inputs, plans, tool calls, results, approvals and final effects.
- Human accountability: name the person or role responsible for the outcome.
- Fallback path: make it straightforward to stop the agent and complete the process manually.
This is a design recommendation. The evidence base remains immature, with many published demonstrations testing short tasks in controlled environments rather than sustained enterprise operation.
Prefer orchestration over unconstrained autonomy
A production workflow should usually be an orchestrated state machine with model calls inside it, rather than a model receiving a broad goal and deciding the entire process. The orchestrator should own state transitions, retries, deadlines, idempotency, rate limits and escalation.
A practical pattern is:
- Classify and validate the request.
- Retrieve only approved context.
- Generate a proposed plan.
- Check the plan against deterministic policy.
- Execute low-risk steps through narrow tools.
- Pause for approval before defined high-impact actions.
- Verify the result independently.
- Record evidence and close or escalate.
Separate planning from execution where possible. The model can propose an action, but a policy service should decide whether the action is permitted. Downstream systems must enforce authorisation themselves. OWASP’s guidance on excessive agency explicitly recommends complete mediation, meaning every request to a downstream system is checked against security policy rather than trusted because an agent selected it. See OWASP’s Excessive Agency guidance.
Make tools narrow, typed and least-privileged
Tool design is a primary control surface. A tool called send_email is easier to constrain than a tool called run_command; update_invoice_status is easier to review than unrestricted database write access.
For each tool, define:
- allowed operations and objects
- input schema and validation rules
- permitted data classification
- rate and quantity limits
- required approval level
- failure and retry behaviour
Do not expose unused functionality “for convenience”. OWASP identifies three related causes of excessive agency: excessive functionality, excessive permissions and excessive autonomy. Its examples include read-only use cases backed by identities that can also modify or delete records.
The Model Context Protocol, or MCP, standardises connections between AI applications and external tools, data sources and workflows. It does not make a connected tool trustworthy by itself. The MCP documentation describes the protocol’s purpose, and the specification release of 28 July 2026 records changes to authorisation hardening and routing. Treat remote MCP servers as third-party software and third-party content as untrusted input. Review, pin and monitor servers, and test them with synthetic data before granting access to sensitive systems.
Design identity for the action, not just the agent
An agent may act as a user, as a workload, or as a workload delegated authority by a user. These models have different audit and containment properties.
Use user-context authorisation when the action must reflect the individual’s existing rights. Use a distinct workload identity when the process is organisational and should not inherit a person’s broad access. In either case:
- issue short-lived credentials
- separate read and write scopes
- require step-up approval for sensitive operations
- log the originating user, agent identity, tool and downstream resource
- revoke credentials independently of the model runtime
MCP’s 2026 work on issuer validation, credential isolation and scope step-up is relevant infrastructure, and the MCP Enterprise-Managed Authorization extension, stable since June 2026, adds central provisioning for connected servers. Protocol support still leaves the enterprise identity design to you. Organisations should still test token scope, revocation, confused-deputy scenarios and cross-tenant isolation in their own environment.
The security objective is to prove which principal was authorised to perform the specific action, on which resource, under which policy, at what time.
Use approval as a control, not a ritual
Human approval is useful when the reviewer can understand the proposed action, its consequences and the evidence supporting it. A prompt saying “Allow agent to continue?” is weak control design.
Approval requests should show:
- the intended business outcome
- the exact records or systems affected
- the proposed changes
- uncertainty or policy exceptions
- rollback or recovery method
- the identity under which execution will occur
Ask for approval after the agent has prepared a concrete, reviewable result, not before it has done harmless investigation. OpenAI’s current agent guidance makes a similar distinction between reversible or read-only work and external actions such as deployment, merging or publishing. See OpenAI’s agent guidance.
Do not assume frequent approval produces reliable oversight. In a May 2026 engineering account, Anthropic reported that users approved roughly 93% of permission prompts in one product, describing approval fatigue as a reason to combine human review with stronger containment. This is a provider report about its own telemetry and systems; independent validation is separate work. It still supports a practical design rule: reduce unnecessary prompts and make the remaining approvals meaningful. See Anthropic’s account of agent containment.
Contain the runtime and the data path
Prompt instructions and model safeguards are probabilistic. Runtime boundaries can be deterministic. Use separate environments, network egress controls, filesystem boundaries, secret isolation, resource quotas and termination mechanisms according to the potential impact of failure.
Containment should cover three paths:
- Execution: what code, processes and tools can run.
- Network: which hosts, APIs and data stores can be reached.
- Data: which records can be read, retained, copied or transmitted.
Keep secrets outside the model context where possible, using brokered operations that perform a narrow action without exposing reusable credentials, and monitor outbound traffic for unusual destinations, volume and timing.
AISI’s 2025 sandboxing work separates isolation into tooling, host and network controls, and recommends choosing the strength of isolation according to the evaluation risk. Its toolkit is aimed at safe testing rather than proof of production safety, but the principle transfers well to enterprise deployment. See AISI’s Inspect Sandboxing Toolkit.
Test failure modes, not just task completion
An agent evaluation should measure both useful completion and unacceptable behaviour. Include malformed inputs, ambiguous requests, stale data, tool outages, duplicate events, malicious documents, poisoned memory, prompt injection, excessive retries, privilege escalation and partial downstream failure.
Test tool contracts, orchestration and policy rules, identity and authorisation, end-to-end business outcomes, adversarial scenarios, and recovery and termination.
Record the model version, prompts, tools, permissions, test budget and environment; results from one model or harness are not a general assessment of autonomous AI.
AISI reported in 2026 that fixed-budget evaluations can underestimate agent capability when test-time compute is increased. That makes evaluation budgets a material variable. See AISI’s analysis of test-time compute. NIST’s AI Risk Management Framework remains a useful governance structure for assigning roles, documenting system boundaries and measuring controls, although its 2024 generative AI profile predates many current agent architectures. See the NIST AI RMF resources.
What we are watching
- Whether agent identity standards mature beyond user-delegated OAuth to support workload identity, delegation chains and reliable revocation.
- Whether MCP and similar protocols develop stronger mechanisms for server provenance, tool integrity and policy enforcement.
- Independent, sustained production evidence about approval fatigue, incident rates and recovery effectiveness.
- Evaluation methods that test long-running agents, adaptive attackers, memory poisoning and cross-system tool chains.
- Evidence that runtime containment remains effective when agents receive more tools, more context and larger execution budgets.
- Whether organisations can maintain an auditable inventory of agents, tools, permissions, prompts, models and downstream effects as these systems multiply.