Table of Contents
In this Help Net Security interview, Rohan Sen, Principal, Cyber, Data, and Tech Risk, PwC US, discusses how organizations can design autonomous AI agents with strong governance from day one. As AI becomes more embedded in business ecosystems, overlooking agent-level security can open the door to reputational, operational, and compliance risks.
When designing autonomous AI agents, what are the most critical governance and risk control mechanisms that must be built in from the outset? Can you give examples of what good vs. weak implementation looks like in practice?
The most critical step is treating autonomous agents as digital identities with real-world impact, requiring the same level of governance as human users. This includes enforcing least-privilege access, assigning unique credentials, and logging all actions for full auditability from the outset.
Strong implementations are built in layered safeguards: tightly scoped permissions, sandboxed environments, strict escalation paths, and real-time monitoring. These controls allow organizations to oversee, contain, and roll back agent activity if needed.
Weak implementations treat agents as simple automation, granting broad access with no ownership or oversight. Without input validation or usage controls, agents become vulnerable to prompt injection and adversarial manipulation, turning them into security blind spots.
Ultimately, the difference comes down to how seriously the organization treats the agent’s identity, authority, and risk. Well-governed agents are supervised and constrained, while poorly governed ones are accidents waiting to happen.
From your perspective, what operational or reputational risks are most likely to emerge from poorly governed autonomous agents in the next 12–24 months?
Over the next 12–24 months, the most immediate risks from autonomous agents include:
- Impersonation and brand damage: Malicious actors can exploit unsecured agents to impersonate executives, employees, or customer service reps – leading to phishing, fraud, and reputational harm.
- Unintended business actions: Over-permissioned agents with control over financial workflows, vendor systems, or sensitive data can initiate irreversible actions without human oversight. One bad output can trigger serious consequences.
- Regulatory and compliance exposure: Agents handling personal or sensitive data may inadvertently violate privacy or disclosure rules. Many lack built-in explainability, making it difficult to demonstrate compliance or investigate harmful outcomes. This gap increases the risk of audits, fines, or legal action.
- Incident response gaps: Many organizations aren’t equipped to detect, isolate, or remediate misbehaving agents in real time. Without dedicated frameworks and containment tools, response efforts may be too slow to prevent damage.
What concrete steps should business and IT security leaders take today to build resilience into their AI ecosystems, particularly for agents that make decisions or take autonomous actions?
Steps for business and IT security leaders to build resilience into AI ecosystems include:
- Treat agents as actors, not tools: View autonomous agents as entities with real agency and the potential to impact systems, decisions, and data – requiring the same governance and oversight as high-privilege human users.
- Implement strong foundational controls before deployment: Enforce least-privilege access, assign unique credentials, use robust authentication, and make sure all agent actions are immutably logged before granting autonomy.
- Conduct regular red teaming and stress testing: Simulate scenarios like prompt injection, adversarial inputs, and logic traps to uncover vulnerabilities and validate whether existing controls hold up under real-world conditions.
- Classify agents based on risk and autonomy: Develop a tiered system to differentiate between low-risk and high-risk agents, applying stronger safeguards such as human-in-the-loop reviews, real-time monitoring, or auto-shutdown for higher-risk functions.
- Foster organization-wide awareness and preparedness: Make sure teams across security, HR, finance, and operations understand how agents function, how they can be exploited, and what to do when something goes wrong.
In the event that an autonomous agent begins to act outside of intended bounds, due to prompt injection, emergent behavior, or adversarial input, what should a well-prepared incident response plan look like?
Incident response for AI agents starts with preparation. This means maintaining a registry of all deployed agents by detailing what systems they touch, what permissions they hold, and who owns them. Additionally, set behavioral baselines and detection thresholds. Monitor for deviations like unexpected data access, unusual volumes of output, or unfamiliar decision paths.
Leaders should also confirm that predefined kill switches, access that can be quickly turned off, or isolation tools are in place and regularly tested. These allow for quick containment without requiring a full system shutdown.
A robust response must also involve legal, compliance, communications, and leadership teams. If an incident affects customers, triggers regulatory thresholds, or generates media attention, coordination across teams is essential. Organizations need incident response plans tailored to AI-specific risks – tested, refined, and understood by everyone involved.
When evaluating AI vendors, prioritize questions about governance, security, and risk control. Start by asking how agents are authenticated and authorized. Look for role-based access, scoped permissions, and complete audit trails. If the vendor can’t explain this, it’s a red flag.
Next, ask what safeguards are in place to prevent unsafe decisions. Strong responses will reference policy enforcement, sandboxing, escalation logic, and real-time overrides. Also ask about adversarial testing, specifically how the vendor defends against prompt injection and emergent behavior.
Visibility is also critical. Confirm whether you’ll have access to tamper-proof logs that track every action, because without this, you could lose control when something goes wrong. Finally, ask which governance frameworks the vendor follows. If they can’t describe their risk posture, you’re likely inheriting risks they haven’t addressed.
