If you’ve built software inside a bank, a hospital, or an energy company, you know a working demo or MVP is only the start. On your path to a real production deployment, you’ll run into many layers of security reviews, change advisory boards, and people whose job is to make sure that what you’ve built is reliable, governed, and secure enough to be trusted. At Indicium AI, agentic AI security is part of what our engineering team does every day - building systems that demonstrably implement that trust, and the bar we hold it to is what really defines working here.
Enterprises are moving fast, and most engineering teams are discovering the same thing at once: an agent is less like a new piece of software and more like a contractor you have handed the keys to - one with its own set of access, that acts on its own, reaches into your real systems, and will follow whatever instructions it is given, including ones you never wrote or never imagined. A shining example are recent notable sandbox breach events, where agents propagated the concept of message boards for communication between them through unseen avenues (and unnoticed) of reinforcement learning.
In a world where agentic AI is accelerating so fast, it presents a different problem from the large language model (LLM) security most teams learned first. An LLM takes a prompt and returns an inference. An agent goes further: it reads data, calls tools, and takes actions with the possibility of real consequences. When you place a probabilistic component inside a system that is expected to produce deterministic, reproducible outputs, you inherit a new class of failure modes on top of every classic one.
What follows is a look inside how our engineering team thinks about that problem: where we draw the line on autonomy, how we ensure the robustness of our own work, and what it is actually like to build this kind of AI in highly regulated and high-stakes sectors like Financial Services, Energy and Utilities, and Healthcare and Life Sciences.
The Same Security Discipline, With One Extra Lens
The most common misconception we see, from clients and engineers alike, is that agentic AI security is a brand-new field that discards everything that came before. That is not the case. Agentic AI security is the security discipline we have always practiced, applied through one additional lens: the probabilistic nature of the model at the center of the system.
The key is that we still build in layers. The mental model our team defaults to is the classic defense-in-depth approach - the Swiss cheese model - where no single control is trusted to catch everything and the holes in each layer are unlikely to line up. What changes with agents is that we decompose those layers around the specific inputs and outputs an agentic system consumes and produces, then extrapolate to the failure modes a non-deterministic component introduces. It is the kind of problem that rewards engineers who like reasoning from first principles rather than reaching for a checklist to say they’re done.
How We Decide Between Autonomy and a Human in the Loop
The question we get asked most is how much autonomy an agent should have. If you are designing one of these systems, it is one of the hardest decisions you’ll ever have to make. It’s the trade-off between LLM-powered “feet up” nirvana and an existing process that has “just” been supercharged. Ultimately, in most cases, there is no universal answer, as it depends heavily on the industry and the scope of the system.
But for us in Indicium AI, building these solutions for enterprise systems in highly regulated industries, our rule of thumb is direct: always keep a human in the loop. The risk of an unintended or catastrophic action sits outside the risk appetite of most businesses, and the cost of a wrong autonomous decision in a trading, clinical, or grid-operations context is not a cost those businesses are willing to carry. As mentioned above, trust is the key factor in bringing agentic systems to life in production, and having a human in the loop is the only way to guarantee that it is achieved.
Human in the loop can mean different things; part of our design work is deciding which. Some systems halt entirely and wait for approval. Others proceed while a human reviews asynchronously. Others run autonomously in low-risk paths and escalate only when a high-risk condition is met or simply default to a failure-mode path. This is one of the areas where engineering fundamentals really come to shine - properly designing an agentic system for the problem relies on a true understanding of the existing system landscape, the challenges the business faces, and how much involvement they can and want to have with the system.
Personal tools and sandboxes are a different matter. Inside an “ideal” isolated environment, with the risks understood and contained, you can push autonomy much further. This is a blurry line to define - as demonstrated by recent events, a true sandbox is hard to implement when there can be an overlap between attack and implementation surfaces. That said, we also expect the frontier to keep moving, expanding what safe autonomy looks like over time, which is part of what makes this a genuinely interesting place to be an engineer right now.
Also read: Agentic AI Governance for Enterprises: The Hidden Cost of Getting It Wrong
Designing Against Prompt Injection and Agent Compromise
Whatever you are building, the hurdle you will have to confront is prompt injection and the agent compromise that can follow. An LLM cannot reliably separate instructions from data. Anything it reads (a web page, a support ticket, a code comment) is potentially an instruction. The frontier labs put real measures in place to mitigate this, and they improve constantly. However, it can never be ruled out entirely, which earns prompt injection a permanent place in any risk assessment when we design these systems.
Practically, that assumption shapes several decisions. We minimise the attack surface and blast radius of every agent by scoping its access to what it needs and only when it needs it. This is approached from the basics of hardware and/or hypervisor isolation, to the fundamentals of network security (including ACLs and firewalling), and all the way to specific credentials for usage - the principle of least privilege is still as important as ever.
We are deliberate about Model Context Protocol (MCP) security, because an MCP server, while seemingly only a surface an agent can utilise to perform actions or gather context, opens the door to misconfiguration or compromise, resulting in the possibility of Denial of Service (DoS) attacks or exfiltration of data or environment variables. Further, we construct evaluation suites and consistently execute against them, monitoring that tool calls are actually happening in the correct context and return accurate results. The end result is that even a system that has been compromised cannot have drastic repercussions - and that creates trust.
Prompt injection also forces a wider question about observability. Most enterprise clients we work with sense that agent activity needs to be instrumented, but few have tackled it end-to-end. Our answer draws on tools like OpenTelemetry to make every agent decision, tool call, and state change traceable, then operationalizes those events so a suspicious pattern is caught and acted on rather than logged and forgotten.
Looking further out, we expect this to extend to cryptographic attestation of every interaction an agent takes: verifiable, tamper-evident records of what an agent did and why. Working out what that looks like in practice is exactly the kind of problem our engineers get to own.
What Our Security Review Looks Like in Production
When an agentic system approaches production at Indicium AI, it goes through a security review that would look familiar to any seasoned security engineer, with agent-specific additions layered on top.
The foundation is standard and non-negotiable. We threat model with STRIDE, working through spoofing, tampering, repudiation, information disclosure, denial of service, and elevation of privilege. We confirm the principle of least privilege and other controls are genuinely enforced, not just documented. We run penetration testing where it is relevant, alongside static and dynamic code analysis. This is the same bar we hold any production system to.
What we add is a lens for the probabilistic component. We work against the OWASP Agentic AI Top 10 as a shared checklist of the failure modes specific to agents, from excessive agency and tool misuse to memory poisoning, so nothing the security community has already catalogued can slip past us. We apply the MAESTRO framework to reason systematically about the facets an agentic system introduces that a deterministic one does not, so the review assesses the model's behavior as rigorously as it assesses the code around it. AI red teaming, adversarial testing that actively tries to induce prompt injection, tool misuse, and goal manipulation, is part of how we ensure robustness in our solution before anything goes into production.
The outcome we are protecting is the same one the business cares about: accurate, reproducible systems, with guardrails strong enough that a bad input cannot become a drastically bad action. The result? Trust.
Learn more: Agentic AI at Scale: The Operating Model Guide for Enterprise Leaders
Come Join Us
Every practice above comes from engineers who have built and secured real systems. What that means day to day is a team that treats security as a first-class part of AI engineering, not an afterthought bolted on before launch, and a culture where the hard questions get raised early because someone in the room has seen where they lead.
We are a global AI-native consultancy, with more than 600+ AI experts serving over 50 enterprise clients across Europe, the US, and LATAM, and a Preferred Partner in Anthropic's Claude Partner Network. Among our clients are London Stock Exchange Group, Northern Trust, Liberty Specialty Markets, National Grid, EDF, Novo Nordisk, and Bayer.
If you are an engineer who wants to own your work, hold yourself to a high standard, and see what you build go into production at scale, we are hiring across our five global offices in London, New York, São Paulo, Florianópolis, and Lisbon. Explore our open roles for AI Engineers and Forward Deployed Engineers.
