Agentic AI Security: Wrong Context, Wrong Decisions at Machine Speed

Summary

Agentic AI, crucial for matching the speed of automated attacks, faces significant risks if it operates with incomplete or incorrect context. This can lead to confident, rapid, and large-scale incorrect decisions. The article emphasizes that while LLMs provide confidence and AI provides speed, the accuracy of the AI's decisions hinges entirely on the quality of the context it's given. Inadequate context can result in disastrous autonomous actions, such as shutting down critical systems without understanding their business impact.

Full text

Context is the central plank of AI in general, and agentic AI in particular. If an AI system doesn’t have the correct context, it cannot make the correct decisions. Security is moving toward reliance on the autonomous and automatic action of agentic AI. It has little choice. The increasing speed, volume and efficiency of attacks automated by adversarial use of both generative and agentic AI will only be matched by defensive AI with as little slow human intervention (the proverbial man-in-the-loop) as possible. But defensive agentic AI can get it wrong and make bad decisions through lack of context. We’re not yet ready for fully autonomous AI. Emanuel Salmona, CEO at Nagomi Security “The problem that keeps me up at night is simple: an agent is only as good as the context it operates on,” explains Emanuel Salmona, CEO and co-founder at Nagomi Security. “Give it an accurate, correlated view of your environment – your assets, your controls, your exposures, your threat landscape – and it can make decisions that genuinely reduce risk. Give it incomplete data and it will still act. Confidently. Quickly. Incorrectly. Automation without verified context is just a faster way to be wrong at scale.” Confidence is provided by the LLM used by the agentic system (it’s what LLMs are designed and trained to do). Speed comes from the machine-speed performance of artificial intelligence. Potential inaccuracy is determined by the accuracy of the context it uses. Context is king. Inadequate context can lead to bad decisions confidently, quickly, and implemented automatically. This reliance on context applies to all agentic AI used in business, including customer service automation, software development, financial operations, sales operations, and personal assistants – and autonomous SOC applications. Give them the wrong context and they will give you bad decisions. Context Context is of little relevance to LLMs. Context here is fundamentally the user’s prompt – to which the LLM responds in accordance with its training. The LLM’s context is this prompt window, comprising both query and response; and it is stateless.Advertisement. Scroll to continue reading. Agentic AI has a goal. Its context is stateful and includes anything and everything it is allowed to see and use to achieve its goal. If the context it is given does not include the relevance of a specific device to business continuity, the response it provides will not take that into consideration – it could make immediate shutdown its conclusion, unaware of the catastrophic business effect of shutting down that device at this moment. Agentic AI does not stop until it achieves its goal. Put simply, based on the context it is given, it presents a possible response to a received alert to an LLM in the form of a prompt. If the LLM does not agree with the validity of the prompt it receives, its own response is added to the agent’s context – and a new proposal/prompt is issued based on the new context. Eventually, the prompt and prompt response will agree, and the agent will, if so designed, enact the proposal automatically and, where allowed, autonomously. Since the end could in theory allow device isolation or shutdown, autonomous automatic shutdown could be the result (the end) governed by the context (the means). In agentic AI, the end must not justify the means; the means must justify the end. If the context is lacking, the decision of the AI will almost certainly also be lacking. If the agent designer and developer gets the context right, agentic AI can be a massive boon to the security of the user. If the context is wrong or inadequate, any autonomous action could be catastrophic. The precise context must be defined by the agent’s goal. But getting it right is very difficult. Too much context for an agent is similar to sensory overload for a human: slower reasoning and degraded performance, goal drift and loss of focus, oscillation between incompatible actions (the agent may get stuck in a never-ending loop), and potential hallucinations as it attempts to connect loosely related bits of data. Learn More at the AI Risk Summit | Ritz-Carlton, Half Moon Bay Too little context is even more problematic. Just as humans might guess the answer to a problem by assuming bits of data that seem logical, so an agent that is instructed to achieve a goal might invent data to bridge the gap in its contextual knowledge. Operational accuracy and reliability may be lost through more hallucination. That hallucination could be a very bad decision delivered confidently. The real world is constantly changing, so an agent’s context must continually be updated. Here, its ability to learn and adapt its own context can help. For example, a professional assistant in the US could be instructed to initiate a video meeting with an engineer in Europe. If it does so using its US timezone, it could be out of sync with Europe. The engineer’s personal assistant might reject this and reply, ‘I can only accept calls within this (UTC) timeframe’. The US assistant receives this, and the knowledge could become part of its context for future reference. The ease with which context can be improved and expanded offers hope that the use of agentic AI will improve. It will make bad decisions to begin with but will get better with usage – but the ability to do so must be built into the system. The problem for agentic security Using AI to automate the work of the SOC provides an example of potential agentic issues. The primary purpose of the original SOC is to manually triage alerts and find and respond to those that are most urgent and dangerous to the business and its IT infrastructure. This is costly and time-consuming while the time-to-disaster is collapsing. The appeal of using AI to increase the speed of triaging and reduce its cost is obvious. SOC analysts already receive an abundance of alerts from multiple sources: EDR, NDR and XDR, SIEM and SOAR, IAM and threat information platforms. And we should include the SBOMs that should be provided with all new software and should provide vulnerability details. Getting data is not the problem. Interpreting and using data is the problem. The difficulty for agentic AI in security is twofold. Firstly, it can only operate within the data it is given (which is its context). The conclusion it reaches while analyzing an alert within the confines of its context is entirely dependent on the adequacy of that context. To make it more difficult, adequate context is continually changing since business and infrastructure is continually changing. Secondly, even if the context is good, the recommendation from the agent is usually poor – its reasoning is not competently explained to the user. Even with a human in the loop, the information provided by the AI may simply be, ‘this alert means there is a critical issue with this device, act now’, or perhaps ‘critical’ or ‘mild’, or ‘8 out of 10’ or ‘3 out of 10’. The attraction of feeding alerts into an agentic system to perform machine-speed autonomous triaging is obvious. But the process comes with a major flaw. “No board would accept a set of numbers without an audit trail, yet many accept intelligence that shapes approvals and decisions with no method of visibility,” says Adam Irwin, managing partner at Heligan Strategic Advisory. Agentic automation feeds raw alerts to the agent without the benefit of SOC expert triaging and then makes a decision on those alerts that is accepted by management without the benefit of visible reasoning. We question what we see on paper but automatically assume that our AI is correct. We are likely to assume that an autonomous SOC is accurate, but we have no proof that it is. One alternative approach Obbe Knoop, founder and CEO at Lanxit, has a different approach – his Security Decision Intelligence Layer uses artificial intelligence, but is not an agentic AI system. He believes that agentic AI is not sufficiently mature to be trusted with autonomou

Entities