Anyone Using Agentic AI Needs to Understand Toxic Flows

1 month ago 7

finger touching a ring of light reflecting data and technology

Source: sdecoret via Adobe Stock Photo

Today's business elite is breathless for agentic AI possibilities, as CEOs grasp AI as an efficiency lifeline. Risks of functional failures aside — and they're most surely a big elephant in the room — security researchers are concerned about the emerging cyber resilience risks that all of these agentic deployments add to the risk register.

Toxic flows are one of the emerging classes of agentic AI risks that researchers say need to be on the radars of executives, engineers, and security people alike. Flows between AI agents, IT tools, and enterprise software are beset by a risky combination of exposure to untrusted input, excessive permissions, access to sensitive data, and external connections that can be used by attackers to steal data.

These toxic flows could be at the heart of what many believe will be the path to better agentic AI risk management, provided the industry can implement controls for them.

How Agentic AI Risks are Different

In many ways, AI systems are similar to any other software: vulnerable to flaws in code, misconfigurations, and broken authentication. Agentic AI adds a new wrinkle to the tapestry of threats posed by the modern software stack, says Luca Beurer-Kellner, co-founder of Invariant Labs. The nondeterministic nature of agentic behavior makes it really hard to predict risky behavior in advance.

"The whole premise of agentic AI systems is that they can do things for you without the developers having to anticipate them. That's an amazing property and makes it promising, but it's hard to anticipate ahead of time what kinds of risks we are exposing ourselves to," says Beurer-Kellner. "That's different from traditional software, because it is typically code and algorithms and processes that are well known ahead of time."

He has been heading up research efforts to drive awareness around toxic flows by Snyk, which recently acquired Invariant.

When AI agents with inherently unpredictable behavior are connected to some of the most sensitive systems in the enterprise, be they customer databases, financial systems, or development platforms, big issues start to arise. This is the role of model context protocol (MCP) servers, connectors that help developers sync up data sources with generative AI-powered tools. They are being called the "USB-C port" for AI apps because they make it possible for applications to communicate seamlessly with data sources and other tools.

Make no mistake, MCP is going to have to connect a lot of sensitive systems to AI agents, as developers march to CEO orders to make accounting business functions more efficient through agentic AI. This is what it will take to power the most valuable use cases, but it also drastically increases the risks of prompt injections, hallucinations, and other exploitable flaws in LLMs.

"Whenever there's a slip-up, whenever there's a hallucination, whenever there's an attacker, the consequences are much more severe," Beurer-Kellner says. "It's no longer just happening in a chat window and just a funny hallucination. It could be like an extra zero on a bank transaction. These are the kind of mistakes you don't want to happen."

Given the business mandates, though, security professionals can't say no to agentic AI connecting to sensitive systems — but they can start to help the business structure connections to control risk.

The Lethal Trifecta and AI Kill Chain

Some of the riskiest agentic AI deployments occur when systems are combined in a way to invoke what software engineering luminary Simon Willison recently called "the lethal trifecta for AI agents."

This is when AI agents are designed to combine access to private data, exposure to untrusted content and the ability to communicate externally in a way that can be used to steal the data.

"If your agent combines these three features, an attacker can easily trick it into accessing your private data and sending it to that attacker," Willison wrote in his blog.

Unfortunately, all too many AI agents today are prone to the lethal trifecta. In recent research on toxic flow analysis, Beurer-Kellner and his Invariant Labs team pointed to the trifecta as a prime breeding ground for toxic flows. They demonstrated the recent GitHub MCP exploit as a classic example of such a flow in action, showing how attackers could attack fully trusted tools by using untrusted information to exfiltrate data.

And this is no isolated research. Security researcher Johan Rehberger demonstrated that no popular AI tool or agent is immune to a plethora of issues through his Month of AI Bugs vulnerability publishing blitz in August. Rehberger dropped dozens of consequential vulnerabilities across just about every major platform, plenty of them made possible by the lethal trifecta. Over the course of the month, he also put forward his own deadly trio of agentic AI problems, one he calls the AI Kill Chain.

Several of his discovered vulnerabilities were exploitable via a three-step process: Prompt injection, confused deputy problems and automatic tool invocation.

The connection between the AI Kill Chain and the Lethal Trifecta shows that many of agentic AI's exploits will rest on the ability to pick apart the fabric that weaves together agentic prompts and connections with sensitive data.

"Giving an AI agent access to private data isn't the risky part. It's what you combine it with," summed up KPMG's agentic AI Agent Engineering Leader Justin O'Connor.

Toxic Flow Analysis Enters the Chat

To any security veteran, toxic combinations shouldn't be a new concept. It's a longstanding issue in identity management, which has had to develop controls to prevent problematic access combinations such as finance users creating new vendors and approving payments at the same time or IT admins managing user access and also deleting system logs.

Toxic flows in agentic AI are often also tied up in privilege weaknesses, but they take dangerous mash-ups to new heights of risk and complexity.

Beurer-Kellner at Snyk's Invariant Labs hopes that his team can help organizations start to surface these issues through a framework they've developed, designed to analyze AI-powered apps for potential toxic flows. Toxic Flow Analysis is now being delivered through Snyk's open source MCP scan tool.

The analysis models the flow of data and tool usage within an agent system to look for toxic combinations. This is different from prompt security solutions, which look solely at the secure implementation of agent systems. The idea behind toxic flow analysis is to create a flow graph of an agent system and model all of the potential sequences of tool uses, together with other properties like the level of trust, the sensitivity of the system or data it handles, and whether the tool could be used as an exfiltration sink.

"The key word in 'toxic flow analysis' is actually 'flow,'" says Danny Allan, CTO of Snyk. "If you don't understand the flow by definition, you're not going to get things like authorization right. Because the insecurities happen at the boundaries of these different components that may have security built into them and their own components, but not across the components."

Read Entire Article