Lethal Trifecta

Function approximation error can cause algorithms to become unstable or even diverge, and the risk is highest when function approximation, bootstrapping and off-policy learning occur simultaneously (the "deadly triad").

The "lethal trifecta" popularized by Simon Willison points out that when an Agent simultaneously has access to private data, exposure to untrusted content, and an outbound communication channel, its data can be tricked out of it.

The 'fatal triad' of security is: access to private data, exposure to untrusted tokens, and the existence of a data exfiltration channel.

Reliable defenses should be structural — that is, removing one leg of the fatal triad — and Willison considers the data exfiltration path to be the easiest leg to remove.

If all three capabilities must remain enabled, agents should be split: an isolated model with no tools handles untrusted text, and the agent with permissions never sees that text.

Security researcher Simon Willison named the risk pattern of simultaneously possessing all three capabilities—access to private data, exposure to untrusted text, and a channel for exfiltrating data—the 'lethal trifecta'.

Access to private data, exposure to untrusted text, and a channel for exfiltrating data are each safe in isolation; it is only when combined that they constitute a vulnerability.

Simon Willison summarized the root cause of prompt injection risk as the "fatal triad": private data, untrusted content, and external communication—when all three are present, incidents will occur, and the solution is to eliminate one of them.