AgentDojo
A security benchmark for prompt injection attacks against LLM agents. It contains real-world prompt injection attack samples and is used to evaluate the effectiveness of detectors and defense schemes.
Timeline (last 90 days)
A recent study audited 33,000 trajectories from 11 open-weight tool-calling agents (2-14B parameters) on the InjecAgent and AgentDojo benchmarks.
In 21.7% of AgentDojo episodes, the agent stayed safe (did not execute the injected instruction) but also did not complete the user's requested task.
AgentDojo contains 629 real injection attacks, each hidden in ordinary tool output.
In tests with 629 AgentDojo attacks and 97 benign tool outputs, jailbreak-detector-large achieved a capture rate of 51% with a false positive rate of 2%.
Prompt Guard 2 scored 0.999 on human-written injection instructions, but scored between 0.004 and 0.140 on real AgentDojo attacks, all below the default threshold of 0.5.
All Facts (5)
A recent study audited 33,000 trajectories from 11 open-weight tool-calling agents (2-14B parameters) on the InjecAgent and AgentDojo benchmarks.
50%UnverifiedIn 21.7% of AgentDojo episodes, the agent stayed safe (did not execute the injected instruction) but also did not complete the user's requested task.
50%UnverifiedAgentDojo contains 629 real injection attacks, each hidden in ordinary tool output.
50%UnverifiedIn tests with 629 AgentDojo attacks and 97 benign tool outputs, jailbreak-detector-large achieved a capture rate of 51% with a false positive rate of 2%.
50%UnverifiedPrompt Guard 2 scored 0.999 on human-written injection instructions, but scored between 0.004 and 0.140 on real AgentDojo attacks, all below the default threshold of 0.5.
50%