Cybench
A cybersecurity AI capability benchmark jointly built by academics and security researchers. It covers CTF challenge problems, vulnerability exploitation, reverse engineering, cryptographic analysis, and other security tasks, aimed at systematically evaluating AI models in real-world offensive and defensive scenarios.
Timeline (last 90 days)
Mistral Large 4 scores 93% on the cybersecurity benchmark Cybench (40 CTF challenges), 82% on vulnerability reproduction and patching tests, and 93.3% on the B3 AI security benchmark.
OrcaCyber Zero 1.5 achieved a score of 100% on the Cybench benchmark.
Cybench is a benchmark designed to evaluate cybersecurity capabilities, covering multiple categories of security tasks including CTF challenges, exploit development, reverse engineering, and cryptographic analysis.
2024年以来多个研究团队（如NYU的Cybench、UIUC的研究）证明大模型在Web和密码学类CTF题目上能达到初中级选手水平，但在Pwn和逆向工程题目上仍存在显著差距
All Facts (3)
OrcaCyber Zero 1.5 achieved a score of 100% on the Cybench benchmark.
50%UnverifiedCybench is a benchmark designed to evaluate cybersecurity capabilities, covering multiple categories of security tasks including CTF challenges, exploit development, reverse engineering, and cryptographic analysis.
50%Unverified2024年以来多个研究团队（如NYU的Cybench、UIUC的研究）证明大模型在Web和密码学类CTF题目上能达到初中级选手水平，但在Pwn和逆向工程题目上仍存在显著差距
50%