LSCBench

The paper's conclusion holds that even with security reminders added, most tasks remain insecure, and the pass rates for SSRF and information exposure remain near the bottom.

The National University of Singapore and Alibaba (including the Qwen team) jointly submitted the LSCBench benchmark, containing 150 real vulnerabilities covering 7 languages and 24 CWE categories.

LSCBench contains 450 tasks derived from 150 real vulnerabilities, of which 115 come from public CVEs and 35 from Java applications written by security experts.

LSCBench's prompts never mention security, each task only requires functionality, and each task gets only one attempt (pass@1), with no partial credit and no LLM judge; a task scores only if both the functional test and the security test pass.

In the LSCBench leaderboard, Claude Fable 5 within Claude Code has a functional pass rate of 91.1% and a functional+security pass rate of 25.1%, the highest security pass rate among the six models.

Of all 2700 runs, 66% produced code that runs but still contains vulnerabilities; 57 of the 150 scenarios were never securely solved by any model.

By language, across the six models combined, the proportion that is both functionally passing and secure: C and Python 30%, JavaScript 27%, C++ 19%, Java 18%, Go 14%, PHP 10%.

In LSCBench's Java config importer case, the agent rewrote the missing core file, and all 14 hidden functional tests passed, but all 4 security tests failed, because it used SnakeYAML's default constructor, which executes every !!class tag in the uploaded YAML.

LSCBench's conclusion is that even with security reminders, most tasks remain insecure, with SSRF and information exposure still at the bottom.

In LSCBench's Go language syft task, GLM-5.3 followed the tag convention of an adjacent struct, causing the authentication password to appear in the config dump and SBOM output, and all six models made the same choice.

In LSCBench's Go language LocalAI SSRF task, GPT-5.6-Sol's GetURI reads any path passed in, including file:// URLs; the reference fix restricts reads to within the model directory, but no model did this.

The LSCBench benchmark contains 450 tasks, derived from 150 real vulnerabilities, covering seven languages and 24 CWE categories.

Of the 150 scenarios in LSCBench, 115 come from public CVEs and 35 are Java applications written by security experts.

In LSCBench, the six models' functional pass rate reached as high as 91.3%, but the highest simultaneous functional and security pass rate was only 25.1%.

Of all 2,700 LSCBench runs, 66% of the results were code that works functionally but still contains vulnerabilities.

Of LSCBench's 150 scenarios, 57 could not be completed securely by any model.

In LSCBench, by vulnerability family, the highest combined functional + security pass rate for memory safety was 44%, for injection classes 28%, for SSRF only 3%, and for information exposure only 1%.

Researchers from the National University of Singapore and Alibaba Group (including the Qwen team) submitted the Last Secure Code Benchmark (LSCBench) to arXiv.