LiveNerf
An open-source continuous monitoring benchmark project created by community developers. It tracks whether large language models experience capability degradation over time by running benchmarks such as GPQA and SWE-bench on a daily basis and plotting capability curves.
Timeline (last 90 days)
LiveNerf was initiated by developer ninjahawk, with the code hosted in the GitHub repository ninjahawk/livenerf.
As of the Day 8 update, the LiveNerf repository's star count was approaching 1,000, and it had made the front page of Hacker News.
At the Day 8 update, LiveNerf was approaching the stage of establishing a performance baseline.
The author of the LiveNerf project takes a neutral stance and does not operate under the assumption that "Opus 5.5 will degrade," viewing a finding of no performance degradation as equally valuable.
As of the article's publication, LiveNerf was still in the data collection phase and had not yet reached a conclusion about whether Opus 5.5 has been downgraded.
Livenerf is an open-source project for continuously monitoring whether the Claude Opus model has been nerfed (weakened/dumbed down).
The author stated that a reliable baseline will not be established until day 10, and no formal judgment will be made before day 20.
A Reddit developer launched an open-source benchmark project called LiveNerf that continuously tracks the performance of Opus 5.5 starting from the first day of the model's release.
Using Claude to select the most difficult questions may itself introduce selection bias, which is one of the methodological limitations of the LiveNerf project.
LiveNerf reruns a set of independent benchmarks every day, such as GPQA and SWE-bench, and plots the results in charts recorded in a GitHub repository.
4 more timeline events
All Facts (9)
The author of the LiveNerf project takes a neutral stance and does not operate under the assumption that "Opus 5.5 will degrade," viewing a finding of no performance degradation as equally valuable.
50%UnverifiedLiveNerf was initiated by developer ninjahawk, with the code hosted in the GitHub repository ninjahawk/livenerf.
50%UnverifiedLivenerf is an open-source project for continuously monitoring whether the Claude Opus model has been nerfed (weakened/dumbed down).
50%UnverifiedLiveNerf uses the previous generation Opus 5 as the comparison baseline.
50%UnverifiedUsing Claude to select the most difficult questions may itself introduce selection bias, which is one of the methodological limitations of the LiveNerf project.
50%UnverifiedLiveNerf's criterion is: if Opus 5.5's score deviates by more than 7.5 points, it is treated as a potential signal of capability degradation.
50%UnverifiedLiveNerf reruns a set of independent benchmarks every day, such as GPQA and SWE-bench, and plots the results in charts recorded in a GitHub repository.
50%UnverifiedThe LiveNerf author uses Claude to filter out the subset of questions on which Opus 5.5 is most prone to errors for testing.
50%UnverifiedA Reddit developer launched an open-source benchmark project called LiveNerf that continuously tracks the performance of Opus 5.5 starting from the first day of the model's release.
50%