ProteinGeoBench
A benchmark for protein structure analysis comprising 131 protein structure tasks, divided into task families such as metadata queries and numerical computation, used to evaluate the structural querying and geometric computation capabilities of AI agents.
Timeline (last 90 days)
The ProteinGeoBench benchmark contains 131 protein structure tasks.
The researchers collected 4,249 run trajectories from four frontier models, plus 4,486 additional trials from 26 models.
Three of the four frontier models have scores densely clustered between 0.785 and 0.816, differing by only 0.031 and statistically indistinguishable from one another; the fourth model scores 0.707, and its gap from each of the first three is significantly different from zero.
Among the four frontier models, three perform better on metadata query tasks than on numerical computation tasks, and only the fourth model has a slight edge on numerical tasks.
Of the 131 tasks, only 28 are numerical computation type, yet half of the twelve tasks on which all four models failed belong to numerical computation.
The paper points out that if each task is run only once, 23.9% of bootstrap resamples would rank a different model first.
On the selected 15-task slice, removing network access causes a larger score drop than blocking the eight structural database domains.
The two comparisons—removing network access and blocking structural database domains—were done in different evaluation rounds, so their independent effects cannot be cleanly decomposed.
Replacing each model with its own command-line agent (harness) changes scores by a range from -0.071 to +0.086.
In the extended trials on 26 models, numerical computation performance rises together with non-numerical performance.
4 more timeline events
All Facts (14)
The ProteinGeoBench benchmark contains 131 protein structure tasks.
50%UnverifiedThe researchers collected 4,249 run trajectories from four frontier models, plus 4,486 additional trials from 26 models.
50%UnverifiedThree of the four frontier models have scores densely clustered between 0.785 and 0.816, differing by only 0.031 and statistically indistinguishable from one another; the fourth model scores 0.707, and its gap from each of the first three is significantly different from zero.
50%UnverifiedAmong the four frontier models, three perform better on metadata query tasks than on numerical computation tasks, and only the fourth model has a slight edge on numerical tasks.
50%UnverifiedOf the 131 tasks, only 28 are numerical computation type, yet half of the twelve tasks on which all four models failed belong to numerical computation.
50%UnverifiedThe paper points out that if each task is run only once, 23.9% of bootstrap resamples would rank a different model first.
50%UnverifiedOn the selected 15-task slice, removing network access causes a larger score drop than blocking the eight structural database domains.
50%UnverifiedThe two comparisons—removing network access and blocking structural database domains—were done in different evaluation rounds, so their independent effects cannot be cleanly decomposed.
50%UnverifiedReplacing each model with its own command-line agent (harness) changes scores by a range from -0.071 to +0.086.
50%UnverifiedIn the extended trials on 26 models, numerical computation performance rises together with non-numerical performance.
50%UnverifiedDifferent experiments used different task subsets, so it is impossible to rank the four factors—model, tool framework, access conditions, and task composition—against one another.
50%UnverifiedThe paper recommends that evaluation report results separately by task family rather than giving only a single total score.
50%UnverifiedThe paper recommends that evaluation reports explicitly state the number of runs per task, disclose the tool framework (harness) used, and disclose network and database access conditions.
50%UnverifiedWhen a benchmark's total score is diluted by a large number of metadata query tasks, a high score may mask a model's real weakness in geometric computation.
50%