Driftproof
An open-source evaluation tool used to conduct controlled experiments and assess the effectiveness of AI agent skills.
Timeline (last 90 days)
Driftproof is an open-source evaluation tool, and the author is also the tester; the conflict of interest has been disclosed in the article.
The rule for judging a skill as clearly helpful is: on at least one task the two score intervals (mean plus or minus one standard deviation) do not overlap and the gap is no less than 0.05, while no task clearly moves in the opposite direction; a clear decline on any one task outweighs improvements on other tasks.
The evaluation method is three plain-text tasks corresponding to each skill, with each task answered both with and without the skill, each set repeated 3 to 10 times, and each answer scored 5 times.
The tester lists three limitations: each skill has only three plain-text tasks written by the testing party; tool calls, media generation and deployment services were not tested; and there is only a single judge model.
All Facts (4)
Driftproof is an open-source evaluation tool, and the author is also the tester; the conflict of interest has been disclosed in the article.
50%UnverifiedThe rule for judging a skill as clearly helpful is: on at least one task the two score intervals (mean plus or minus one standard deviation) do not overlap and the gap is no less than 0.05, while no task clearly moves in the opposite direction; a clear decline on any one task outweighs improvements on other tasks.
50%UnverifiedThe evaluation method is three plain-text tasks corresponding to each skill, with each task answered both with and without the skill, each set repeated 3 to 10 times, and each answer scored 5 times.
50%UnverifiedThe tester lists three limitations: each skill has only three plain-text tasks written by the testing party; tool calls, media generation and deployment services were not tested; and there is only a single judge model.
50%