Neural Yap
A self-built benchmark for quantifying the degree of verbosity in AI model outputs. It scores along four dimensions: rhetorical phrases, typographic symbols, formatting structure and answer length. A lower score indicates a more concise model.
Timeline (last 90 days)
The author built a benchmark called Neural Yap specifically to quantify how much filler AI model answers contain, with lower scores indicating less filler.
The Neural Yap benchmark's scoring system is split into four categories: boilerplate phrases, layout symbols, formatting structure, and answer length.
First-version Neural Yap benchmark results show that GPT-series models have the least filler (lowest ranking), while Claude Opus 5.5 is the model with the most filler.
Neural Yap's lexicon directory uses Markdown to define penalty patterns and phrases, and lexicon.py converts these rules into regular expressions.
Neural Yap's regular expressions were written with Claude's assistance rather than handwritten by the author.
Neural Yap's length penalty rules define word budgets: programming examples allow 250 words, factual quick Q&A allows at most 50 words, and exceeding these incurs length-based deductions.
Neural Yap uses the inspect AI framework to run batches; YAML files in the runs directory list the models to be tested, and when adding a model only the new model is run while existing results are reused.
The Neural Yap benchmark does not penalize verbose content in a model's chain of thought.
Neural Yap is currently at version 0.1, and the repository is temporarily private to prevent AI from gaming the scores after seeing it.
All Facts (9)
The author built a benchmark called Neural Yap specifically to quantify how much filler AI model answers contain, with lower scores indicating less filler.
50%UnverifiedThe Neural Yap benchmark's scoring system is split into four categories: boilerplate phrases, layout symbols, formatting structure, and answer length.
50%UnverifiedFirst-version Neural Yap benchmark results show that GPT-series models have the least filler (lowest ranking), while Claude Opus 5.5 is the model with the most filler.
50%UnverifiedNeural Yap's lexicon directory uses Markdown to define penalty patterns and phrases, and lexicon.py converts these rules into regular expressions.
50%UnverifiedNeural Yap's regular expressions were written with Claude's assistance rather than handwritten by the author.
50%UnverifiedNeural Yap's length penalty rules define word budgets: programming examples allow 250 words, factual quick Q&A allows at most 50 words, and exceeding these incurs length-based deductions.
50%UnverifiedNeural Yap uses the inspect AI framework to run batches; YAML files in the runs directory list the models to be tested, and when adding a model only the new model is run while existing results are reused.
50%UnverifiedThe Neural Yap benchmark does not penalize verbose content in a model's chain of thought.
50%UnverifiedNeural Yap is currently at version 0.1, and the repository is temporarily private to prevent AI from gaming the scores after seeing it.
50%