Making LLMs Dumber? An Empirical Study of Performance Throttling and Believable Capability Reduction

Weak-teacher LoRA precisely lowers LLM scores, but is locally believable, globally detectable, and not portable across tasks.
This study systematically examines "believable performance throttling"—actively lowering LLM capability to a target level and checking whether its errors resemble those of a genuinely weak model. The team built the IRT-calibrated CapLadder suite (1,985 items) and proposed weak-teacher LoRA, achieving a holdout calibration error of just 1.9 percentage points on reasoning tasks, outperforming prompting, temperature, and weight noise. However, multiple-choice tasks revealed an "answer-token attractor": 97% of new errors concentrate on one letter, varying by model family, and logit intervention recovers 52%–59% of lost accuracy. Reasoning-task error-category distributions closely match a real weak model (JS=0.00), but cross-benchmark transfer shows a 14.2 percentage-point error. The conclusion: precise score control does not equal portable or believable skill reduction, and the reliability of benchmark scores as capability proxies needs reconsideration.
Introduction: When Benchmark Scores Become a Manipulable Target
In AI, benchmark scores are often treated as a direct proxy for model capability. But a reverse question is now drawing researchers' interest: can we actively lower a model's performance to a specified level? More importantly, are the failure modes of this artificial "dumbing down" similar to those of genuinely weaker models? An empirical study titled Can We Make an LLM Dumber, or Is It Just Playing Dumb? is the first to systematically explore the feasibility and limits of "believable performance throttling."
The researchers proposed three evaluation dimensions: target calibration (can the target score be hit precisely?), item difficulty structure (are errors concentrated on questions of reasonable difficulty?), and similarity of the error distribution to a genuinely weak reference model. Based on these, they built CapLadder—an IRT-calibrated test suite of 1,985 items covering math, code, multiple-choice reasoning, and tool calling.

IRT (Item Response Theory) is a statistical framework used in educational and psychological measurement. Its core assumption is that the probability of a test-taker answering an item correctly is jointly determined by the test-taker's latent ability parameter and the item's parameters such as difficulty and discrimination. In this study, IRT calibration means that each item's difficulty parameter in CapLadder is estimated via an IRT model, rather than simply ranked by subjective difficulty or pass rate. This allows the researchers to map model capability onto a continuous latent trait scale, precisely set the ability level corresponding to a "target score," and thereby select items of appropriate difficulty to assess whether the model has truly reached that level. The value of IRT lies in providing a comparable yardstick between item difficulty and model capability, giving statistical interpretability to the operation of "tuning a model to a certain score"—rather than merely hitting a target accuracy on some fixed test set.
Method: The Weak-Teacher LoRA Knob and Comparison with Other Interventions
The research team designed a "weak-teacher LoRA" method that adds a lightweight LoRA adapter to the model, "turning down" a strong model's capability to a specified level. On reasoning tasks, this method can reach unseen target levels, with a holdout-set calibration error of only 1.9 percentage points.
By contrast, prompt engineering, temperature adjustment, and weight noise all fail to simultaneously achieve comparable adjustment range and precision. This means that performance throttling via fine-tuning is far superior in controllability to inference-time heuristic interventions.
But the researchers did not stop there. They further examined the "believability" of this throttling—that is, whether the errors it produces are distributed as naturally as those of a genuinely weak model.
LoRA (Low-Rank Adaptation) is a parameter-efficient fine-tuning technique that does not modify all of the original model's weights; instead, it injects the product of two low-rank matrices as an incremental update alongside specific linear transformations in Transformer layers. During training, only these two small matrices are updated, typically amounting to just 0.01%–1% of the original model's parameters. The "weak-teacher LoRA" in this study refers to training a LoRA adapter targeting a deliberately capability-limited "weak teacher" model (or weakened signal), such that when superimposed on the strong model it pulls the strong model's output behavior down to a specified capability level. Because LoRA only changes a low-rank subspace, it is lighter and easier to switch than full-parameter fine-tuning—but this also means its adjustment capacity is constrained by the subspace, which may be one reason for the cross-benchmark transfer failures. Understanding LoRA's low-rank bottleneck helps explain why the knob calibrates well on one benchmark yet deviates substantially on another.
Key Finding: Answer-Token Attractors and Error Distributions
On multiple-choice tasks, the weak-teacher LoRA knob exposed a serious problem: it converges to an "answer-token attractor." As many as 97% of newly introduced errors concentrate on a single letter, and the attractor varies by model family—D for Qwen, A for Llama.
Through logit intervention, the researchers identified a dominant approximately constant offset. Subtracting this estimated offset on the holdout set recovers 52% to 59% of the lost accuracy and eliminates the dominant attractor. Option-permutation experiments further confirm that this bias tracks token position rather than option content—the model simply tends to output the letter at a certain position, rather than genuinely understanding the question.
This finding reveals the tension between local plausibility and global detectability: at the level of individual items, errors may look plausible; but in the aggregate distribution, anomalous patterns are clearly visible.
Logit intervention refers to directly modifying the unnormalized logit vector at the model's output layer—for example, adding or subtracting an offset vector—before applying softmax to obtain a probability distribution. By comparing the logit differences on each multiple-choice option token between the throttled model and the original strong model, the researchers estimated an approximately constant offset direction that systematically raises the logit of a certain letter token (such as D or A). Subtracting this estimated offset on the holdout set amounts to a correction of the output distribution, thereby recovering 52%–59% of the lost accuracy and eliminating the dominant attractor. The significance of this operation is that if the errors caused by throttling stem merely from an estimable constant logit offset, then the "dumbing down" process is highly mechanical in nature rather than genuinely simulating a decline in capability. Logit intervention thus serves as a diagnostic tool to distinguish between "a superficially plausible error distribution" and "an underlying systematic token bias."
Error-Category Distribution: Surface Believability on Reasoning Tasks
Interestingly, on reasoning tasks, the distribution of human-annotated error categories is almost identical to that of a genuinely weak reference model (JS divergence = 0.00), with high inter-annotator agreement (κ = 0.94). This indicates that, on reasoning tasks, the errors produced by weak-teacher LoRA are "believable" at the category level—at least from the perspective of human annotators, they are hard to distinguish from the errors of genuinely weak models.
However, tool-calling tasks behave more naturally, with the Qwen series showing a JS divergence of 0.09 on structured tool calls, indicating a smoother degradation curve. Yet the globally calibrated knob performs poorly when transferred across benchmarks—a 14.2 percentage-point error appeared on the two available benchmarks.
JS divergence (Jensen-Shannon divergence) is a symmetric metric for measuring the difference between two probability distributions, ranging from 0 to 1 (when base 2 is used), with 0 indicating identical distributions. In this study, the researchers tallied the errors produced by the throttled model into a distribution according to human-annotated error categories (such as calculation errors, logical leaps, and missing units), then compared it with the error-category distribution of a genuinely weak reference model and computed the JS divergence. A JS of 0.00 on reasoning tasks means the two distributions are nearly indistinguishable in category proportions—strong evidence of "surface believability." However, JS divergence only captures similarity at the level of category proportions and is insensitive to specific patterns within errors (such as token-position bias), so it cannot on its own prove that the throttled model is mechanistically like a weak model. A JS of 0.09 on tool-calling tasks indicates some but not large distributional difference, with relatively smoother degradation.
Implications: Precise Control Is Not the Same as Portable, Believable Capability Reduction
The core conclusion of this study is that precise score control does not imply portable or believable skill reduction. A "dumbing-down knob" well calibrated on one benchmark may fail completely on another; it can produce convincing error distributions on some tasks while exposing obvious token-position bias on others.
For the field of AI safety and evaluation, this means the reliability of benchmark scores as capability proxies warrants more careful scrutiny. If a model can be precisely "tuned" to any score, then the value of the score itself as a capability signal is weakened. At the same time, detecting whether a model has been artificially throttled may require going beyond aggregate scores to analyze error distributions and token-level patterns.
Researchers also note that believable performance throttling is more feasible on structured tasks such as tool calling, whereas on open-ended reasoning tasks—although error-category distributions match—the token-attractor problem in multiple choice reminds us that surface believability may mask underlying mechanical behavior.
Conclusion
"Making an LLM dumber" is more than a technical gimmick; it touches on the fundamental assumptions of model evaluation. CapLadder and weak-teacher LoRA provide a controlled experimental platform, but the results remind us that truly "believable capability reduction" is far more complex than turning down a knob. Locally plausible, globally detectable, and not portable across tasks—these characteristics together delineate the boundaries of current performance-throttling techniques. For building more robust evaluation systems and detection methods, this study provides an important empirical foundation.
Related articles

Programmers in the AI Era: How Deep Should You Learn Legacy Tech?
How should programmers transform under AI's impact? How deep should legacy tech be learned? Drawing on a Bilibili creator's share, this breaks down the levels of embracing AI, the shift from process-driven to project-driven learning, and how lower technical barriers affect junior and mid-level roles.

OpenAI Releases GPT-6.2 Dual Models Astra and Sol, API Price Cut by 20%
OpenAI releases GPT-6.2 Astra and Sol, with Astra's API price cut 20% to $8/$40 per million tokens and ranking second on the Artificial Analysis Intelligence Index. This article analyzes the dual-model strategy, price-cut impact, and developer selection advice.

Perplexity Computer Launches Multiagent: 16 AI Agents Cross-Validate Each Other
Perplexity launches Multiagent mode for Computer: a lead agent orchestrates up to 16 sub-agents on Claude, GPT, and Gemini, sharing findings and cross-validating work with effort above Ultra.