Training-Free Prompt Compression: A Deterministic Lexical Pipeline That Runs on CPU

The Cost Problem of Prompt Bloat in LLMs

As techniques like Chain-of-Thought reasoning and In-Context Learning become mainstream, real-world prompts routinely balloon past thousands of tokens. Every bit of bloat translates directly into higher inference costs and latency — a very real expense for teams using token-billed commercial APIs or running their own inference clusters.

How to compress prompts without sacrificing output quality has become an engineering problem worth serious, systematic study. Leading compression methods like LLMLingua (Jiang et al., 2023) and Selective Context (Li et al., 2023) do achieve strong compression ratios, but they come with two significant trade-offs: they require auxiliary language models to decide which tokens to drop, and their results are non-deterministic — the same input can produce different compressed outputs across runs.

A new arXiv paper flips this on its head: what if you used no training, no auxiliary models, and nothing beyond classic lexical-level NLP processing? How far can a pure-CPU pipeline compress a prompt without meaningfully degrading output quality?

Eleven Toggleable Lexical Transformations

At the heart of the paper is a configurable processing pipeline made up of eleven independently toggleable lexical transformations:

• Stopword removal
• Filler-phrase deletion
• Contraction and abbreviation substitution
• POS-based pruning
• Lemmatization
• Synonym shortening via WordNet
• Named-entity preservation

The elegance of this design lies in its composability and determinism. Every transformation follows fixed rule-based logic — no randomness, no GPU required, runs on any ordinary CPU. This stands in sharp contrast to learning-based compression methods that rely on LLM scoring, whose black-box nature and resource demands are precisely what this work aims to avoid.

Named-entity preservation is particularly critical. During compression, names, locations, and other proper nouns carrying core information are explicitly protected, preventing aggressive token removal from stripping away key semantics.

Large-Scale Empirical Study: 1,242 Prompts and 18,630 Completion Pairs

To evaluate different pipeline configurations rigorously, the authors constructed a solid evaluation framework. They sampled 1,242 English-only prompts from six data sources — Dolly-15k, LMSYS-Chat-1M, WildChat-1M, MMLU, GSM8K, and HellaSwag — covering eleven automatically derived task categories.

Fifteen pipeline configurations were evaluated, ultimately generating 18,630 paired completions from GPT-4o-mini. Output fidelity was measured not by a single metric but across multiple dimensions: BLEU, ROUGE-1/2/L, BERTScore-F1, and SentenceBERT cosine similarity — covering everything from surface-level overlap to semantic similarity.

This multi-metric, multi-task, large-sample design ensures the findings characterize the overall trade-off between compression intensity and output quality, rather than cherry-picked anecdotes.

The Pareto Frontier of Compression vs. Fidelity

The experimental results yield several practically useful numbers:

• Most aggressive configuration: Average token reduction of 40.3% (SD 9.2), with a BERTScore-F1 of 0.876 relative to the original prompt's output.
• Stopwords-only configuration: Token reduction of 29.6%, with BERTScore-F1 remaining at a high 0.913.

In other words, simply removing stopwords — the most conservative operation — trims nearly 30% of tokens with minimal semantic fidelity loss. Pushing for higher compression ratios requires accepting some quality degradation — this is the Pareto frontier between compression and faithfulness.

The paper further maps this frontier by task category, revealing an important differentiating finding: different tasks tolerate compression very differently.

Commonsense Reasoning Is the Systematic Failure Point Under Aggressive Compression

The most striking finding: under aggressive compression configurations, commonsense reasoning tasks exhibit systematic failure patterns.

This has strong practical implications. Components like stopwords and filler phrases — which look like "redundant" noise — often carry implicit logical relationships and contextual cues in commonsense reasoning scenarios. Aggressively removing them disrupts the model's correct understanding of the problem. By contrast, factual question answering or structured output tasks are far more tolerant of compression.

This is a clear warning to engineers: prompt compression should not be a one-size-fits-all strategy. Task type should dictate the appropriate compression intensity. For reasoning-heavy tasks, conservative stopword removal may be the safe boundary; for information retrieval or formatted-output tasks, more aggressive compression is viable.

Implications for Production Deployment

The value of this work is not in pushing compression ratios to new highs — it's in redefining the feasible boundary under the constraints of "no training, deterministic, CPU-runnable." For teams looking to reduce inference costs without adding extra model dependencies, this lexical compression pipeline offers an interpretable, reproducible, zero-additional-compute baseline.

The authors have released all code, prompts, and per-cell results to support replication, leaving ample room for follow-on research and engineering adaptation. At a time when learning-based compression methods grow ever more complex, this return to classical NLP — with its emphasis on determinism and transparency — offers a pragmatic alternative for real-world deployment.