Inherited Weights: Tokenization Premium and Sovereignty Costs of Language Models

Most language models deployed outside the United States and China are fine-tuned descendants of a handful of pretrained families. A new paper argues that fine-tuning cannot alter the decisive properties a model inherits from its base: vocabulary, knowledge, cultural alignment, memorized data, and license. This means that for Global South countries, using these models implies not only technological dependence but also a hidden sovereignty cost.

Inherited Properties That Fine-Tuning Cannot Mask

The paper's core argument is that fine-tuning only makes surface adjustments on top of the base model, while the base model's key characteristics are passed through unchanged to downstream applications. The vocabulary determines how the model segments text, directly affecting efficiency and fairness in multilingual scenarios; pretrained knowledge determines the framework through which the model understands the world; cultural alignment reflects the values embedded in the base model's training data; memorized data may contain sensitive information; and the license defines the boundaries of use.

This perspective elevates the choice of language model from a purely technical decision to a strategic one. For regions hoping to develop local AI capabilities, relying on the fine-tuning model means core capabilities remain under others' control.

Tokenization Premium: A Hidden Tax in Multilingual Scenarios

The paper conducts reproducible measurements across 31 languages and 7 tokenizers (released 2023–2024), using the 27 articles of the Universal Declaration of Human Rights as the test corpus and explicitly correcting for the bias introduced by unknown-token collapse.

The results reveal significant inequality: Amharic incurs a premium of up to 10× in older tokenizers, and even in the latest commercial tokenizers it remains at 6.8×. South American indigenous languages show premiums between 1.6× and 2.7× under global tokenizers. This means that processing the same semantic content in these languages requires more tokens, directly driving up inference costs and reducing the effective context length.

More interestingly, a vocabulary built on Portuguese reduces the premium for 11 of the 13 Latin-script Global South languages tested against all global tokenizers, with Guarani and Quechua seeing premium reductions of about 24%. This demonstrates that targeted vocabulary design can significantly mitigate the tokenization premium problem, offering a viable technical path toward linguistic diversity.

Tokenizer and BPE

A tokenizer is the first step in how a language model processes text: it splits the raw string into subword units (tokens) and then maps each token to an integer ID fed into the model. Mainstream models generally use byte-pair encoding (BPE) or its variants: starting from a character-level vocabulary, they repeatedly merge the most frequent adjacent symbol pairs in the corpus until a preset vocabulary size is reached. The composition of the vocabulary is entirely determined by the language distribution of the pretraining corpus—English and a few high-resource languages occupy the vast majority of merge slots.

This has a direct consequence: the same semantic content is split into more, finer-grained tokens in languages with insufficient vocabulary coverage. For example, a common word that corresponds to 1 token in English may be split into 4 to 8 tokens in Amharic or Quechua. Since inference cost and context-window occupancy are proportional to token count, the imbalance in vocabulary coverage translates into a real difference in economic cost—this is the technical root of the "tokenization premium."

Unknown-Token Collapse

A common measurement pitfall in multilingual tokenization evaluation is that when a tokenizer encounters a character outside its vocabulary, it maps it to a uniform unknown token (UNK) or replacement character. Without correction, text that originally requires multiple tokens is compressed into a small number of UNKs, statistically "lowering" the token count and making the language appear to have a very low or even negative premium.

The paper explicitly corrects for this bias, typically by using byte fallback or splitting unknown characters byte-by-byte according to UTF-8, ensuring that content in any language can be fully represented before counting tokens. This correction is crucial to the credibility of the conclusions—without it, the languages with the poorest vocabulary coverage would paradoxically show the lowest premiums, leading to conclusions entirely opposite to the facts.

Sovereignty Audit: Eight Tests Buyers Can Run Themselves

The paper proposes an eight-test black-box sovereignty audit framework that buyers can run without access to model weights. The value of this framework lies in turning sovereignty assessment from abstract discussion into an operational tool. For government agencies, enterprises, or research organizations, this means specific sovereignty dimensions can be introduced into procurement decisions, rather than focusing solely on benchmark scores.

The eight audit tests cover multiple aspects of model behavior. Although the paper's abstract does not list them one by one, the design philosophy is to enable users to independently verify whether a model meets local needs and values.

From Raw Accuracy to Accuracy per Joule: A Paradigm Shift in Design Goals

The paper's third contribution is an institutional argument: under energy and accountability constraints, accuracy per joule, calibration, and selective risk should replace raw accuracy as design goals, and metrics must be used that cannot be inflated by abstention behavior.

This view hits a blind spot in the current AI evaluation system. Raw accuracy can be inflated by avoiding uncertain samples, whereas calibration and selective risk metrics can reveal a model's reliability in real-world scenarios. As energy costs increasingly become a bottleneck for AI deployment, accuracy per joule is directly linked to sustainability and accessibility.

The paper concludes by mentioning preliminary internal measurement results of a proprietary architecture built around these goals, and explains its limitations. The data and measurement code are publicly available, providing a reproducible foundation for future research.

Calibration and Selective Risk

Calibration measures the consistency between a model's output confidence and its actual accuracy: a well-calibrated model should have an actual accuracy close to 80% on samples where it claims "80% confidence." Raw accuracy cannot reflect this—a model may have high accuracy but a completely miscalibrated confidence distribution, preventing downstream systems from setting thresholds accordingly.

Selective risk focuses on the error rate of "accepted predictions": when a system allows the model to abstain and answer only high-confidence samples, a trade-off curve forms between coverage and error rate. Looking only at raw accuracy can be inflated by abstention behavior—after the model abstains extensively, the remaining samples are naturally easier to answer correctly. Therefore, coverage and selective risk must be reported together to truly characterize a model's usability in human-AI collaboration and risk-sensitive scenarios.

Implications for the Global South

This research connects the technical choices of language models with sovereignty issues, providing a quantifiable framework. For Global South countries, the key insight is: fine-tuning is not a panacea; the inherited properties of the base model determine the capability ceiling and dependency relationships. Customizing vocabularies to reduce the tokenization premium, running black-box sovereignty audits, and adjusting design goals are feasible steps toward technological autonomy under existing constraints.

The paper's limitation is that the measurement results of the proprietary architecture are preliminary and internal, requiring more independent verification. However, the measurement methods and analytical framework it proposes already provide valuable tools for discussing the geopolitical economy of language models.