Reversible Pseudonymisation for External LLM Documents: An Open-Source Bilingual Benchmark and Engine

Why Do We Need Reversible Pseudonymisation?

When sending work documents to external large language models, personal data leakage is a primary risk. Reversible pseudonymisation is a common safeguard: before a request is sent, personal data in the document is replaced with placeholders; after the model returns an answer, the placeholders are restored locally to the original information. This workflow protects privacy without compromising the LLM's understanding of and answer to the document content.

However, can existing open-source tools reliably accomplish this task? A new study provides the answer: the authors built an open-source bilingual benchmark, evaluated five existing open-source tools and a regular-expression baseline, and developed an open-source engine based on the two strongest of them.

Differences Between Reversible Pseudonymisation, Irreversible Anonymisation, and Pseudonymisation

The core of reversible pseudonymisation lies in "restorability": it replaces sensitive information with placeholders but preserves a mapping so that the model output can be restored to the original text. In contrast, irreversible anonymisation (such as hashing, generalisation, noise injection) permanently destroys the original value and cannot be restored; pseudonymisation usually refers to replacing real values with fictitious identifiers (e.g., replacing "John Smith" with "User A"), but the mapping table between pseudonyms and real values must be securely stored—essentially the same category as reversible pseudonymisation, though pseudonymisation emphasises that the replacement values look like real data. The engineering challenge of reversible pseudonymisation is: the LLM must understand the replaced text (placeholders must not break semantics), the mapping table must be securely stored, and placeholders must be accurately identified and restored in model answers, avoiding erroneous replacement or omission.

Impact of Placeholder Design on LLM Understanding

The form of placeholders directly affects the LLM's understanding of the document. If placeholders are too abstract (e.g., "<PERSON_1>"), the model may struggle to associate them with context; if too specific (e.g., retaining some real characters), they may leak information. A common practice is to use semantically neutral placeholders (e.g., "[NAME]", "[EMAIL]") while keeping their position and grammatical role in the document unchanged. In addition, placeholders need to be unique so they can be precisely matched and restored in model answers. The "mask precision" evaluated in the benchmark measures whether a tool can correctly identify and replace all entities that should be pseudonymised, while the "full-value visibility ratio" measures whether original values are accidentally exposed after restoration—together these two metrics form the core evaluation dimensions of reversible pseudonymisation tools.

Benchmark Design: Rigour and Transparency

The benchmark design emphasises reproducibility and fairness. It excludes each tool's own author's dataset to avoid bias; compares the engine and each tool's detection performance on the same datasets; reports mask precision; and tests on a held-out set—composed of synthetic documents whose files, hashes, decision rules, run plans, and stopping rules were published before the run.

All evaluation data are synthetic, and the benchmark was run by the engine's authors themselves, but the code, data, predictions, and raw answers are all made public for community review.

Performance of Existing Tools: Below Threshold

On external datasets, no existing tool met all predefined thresholds. Specifically, in British English, neither the identifier threshold nor the leakage threshold was met; in Russian, the identifier threshold was borderline.

On the final held-out set of 1,000 documents per language, the best existing tool had at least one full value visible in 61% (Russian) and 58% (British English) of documents. This means more than half of the documents carry a privacy leakage risk.

Definitions of Identifier Threshold and Leakage Threshold

The "identifier threshold" and "leakage threshold" mentioned in the benchmark are pre-set acceptance criteria. The identifier threshold usually refers to the minimum recall or F1 score when a tool detects personal data entities—i.e., it must identify at least a certain proportion of the real personal data in the document; the leakage threshold refers to the maximum proportion of full or partial original values allowed to appear in the restored document. These two thresholds jointly constrain two types of tool errors: missed detection (sensitive information not replaced) and over-replacement (damaging non-sensitive content). In British English, neither threshold was met, indicating that existing tools either miss detections severely or expose original values during restoration; in Russian, the identifier threshold was borderline, meaning detection capability barely meets the bar but is not stable enough. These thresholds are not set arbitrarily but based on the practical needs of privacy protection—for example, the leakage threshold usually requires the full-value visibility ratio to be below 1%, because even a small leak may lead to re-identification of personal data.

Breakthrough of the New Engine: Drastic Reduction in Leakage

The new engine built on the two strongest tools performs significantly better on the same documents: in Russian and British English, the proportion of full values visible dropped to 0.3% [0.1; 0.9] and 0.2% [0.1; 0.7], respectively; the proportion of any partial value visible was 0.8% and 0.2%, respectively. At the same time, the engine met all detection thresholds in both languages.

In British English, the engine could still restore at least 99% of placeholders after 760 answers. But in Russian, the restoration rate for name placeholders was only 94.6% [93.6; 95.6], below the 99% threshold, mainly because the inflection rules it borrowed from pii-guard produced incorrect surname forms.

Inflection and the Challenge of Name Restoration

The core reason for the low Russian name restoration rate (94.6%) is inflection. Russian is a highly inflected language: nouns, adjectives, and surnames change according to case, number, and gender. For example, a surname may be "Иванов" in the subject position, become "Иванова" in the genitive, and "Иванову" in the dative. When a tool replaces a name with a placeholder, it needs to recognise and preserve this morphological variation so that the correct form can be restored according to context. If a tool simply treats all variants as the same entity and uniformly replaces them with a fixed placeholder, it cannot distinguish different case forms during restoration, leading to errors. pii-guard's inflection rules may cover only some common patterns and handle irregular surnames or rare variants inadequately, thus producing incorrect surname forms. By contrast, British English names have fewer morphological variations, so the restoration rate exceeds 99%. This flaw suggests that cross-lingual reversible pseudonymisation requires replacement and restoration strategies designed specifically for the morphological features of the target language, rather than simply applying English rules.

Limitations and Future Directions

The engine was developed on six earlier held-out sets produced by the same generator and has not yet been tested on real documents. The flaw in Russian name restoration also highlights the challenge of inflection handling. Nevertheless, the study provides the first open-source bilingual benchmark and a transparent, reproducible evaluation framework for the field of reversible pseudonymisation, advancing privacy-preserving LLM applications.

Conclusion

Reversible pseudonymisation is a practical means of protecting personal data in external LLM requests, but existing open-source tools still fall short in detection and leakage control. The new engine shows significant improvements on synthetic data, especially in reducing full-value leakage, but Russian name restoration still needs optimisation. Future work should focus on testing on real documents and cross-lingual morphological processing.

One-line Summary

A new study builds an open-source bilingual benchmark, evaluates existing reversible pseudonymisation tools, and develops a new engine that significantly reduces leakage but still needs optimisation for Russian name restoration.

Paragraph Summary

Targeting the risk of personal data leakage when sending work documents to external LLMs, this study builds an open-source bilingual (Russian and British English) benchmark and evaluates five existing open-source tools and a regular-expression baseline. The benchmark uses synthetic data, excludes tool authors' own datasets to avoid bias, and pre-publishes held-out set rules to ensure reproducibility. Results show that no existing tool met all predefined thresholds, with the best tool leaking at least one full value in 61% (Russian) and 58% (British English) of documents. A new engine built on the two strongest tools reduces the full-value visibility ratio to 0.3% and 0.2% and meets all detection thresholds; the placeholder restoration rate in British English exceeds 99%, but Russian name restoration is only 94.6%, mainly due to flawed inflection rules. The study provides the first open-source bilingual benchmark and a transparent evaluation framework, but the engine has not yet been tested on real documents.