Conversation Information Loss in LLM API Translation: The isloss Rule-Based Detection Framework

isloss uses rules to detect silent information loss in LLM API translation, revealing interoperability risks.
This article introduces isloss, a rule-based deterministic oracle for detecting silent conversation information loss in LLM API translation. It maps source requests, translated requests, and replies into a shared intermediate representation, ConvIR, and uses explicit rules to distinguish legitimate representation changes from avoidable loss without reference translations or extra model calls. Evaluation shows 76.0% detection of injected faults with a 2.0% false positive rate; in real scenarios, 18.5% of translated requests passed static checks yet silently altered prompt, role, or tool information, and 36.4% of translated reply streams caused exceptions or corruption in the official OpenAI SDK. Targeted experiments show that when tasks depend on affected information, translation-induced changes alter model behavior, proving that conversation information preservation is an independent correctness requirement. The article recommends adding preservation tests to cross-API integrations and notes rule maintenance and production diversity validation as future directions.
Introduction
As the LLM application ecosystem expands, developers increasingly rely on translation and compatibility layers to migrate conversations across APIs. However, a request that is accepted by the target API may have already silently lost information such as instruction roles, tool interactions, error states, or generation controls. This "silent loss" does not trigger errors, yet it changes model behavior. A recent study proposes isloss—a rule-based deterministic oracle specifically designed to detect whether conversation information is preserved during LLM API translation.

What Is Conversation Information Loss?
In LLM application architectures, APIs from different vendors differ in message formats, role definitions, tool calls, and parameter controls. For compatibility, developers often use translators, gateways, or client libraries to perform format conversion. The problem is that even when a translated request is accepted normally by the target API, it may have lost key information from the original request.
For example, the source API may support a particular system instruction role, but the target API downgrades it to a regular user message, changing instruction priority; or the intermediate state of a tool call may be omitted during translation, affecting subsequent reasoning. Such loss is silent—no exceptions, no errors—but model output may deviate from expectations as a result.
The LLM API message model: mainstream LLM APIs (such as OpenAI Chat Completions and Anthropic Messages) model a conversation as an ordered list of messages, each with a role (system, user, assistant, tool) and content (text, images, tool call requests/results). Vendors differ in role semantics, tool call representation (such as the schema for function calling and the format for returning results), and generation control parameters (temperature, top_p, stop, max_tokens, etc.). The translation layer must map between these heterogeneous representations, and improper mapping leads to information loss. For example, when converting Anthropic's system parameter into an OpenAI system message, incorrect positioning or merging can change instruction priority.
Core Design of isloss
The core idea of isloss is to map the source request, the translated request, and the corresponding replies into a shared intermediate representation, ConvIR (Conversation Intermediate Representation), and then apply a set of explicit rules to distinguish "legitimate representation changes" from "information loss."
Unlike traditional schema validation or reference-based differential testing, isloss requires neither a reference translation nor additional model calls. It uses rules to determine whether a loss is avoidable: if the target API itself supports the information but the translation layer discards it, the loss is considered avoidable; if the target API genuinely does not support the feature, it is considered degradation.
This design allows isloss to run independently, making it suitable for continuous integration and automated testing scenarios.
Deterministic oracle: In software testing, an oracle is the basis for deciding whether a test passes. A deterministic oracle does not rely on randomness or model inference and always produces the same verdict for the same input. isloss uses rules rather than models to judge information loss, so its results are reproducible, interpretable, and cheap to run, making it suitable for frequent execution in CI. Compared with differential testing (which requires a reference implementation or multiple-run comparison), a deterministic oracle needs no reference translation and can directly pinpoint the type of loss.
Evaluation Results: Fault Detection and Real-World Scenarios
The research team evaluated isloss through specification-based fault injection, manual review, and experiments on translators, gateways, and client libraries.
On the held-out test set, isloss detected 95 of 125 injected faults (76.0%), with 2 false positives out of 100 (2.0%). This shows that the rule-based approach achieves high precision and recall in identifying silent loss.
More notable is its performance in real-world scenarios. On a synthetic request suite covering six configurations across five translator codebases, 127 of 685 completed translations (18.5%) passed the target API's static checks yet silently altered prompt, role, or tool information—information that the target API could have preserved.
On the translated reply side, 203 of 558 streams (36.4%) triggered exceptions or silent corruption in at least one tested official OpenAI SDK. This means information loss affects not only requests but also reply parsing and downstream processing.
Targeted experiments across six models show that when tasks depend on the affected information, translation-induced changes alter model behavior. This confirms that conversation information preservation is an independent correctness requirement, not a marginal issue.
Fault injection and held-out test sets: Fault injection is a standard method for evaluating detection tools: known defects are deliberately introduced into the translation process (such as deleting a tool call ID or changing a system role to user), and the tool is then checked to see whether it can detect them. A held-out test set refers to samples kept independent of parameter tuning and rule design, used for unbiased evaluation. isloss detected 95 of 125 injected faults (76.0%) with 2/100 false positives (2.0%), indicating that the rule-based approach strikes a good balance between precision and recall. However, the 24% miss rate also suggests that the rule base still has coverage blind spots.
Implications for Developers
For teams building LLM applications, agent execution, or model evaluation systems, isloss highlights a neglected dimension: the correctness of the API translation layer. Traditionally, developers focus on whether a request succeeds and whether the response conforms to the schema, but they rarely verify whether semantic information has been fully migrated.
It is advisable to add conversation information preservation tests to cross-API integrations, especially when complex roles, multi-turn tool calls, or generation control parameters are involved. As a deterministic tool, isloss can be integrated into test pipelines at low cost, helping to uncover problems that do not raise errors but change behavior.
Static checks and semantic correctness: A target API's static checks typically verify only whether a request structure conforms to the schema (field types, required fields, enum values), not whether it is semantically equivalent to the source request. As a result, a request can pass static checks yet be accepted with downgraded roles, omitted tool calls, or lost generation parameters. The value of isloss lies in advancing checks from "structurally valid" to "semantically preserved," establishing conversation information preservation as a correctness dimension independent of request success.
Limitations and Future Directions
isloss currently relies on manually written rules, and its coverage is limited by ConvIR's expressive power and the completeness of the rule base. For rapidly evolving API features, the rules require continuous maintenance. In addition, although the study's synthetic suite and model experiments reveal the ubiquity of the problem, the diversity of real production environments still needs further validation.
Nevertheless, this work establishes "conversation information preservation" as an independent correctness dimension for LLM interoperability, providing a foundation for the development of future tools and standards.
One-line Summary
isloss uses rules to detect silent information loss in LLM API translation, revealing interoperability risks.
Paragraph Summary
This article introduces isloss—a rule-based deterministic oracle for detecting silent loss of conversation information during LLM API translation. It maps source requests, translated requests, and replies through a shared intermediate representation, ConvIR, and uses explicit rules to distinguish legitimate representation changes from avoidable loss, without requiring reference translations or additional model calls. Evaluation shows 76.0% detection of injected faults on a held-out test set with a 2.0% false positive rate; in real-world scenarios, 18.5% of translated requests passed static checks yet silently altered prompt, role, or tool information, and 36.4% of translated reply streams caused exceptions or corruption in the official OpenAI SDK. Targeted experiments show that when tasks depend on the affected information, translation-induced changes alter model behavior, proving that conversation information preservation is an independent correctness requirement. The article recommends adding preservation tests to cross-API integrations and identifies rule maintenance and production diversity validation as future directions.
Related articles

Programmers in the AI Era: How Deep Should You Learn Legacy Tech?
How should programmers transform under AI's impact? How deep should legacy tech be learned? Drawing on a Bilibili creator's share, this breaks down the levels of embracing AI, the shift from process-driven to project-driven learning, and how lower technical barriers affect junior and mid-level roles.

OpenAI Releases GPT-6.2 Dual Models Astra and Sol, API Price Cut by 20%
OpenAI releases GPT-6.2 Astra and Sol, with Astra's API price cut 20% to $8/$40 per million tokens and ranking second on the Artificial Analysis Intelligence Index. This article analyzes the dual-model strategy, price-cut impact, and developer selection advice.

Perplexity Computer Launches Multiagent: 16 AI Agents Cross-Validate Each Other
Perplexity launches Multiagent mode for Computer: a lead agent orchestrates up to 16 sub-agents on Claude, GPT, and Gemini, sharing findings and cross-validating work with effort above Ultra.