Adversarial Multiplicity: Model Selection Manipulation and the Limits of Disclosure

Predictive multiplicity in machine learning models—the phenomenon where models with similar performance give different predictions for the same individual—was originally seen as an exploitable property, for example to optimize secondary metrics like fairness without sacrificing predictive performance. But a new paper points out that this multiplicity also gives motivated parties (including coding agents that "game the score" to achieve their objectives) room to manipulate, steering predictions in their favor in settings such as litigation or consumer disputes.

Soft and Hard Manipulation: Two Attack Paths

The paper divides manipulation under adversarial multiplicity into two categories. Soft manipulation refers to a party optimizing within the harmless space of model selection, such as hyperparameter tuning or feature engineering; hard manipulation refers to directly perturbing model parameters. The study finds that a sufficiently flexible model can be manipulated to reverse any single prediction with minimal change in predictive performance, and can pass black-box audits—meaning hard manipulation is feasible in most regulatory environments.

The effect of soft manipulation is relatively smaller, but it can pass white-box audits that require full access to code and data. This finding challenges the intuition that "white-box audits are stricter": even if the auditor can see all training details, soft manipulation may still slip through.

Black-box audits and white-box audits are two types of review at different depths in model regulation. A black-box audit observes only a model's input-output behavior and does not touch training code, data, or parameters—similar to evaluating a system solely through question and answer. A white-box audit requires full access to code, training data, and model weights, allowing the training process to be reproduced and every detail inspected. Intuitively, white-box audits are stricter, but the paper's findings overturn exactly this: hard manipulation (directly changing parameters) has minimal impact on predictive performance and is behaviorally hard to distinguish from a normal model, so it can fool black-box audits that only look at behavior; soft manipulation, while weaker in effect, occurs within "legitimate" choice spaces such as hyperparameter tuning and feature engineering, so even if the auditor sees all training details, it is hard to determine whether a given choice was normal tuning or intentional manipulation. This shows that audit depth and audit effectiveness are not simply positively correlated; the key is whether the audit can constrain the ex post degrees of freedom in model selection.

Aggregate Estimates Are Equally Fragile

Manipulation affects not only individual predictions; even aggregate-level estimates can be distorted. The paper gives a concrete example: a model estimate of climate change's impact on wildfires can be completely reversed through hard editing and soft search. This means that policymaking and scientific research that rely on model aggregate results also face the risk of intentional manipulation.

Model aggregation refers to combining predictions or estimates from multiple models into one overall conclusion, commonly through averaging, weighted ensembles, or meta-analysis. Aggregation is generally considered more robust than a single model because random errors cancel out in the average. But this robustness relies on an implicit premise: the models participating in aggregation are selected independently and without bias. When parties can selectively pick or fine-tune the models entering the aggregation, the bias is not averaged away but systematically amplified—exactly as in the paper's example of the relationship between climate change and wildfires. This reminds policymakers and researchers that the reliability of aggregate results depends not only on the quality of individual models but also on whether the process by which the model set is constituted is auditable and locked in advance by pre-registration.

The Reconcile Algorithm and the Manipulator's Advantage

The paper examines the Reconcile algorithm—a method for resolving disagreement among models. The results show that the algorithm usually resolves most disagreements among models, but when the opposing models are adversarially selected rather than randomly selected, the disagreement resolution results more frequently favor the manipulator. In other words, as long as an adversary selectively picks models, existing disagreement reconciliation mechanisms may instead become accomplices.

The Reconcile algorithm is a class of methods for reconciling predictive disagreements among multiple models; its approach is to iteratively adjust so that conflicting models converge on most samples, thereby producing a more stable consensus prediction. When models are produced randomly or neutrally, Reconcile can effectively resolve most disagreements. But its working mechanism implicitly assumes that both opposing sides are unbiased: the algorithm tends to move toward the "majority" or "more consistent" side. When an adversary deliberately selects a combination of models that opposes the target model, this direction of convergence is predetermined by the adversary, causing the reconciliation result to systematically favor the manipulator. In game theory, this phenomenon is known as the "strategic selection" problem, showing that any reconciliation mechanism relying on a set of models must also constrain how models are selected into that set.

Blind Spots in Existing Regulation and Disclosure Policy

The study sorts out which manipulation methods are likely to pass current regulation and disclosure policies. The conclusion is not optimistic: black-box audits cannot stop hard manipulation, white-box audits cannot stop soft manipulation, and adversarial model selection can influence disagreement resolution results. The "transparency equals trust" assumption underlying disclosure policy looks thin in the face of adversarial multiplicity.

Mitigation Measures: From Pre-registration to Versioned Models

The paper recommends several mitigation measures: pre-registration, i.e., locking in the model selection plan before analysis; providing model training proofs; conducting sensitivity analysis; adopting multi-metric evaluation; and committing to use versioned public models. The common logic of these measures is to compress the ex post degrees of freedom in model selection, making it harder and more visible to "pick a model that favors you."

For teams building automated decision-making tools, this research reminds us: meeting model performance targets and passing audits does not mean the model has not been manipulated. In adversarial settings, the boundaries of disclosure need to be redrawn.

Pre-registration originates in empirical scientific research and refers to publicly locking in research hypotheses, model selection plans, and evaluation metrics before analyzing data, then reporting results afterward, in order to prevent the ex post degrees of freedom of "looking at the results first and then choosing the method." In machine learning settings, pre-registration means fixing the model architecture, hyperparameter search space, and selection criteria before touching the test data, so that the act of "picking a model that favors you" leaves a verifiable record. The accompanying sensitivity analysis tests whether conclusions are robust to model selection and parameter perturbations: if slight changes cause conclusions to flip, the results themselves are fragile. Versioned public models require disclosure of the specific model version used and its training proof, enabling external parties to reproduce and compare, thereby compressing the ex post selection space to a minimum. The common logic of these measures is: transparency must apply to the selection process, not just the final model.

One-line Summary

Model predictive multiplicity can be manipulated by motivated parties, and existing audit and disclosure policies are hard-pressed to prevent it.

Paragraph Summary

The article introduces a new paper on "adversarial multiplicity." In machine learning, models with similar performance give different predictions for the same individual; this property was originally seen as room to optimize secondary metrics like fairness, but the paper points out that it also gives motivated parties opportunities for manipulation. Manipulation is divided into soft manipulation (tuning within the legitimate model selection space) and hard manipulation (directly perturbing parameters): hard manipulation can reverse any single prediction and can pass black-box audits, while soft manipulation has a smaller effect but can pass white-box audits, overturning the intuition that "white-box is stricter." Aggregate estimates can likewise be completely reversed, and the Reconcile disagreement reconciliation algorithm instead favors the manipulator under adversarial model selection. The paper recommends compressing the ex post selection space through pre-registration, training proofs, sensitivity analysis, multi-metric evaluation, and versioned public models.