Diffusion Model Membership Inference: A New Approach to Difficulty Calibration Along the Time Dimension

Diffusion models are trained on massive datasets that may contain copyrighted or private images. Membership inference attacks (MIA) are used to detect whether a given image belongs to the training set. Traditional methods score by reconstruction loss, flagging low-loss images as members. But this approach often fails because some images are inherently easier to reconstruct—a simple unseen image may have lower loss than a complex training image. Difficulty calibration separates memorization from the image's intrinsic difficulty. Recent research proposes calibrating along two axes of time: training time and diffusion time.

Training Time Axis: Reference Model Calibration (RMC)

The first axis is training time. Researchers introduce Reference Model Calibration (RMC), which compares the loss of the fine-tuned model against early checkpoints. This method cleanly isolates recently memorized data, enabling near-perfect auditing even without text descriptions. The core of RMC: by comparing how the loss for the same image changes across different training stages, it distinguishes "the model has memorized it" from "the image is inherently simple."

However, early checkpoints are not always available. This limits RMC's practical applicability. The researchers therefore further propose a scheme that does not rely on historical checkpoints.

Diffusion Time Axis: Temporal Self-Calibration (TSC)

The second axis is diffusion time. Temporal Self-Calibration (TSC) uses only a single model and calibrates along diffusion time. The key observation: image difficulty persists across noise levels, while memorization is localized in time. By comparing the loss at the target timestep with those at adjacent timesteps, TSC leverages the model's own trajectory to attenuate the influence of image difficulty.

Specifically, TSC establishes a baseline between the loss at the target timestep and the losses at neighboring timesteps, thereby stripping away the image's inherent ease of reconstruction. This method requires no additional checkpoints and can be calibrated using only the current model.

Diffusion time refers to the timestep $t$ in the forward noising process of a diffusion model, from $t=0$ (clean image) to $t=T$ (pure noise). During training, the model learns to predict noise or scores at different noise levels. For membership inference, the model's "memory" of training images tends to concentrate in specific timestep intervals—i.e., the model has overfit the training samples at that noise level; while the image's intrinsic "difficulty" (e.g., texture complexity) affects reconstruction loss relatively consistently across the entire time axis. TSC exploits this difference: it selects adjacent timesteps near the target timestep as references to construct a local baseline, thereby canceling out the loss shift caused by the image's inherent difficulty and retaining only the abnormally low loss induced by memorization. This calibration along the diffusion time axis requires no access to training history and relies solely on the consistency of the current model's behavior across different noise levels.

Parameterization refers to the target quantity that the network is trained to predict in a diffusion model. There are three common forms: $\epsilon$-prediction (directly predicting the added noise), $x$-prediction (predicting the clean image), and $v$-prediction (predicting a linear combination of noise and image, i.e., velocity). Different parameterizations change the scale and distribution of the loss function, making loss-threshold-based membership inference attacks difficult to transfer directly across models. The calibration mechanisms of RMC and TSC act on the spatial structure of the loss rather than its absolute values, and thus can generalize to these different parameterizations—an important property that makes them superior to traditional attack methods.

Experimental Results and Advantages

Empirical results show that these methods substantially outperform state-of-the-art techniques. Unlike most existing attacks, they generalize to different model parameterizations, such as $x$-, $v$-, and $\epsilon$-prediction models. The code is publicly available on GitHub (https://github.com/MichaelJoseph777/TSC).

RMC and TSC approach from the perspectives of the training process and the diffusion process, respectively, offering new ideas for difficulty calibration in membership inference. RMC performs near-perfectly when historical checkpoints are available, while TSC remains effective with only a single model. What they share is the recognition that difficulty is an intrinsic property of the image, whereas memorization is acquired by the model during training—and the two must be distinguished along the time axis.

Conclusion

This research addresses a fundamental problem in membership inference: how to distinguish an image's inherent ease of reconstruction from the model's memorization of training data. By calibrating difficulty along two axes—training time and diffusion time—RMC and TSC significantly improve attack accuracy and generalization. For researchers and engineers concerned with privacy auditing of diffusion models, this method provides a new tool.