DP-SGD
A machine learning training algorithm that combines differential privacy with stochastic gradient descent. It protects training data privacy by clipping gradients and adding Gaussian noise, and serves as the de facto standard method for training privacy-sensitive models.
Core Facts
Timeline (last 90 days)
The memory overhead of the adjoint method scales linearly rather than quadratically with the number of iterations, making it more suitable for DP-SGD scenarios with tens of thousands of iterations.
DP-SGD protects privacy by adding Gaussian noise to gradients, and privacy accounting tools (such as Rényi DP or zCDP) can precisely compute the cumulative privacy loss.
The optimizers covered by DP-μP include DP-SGD, DP-Adam and DP-Muon.
On a synthetic oncology benchmark, active learning + DP-SGD at ε=3 matched the performance of random sampling + DP-SGD at ε=8 in AUROC.
DP-SGD is a mechanism for achieving differential privacy in stochastic gradient descent, implemented by clipping per-sample gradients and then adding Gaussian noise.
Differentially private stochastic gradient descent (DP-SGD) has become the de facto standard for training privacy-sensitive models.
The information leaked during training is measured by conditional mutual information (CMI), and for DP-SGD this leakage bound depends only on two hyperparameters: the sampling rate and the noise multiplier.
All Facts (7)
DP-SGD is a mechanism for achieving differential privacy in stochastic gradient descent, implemented by clipping per-sample gradients and then adding Gaussian noise.
75%UnverifiedDP-SGD protects privacy by adding Gaussian noise to gradients, and privacy accounting tools (such as Rényi DP or zCDP) can precisely compute the cumulative privacy loss.
50%UnverifiedThe memory overhead of the adjoint method scales linearly rather than quadratically with the number of iterations, making it more suitable for DP-SGD scenarios with tens of thousands of iterations.
50%UnverifiedThe optimizers covered by DP-μP include DP-SGD, DP-Adam and DP-Muon.
50%UnverifiedOn a synthetic oncology benchmark, active learning + DP-SGD at ε=3 matched the performance of random sampling + DP-SGD at ε=8 in AUROC.
50%UnverifiedDifferentially private stochastic gradient descent (DP-SGD) has become the de facto standard for training privacy-sensitive models.
50%UnverifiedThe information leaked during training is measured by conditional mutual information (CMI), and for DP-SGD this leakage bound depends only on two hyperparameters: the sampling rate and the noise multiplier.
50%