Evolution Strategies (ES)

A study from RSS proposed a centroid geometric interpretation of evolution strategies (ES): treating perturbed models as particles and rewards as mass, the ES update is equivalent to a centroid displacement.

For any finite population, the mean-subtracted ES update is exactly equal to the centroid displacement multiplied by a scalar factor that depends on the mean reward.

The ES update direction is essentially the direction of movement of the population centroid, and the update magnitude is proportional to the size of the centroid displacement.

The centroid interpretation of ES requires rewards to be non-negative, but since adding a constant to all rewards leaves the update result completely unchanged, this does not constitute a restriction.

In standard ES, subtracting the mean reward from the rewards (mean subtraction) reduces variance; from the centroid perspective, it switches the displacement reference point from the model's current position to the sampled population centroid, with direction and relative magnitude unchanged.

When the population size tends to infinity or antithetic sampling is used, the population centroid falls exactly at the model's own position, and the ES update direction points directly to the reward-weighted centroid.

Antithetic sampling ensures that the population center coincides with the model by generating positive and negative perturbations in pairs, eliminating centroid offset and making the update direction reflect the reward signal more purely, and thus performs more stably in practice.

Evolution strategies (ES) are a class of population-based gradient-free optimization algorithms that iteratively improve a model by sampling perturbations, evaluating fitness, and updating parameters with weights.

Classical ES (such as CMA-ES) maintains a search distribution (usually a Gaussian distribution) and adjusts the distribution's mean and covariance according to fitness.

The ES variant discussed in this paper is in the same lineage as OpenAI ES: rather than explicitly maintaining distribution parameters, it applies Gaussian perturbations to the current model parameters to obtain offspring and updates using the reward-weighted average direction as weights.

The centroid interpretation holds exactly for any finite population, not merely as an approximation in the large-population limit.

The core contribution of this research is not proposing a new algorithm but establishing an exact geometric equivalence that frees the ES update from the gradient-estimation framework.