Non-stationarity
A core challenge in multi-agent reinforcement learning referring to the phenomenon whereby the continuous updating of multiple agents' policies causes the environmental distribution faced by each agent to shift constantly, invalidating convergence guarantees that rely on a fixed-environment assumption.
Core Facts
Timeline (last 90 days)
In multi-agent RL, the simultaneous updating of each agent's policy causes the environment (joint policy) faced by the other agents to change continuously, breaking the stationarity assumption of the Markov decision process in single-agent RL.
Non-stationarity refers to the statistical properties of data (mean, variance, autocorrelation structure, etc.) changing over time, and non-stationarity is especially pronounced in financial markets.
A model trained on two weeks of dry-season data degraded to near-random performance within one month of the wet season starting.
The PACT method attempts to directly model and adapt to interaction-induced non-stationarity, enabling agents to explicitly identify and adapt to distributional shifts caused by interactions with other agents.
Interaction-induced non-stationarity violates the stationarity assumption of MDPs, invalidating convergence guarantees derived from that assumption.
The real world is non-stationary, and a policy's performance during training may deteriorate sharply after the environment undergoes a distribution shift.
Stock price series are typical non-stationary data, and the substantial change between the data distribution at model training time and at prediction time is referred to in statistics as 'concept drift'.
All Facts (8)
Financial markets are characterized by extremely strong non-stationarity and low signal-to-noise ratio, and historical patterns do not necessarily extend into the future.
70%UnverifiedIn multi-agent RL, the simultaneous updating of each agent's policy causes the environment (joint policy) faced by the other agents to change continuously, breaking the stationarity assumption of the Markov decision process in single-agent RL.
50%UnverifiedNon-stationarity refers to the statistical properties of data (mean, variance, autocorrelation structure, etc.) changing over time, and non-stationarity is especially pronounced in financial markets.
50%UnverifiedA model trained on two weeks of dry-season data degraded to near-random performance within one month of the wet season starting.
50%UnverifiedInteraction-induced non-stationarity violates the stationarity assumption of MDPs, invalidating convergence guarantees derived from that assumption.
50%UnverifiedThe PACT method attempts to directly model and adapt to interaction-induced non-stationarity, enabling agents to explicitly identify and adapt to distributional shifts caused by interactions with other agents.
50%UnverifiedThe real world is non-stationary, and a policy's performance during training may deteriorate sharply after the environment undergoes a distribution shift.
50%UnverifiedStock price series are typical non-stationary data, and the substantial change between the data distribution at model training time and at prediction time is referred to in statistics as 'concept drift'.
50%