ML-Agents
An open-source machine learning toolkit from Unity that lets developers train agents within Unity environments. It supports reinforcement learning algorithms such as PPO and is commonly used for robot simulation training.
Core Facts
Timeline (last 90 days)
A common mistake among developers is to directly apply the Behavior Parameters configuration from ML-Agents examples while ignoring the coupling between the underwater physics step size (Fixed Timestep) and the training step size, which leads to unstable training or a policy that cannot be transferred.
ML-Agents works as follows: on the Unity side, observation, action, and reward are defined through Agent scripts, and communication with an external Python training process is done via gRPC; on the Python side, reference implementations of algorithms such as PPO and SAC are provided (based on PyTorch), and Stable-Baselines3 or Ray RLlib can also be connected through the mlagents-envs interface.
For non-standard scenarios such as AUVs, the advantage of ML-Agents lies in fast environment setup and good visualization, but the default DecisionRequester and VectorObservation mechanisms need to be adapted for continuous action spaces.
When the number of parallel environments increases from N to M, batch_size should increase at least linearly by a ratio of M/N; buffer_size should remain 5 to 10 times batch_size.
When the number of environments increases from 10 to 1000, if an MLP with hidden_units of 512 and num_layers of 3 has insufficient capacity, the policy network will degenerate into outputting approximately average actions for all states, manifesting as being unable to learn or as stagnant rewards.
For scenarios with 100 to 1000 parallel robots, the recommended ML-Agents configuration is: batch_size 8192-32768, buffer_size 81920-327680, learning_rate 0.0001-0.0005, time_horizon 200-500, summary_freq 100000-300000, and max_steps kept at 30000000 or appropriately increased.
Enabling threaded: True may help with CPU parallel sampling under multiple environments in ML-Agents.
When using shared_critic: False, consider changing it to True with a large number of parallel environments to reduce the number of parameters.
The square root scaling formula for the learning rate is: learning rate multiplied by sqrt(M/N); when scaling from 10 environments to 1000, the learning rate is adjusted to 0.0003 * sqrt(100) = 0.003, but this may be too aggressive in ML-Agents, so it is recommended to start by trying 1.5-2 times.
In ML-Agents, buffer_size corresponds to the total number of experience steps PPO collects between two policy updates, approximately equal to num_envs times time_horizon.
2 more timeline events
All Facts (15)
ML-Agents is an officially open-sourced machine learning framework from Unity that enables agents to undergo reinforcement learning training by interacting with the game environment to receive reward signals.
65%UnverifiedML-Agents支持近端策略优化（PPO）、软演员-评论家（SAC）等主流算法，并可与Python生态中的PyTorch深度集成
90%Unverified强化学习训练出的NPC相比传统行为树能够在复杂动态环境中涌现出设计者未预设的行为策略
80%UnverifiedA common mistake among developers is to directly apply the Behavior Parameters configuration from ML-Agents examples while ignoring the coupling between the underwater physics step size (Fixed Timestep) and the training step size, which leads to unstable training or a policy that cannot be transferred.
50%UnverifiedML-Agents works as follows: on the Unity side, observation, action, and reward are defined through Agent scripts, and communication with an external Python training process is done via gRPC; on the Python side, reference implementations of algorithms such as PPO and SAC are provided (based on PyTorch), and Stable-Baselines3 or Ray RLlib can also be connected through the mlagents-envs interface.
50%UnverifiedFor non-standard scenarios such as AUVs, the advantage of ML-Agents lies in fast environment setup and good visualization, but the default DecisionRequester and VectorObservation mechanisms need to be adapted for continuous action spaces.
50%UnverifiedWhen the number of parallel environments expands from 10 to 1000 while buffer_size remains unchanged, the trajectory length allocated to each environment is compressed to 1/100 of the original, causing severe trajectory truncation and undermining GAE's estimation of returns.
50%UnverifiedWhen the number of environments increases from 10 to 1000, if an MLP with hidden_units of 512 and num_layers of 3 has insufficient capacity, the policy network will degenerate into outputting approximately average actions for all states, manifesting as being unable to learn or as stagnant rewards.
50%UnverifiedFor scenarios with 100 to 1000 parallel robots, the recommended ML-Agents configuration is: batch_size 8192-32768, buffer_size 81920-327680, learning_rate 0.0001-0.0005, time_horizon 200-500, summary_freq 100000-300000, and max_steps kept at 30000000 or appropriately increased.
50%UnverifiedThe square root scaling formula for the learning rate is: learning rate multiplied by sqrt(M/N); when scaling from 10 environments to 1000, the learning rate is adjusted to 0.0003 * sqrt(100) = 0.003, but this may be too aggressive in ML-Agents, so it is recommended to start by trying 1.5-2 times.
50%UnverifiedEnabling threaded: True may help with CPU parallel sampling under multiple environments in ML-Agents.
50%UnverifiedWhen using shared_critic: False, consider changing it to True with a large number of parallel environments to reduce the number of parameters.
50%UnverifiedWhen the number of parallel environments increases from N to M, batch_size should increase at least linearly by a ratio of M/N; buffer_size should remain 5 to 10 times batch_size.
50%UnverifiedIn ML-Agents, buffer_size corresponds to the total number of experience steps PPO collects between two policy updates, approximately equal to num_envs times time_horizon.
50%UnverifiedML-Agents is Unity's reinforcement learning plugin.
50%