Augmentable Gaussian Splatting: Building Photorealistic Simulation Environments for Robot Training

Why Do Robot Policies Perform Well in Simulation but Fail on Real Hardware?

When training robot policies, teams often rely on animation-style simulation environments. Objects and lighting in these environments are simplified, and while policies run well in simulation, they frequently fail on real hardware. The problem isn't the algorithm itself—it's the massive visual and physical gap between the simulated world and the real world.

A team shared their approach on Reddit: rather than waiting for tech giants to release mature products, they built photorealistic simulation environments themselves. Their core solution combines high-quality Gaussian Splats with traditional Meshes inside a physics engine, making simulated scenes look like the real world while retaining full physical interaction capabilities. According to the original post, this pipeline went through hundreds of iterations before stabilizing.

Gaussian Splats + Meshes: Combining the Strengths of Two Approaches

Gaussian Splatting excels at reconstructing photorealistic scene appearances from multi-view photos, but it isn't a physical object—it can't be directly used for collision detection or rigid body dynamics. Traditional meshes provide the geometric information physics engines need, but visual realism often falls short.

The team's approach places both into the same physics environment: Gaussian Splats handle visual rendering, while meshes handle physics simulation. The resulting scenes "look as real as the real world" while possessing all the physics capabilities currently achievable in mesh environments. The original post emphasizes that this isn't simply overlaying visual textures onto a physics scene—rather, splats and meshes work together in the same coordinate system, so objects the robot manipulates remain consistent across both visual and physical layers.

Another key capability is being augmentable and relightable. The same real-site Gaussian Splat can be repeatedly modified: swapping objects, adjusting lighting, changing layouts—generating hundreds of variants of the same environment. For robot policy training, this domain randomization capability directly affects policy generalization performance.

Gaussian Splatting is a 3D scene representation and rendering method proposed in 2023. It encodes scenes using large numbers of 3D Gaussian ellipsoids with position, covariance, color, and opacity attributes. During rendering, differentiable rasterization projects the Gaussians onto the image plane and blends them, achieving real-time frame rates while maintaining photorealistic quality. Compared to Neural Radiance Fields (NeRF), Gaussian Splatting trains and renders faster and has an explicit, easily editable representation, making it particularly suitable for rapid reconstruction of real scenes from multi-view photos. However, Gaussian Splatting is essentially a radiance-field-style visual representation that doesn't contain closed surface or volume definitions—physics engines cannot directly perform collision detection or contact solving on it, which is precisely why meshes are needed to take on the physical role.

From iPhone Scans to Manipulable Assets: Pipeline Breakdown

The pipeline flow revealed in the original post goes roughly like this: first, scan a real site with an iPhone to obtain high-quality Gaussian Splats; then convert the splats and meshes into the physics environment; next, select which assets need to be manipulated by the robot; the system decomposes these assets into unit parts based on their assumed real-world behavior and places them at corresponding positions in the splat scene for the robot to manipulate.

There's a notable design decision here: assets aren't simply imported as ready-made models—they're decomposed into units based on "assumed real behavior" and then rebuilt. For example, an object that needs to be grasped gets split into independently processable parts, so the physics engine can apply constraints and collision responses to each part. In other words, it's visually one whole but physically a set of related units.

The physics engine used is MuJoCo Warp, with Gaussian Splats rendered directly inside MuJoCo. The robot platform is Franka Panda. This combination isn't common—MuJoCo traditionally excels at rigid body dynamics, Warp provides GPU acceleration, and putting splat rendering inside MuJoCo means visual rendering and physics stepping happen in the same loop, avoiding the hassle of synchronizing two separate systems.

MuJoCo Warp is a GPU-accelerated implementation of DeepMind's MuJoCo physics engine based on NVIDIA's Warp framework. Warp is a Python framework for GPU computing that compiles simulation kernels into efficient CUDA code, enabling large-scale parallel simulation. Traditional MuJoCo excels at rigid body dynamics and contact solving accuracy, but rendering is typically handled by external engines (such as OpenGL or MuJoCo's own limited renderer). In MuJoCo Warp, physics stepping and rendering both happen on the GPU; if Gaussian Splat rendering is also connected to the same loop, it avoids frequent data synchronization and coordinate system conversions between the physics engine and visual renderer—particularly beneficial for policy training that requires large numbers of randomized variants.

The Value of This Work and Open Questions

From a robot learning perspective, this approach addresses a long-standing pain point: the sim-to-real visual gap. Building simulation environments from real-site scan data is closer to deployment conditions than manually constructing animated scenes; augmentability preserves the flexibility of domain randomization. Combined, these theoretically make the world a policy sees in simulation closer to the world it will encounter on real hardware.

However, the original post doesn't provide quantitative comparison data, such as changes in success rates when transferring policies from this simulation to real robots. The pipeline going through hundreds of iterations indicates considerable engineering complexity—the quality ceiling of iPhone scanning, lighting consistency in splat scenes, and the degree of automation in asset decomposition are all variables that will be encountered in practical deployment. The poster expressed willingness to share technical stack details and welcomes other teams to exchange their robot simulation approaches, indicating this solution is still in early exploration stages and community collaboration may be an important source of future progress.

For teams currently building robot simulation environments, this approach offers a direction worth considering: you don't have to choose between visual realism and physical accuracy—the hybrid Gaussian Splat plus mesh approach lets both coexist. MuJoCo Warp's GPU-accelerated rendering path is also worth watching.

The sim-to-real visual gap is a classic challenge in robot learning. Domain randomization randomly varies parameters like lighting, textures, and object positions in simulation, forcing policies to learn features robust to visual changes, thereby improving transferability. Traditional approaches randomize over manually constructed animated scenes, but the randomization range is limited by the scene's own realism. Using Gaussian Splats from real-site scans as a base and then augmenting objects, lighting, and layouts essentially elevates the starting point of randomization from a "cartoon world" to a "photorealistic world," theoretically narrowing the shift between training distribution and deployment distribution. However, excessive randomization may introduce unrealistic combinations—how to balance augmentation magnitude remains an empirical question requiring experimentation.