UniWALM
A unified world-action-language model for robot control. Within a single mixture-of-transformers backbone, it equips the video, action, and language modalities with diffusion heads for synchronous generation, enabling closed-loop decision-making in which language serves as memory.
Timeline (last 90 days)
UniWALM places the three modalities of video, motion, and language in the same mixture-of-transformers backbone to share representations, with each modality equipped with a dedicated diffusion head and denoised synchronously.
UniWALM simultaneously decodes subtask descriptions and textual memory in the same forward pass that generates the motion trajectory, without requiring a separate planning stage.
UniWALM adopts a closed-loop 'language as memory' mechanism, in which text generated by the model itself is re-consumed in the next step as context for subsequent decisions.
Flow Matching with Correction (FMC) is the training objective introduced by UniWALM, which teaches the model during training to recover from perturbed motion trajectories and adds no extra computational cost at inference time.
On the compositional pick-and-place task MolmoSpaces, UniWALM consistently outperforms the same-backbone baseline FastWAM.
On the memory-dependent scenario RMBench, UniWALM achieves a 34.5% success rate without any pretrained language or reasoning model, exceeding the 28.5% of Mem-0's keyframe-assisted setting.
In the real-world Find-the-Item task, generated textual memory raises UniWALM's success rate from 11.1% to 61.1%.
The tasks currently demonstrated by UniWALM are concentrated on pick-and-place and item finding, and generalization to more complex long-horizon tasks and open environments still needs verification.
Whether the training cost and stability of a single backbone with multiple diffusion heads still hold at larger scales remains an open question.
All Facts (9)
UniWALM places the three modalities of video, motion, and language in the same mixture-of-transformers backbone to share representations, with each modality equipped with a dedicated diffusion head and denoised synchronously.
50%UnverifiedUniWALM simultaneously decodes subtask descriptions and textual memory in the same forward pass that generates the motion trajectory, without requiring a separate planning stage.
50%UnverifiedUniWALM adopts a closed-loop 'language as memory' mechanism, in which text generated by the model itself is re-consumed in the next step as context for subsequent decisions.
50%UnverifiedFlow Matching with Correction (FMC) is the training objective introduced by UniWALM, which teaches the model during training to recover from perturbed motion trajectories and adds no extra computational cost at inference time.
50%UnverifiedOn the compositional pick-and-place task MolmoSpaces, UniWALM consistently outperforms the same-backbone baseline FastWAM.
50%UnverifiedOn the memory-dependent scenario RMBench, UniWALM achieves a 34.5% success rate without any pretrained language or reasoning model, exceeding the 28.5% of Mem-0's keyframe-assisted setting.
50%UnverifiedIn the real-world Find-the-Item task, generated textual memory raises UniWALM's success rate from 11.1% to 61.1%.
50%UnverifiedThe tasks currently demonstrated by UniWALM are concentrated on pick-and-place and item finding, and generalization to more complex long-horizon tasks and open environments still needs verification.
50%UnverifiedWhether the training cost and stability of a single backbone with multiple diffusion heads still hold at larger scales remains an open question.
50%