Overview

This repository provides the Reinforcement Learning (RL) training environments for Microduck, a compact ~800g bipedal robot, built upon the mjlab (MuJoCo Warp) framework. Leveraging Proximal Policy Optimization (PPO), these environments facilitate the training of robust control policies that are then exported to ONNX and deployed on the physical Microduck robot via the runtime in the pollen-robotics/microduck project. This collection encapsulates a comprehensive sim2real methodology, incorporating actuator physics, domain randomization, backlash simulation, and crucial reward design principles.

Key Features

• Full Sim2Real Recipe: Implements advanced techniques such as BAM actuator physics, domain randomization, and backlash simulation to bridge the gap between simulation and real-world robot performance.
• Diverse Training Environments: Offers a rich set of ~15 distinct tasks, ranging from core walking and standing to specialized maneuvers like roulades, ground picking, and roller-skating. Backlash variants are available for all major tasks, simulating gear play for enhanced robustness.
• High-Fidelity Actuator Modeling: Utilizes the BAM M6 actuator model, capturing crucial dynamics like voltage control, back-EMF, and friction, with per-environment domain randomization on various parameters.
• Flexible Robot Models: Employs multiple MJCF robot configurations tailored for different task families (e.g., walking, all-collision, roller-skating), exported from Onshape using onshape-to-robot.
• Unified Observation Contract: All policies share a common 61-dimensional observation space, enabling seamless hot-swapping of control policies on the deployed robot at runtime.
• GPU-Accelerated Training: Optimized for training on CUDA GPUs using MuJoCo Warp, with an option to leverage Hugging Face Jobs for GPU-less execution.
• Integrated Deployment Workflow: Includes scripts for exporting trained policies to ONNX and a command-line interface for running them on the physical robot, simulating real-world deployment scenarios.

Typical Use Cases

• Bipedal Locomotion Research: Train and deploy advanced walking gaits and balancing strategies for the Microduck robot on various terrains, including flat and rough surfaces.
• Robotic Manipulation and Interaction: Develop policies for complex tasks like standing up from various positions, sitting down, picking objects from the ground, and kicking a ball.
• Agile Maneuver Development: Explore and train policies for dynamic skills such as forward rolls (roulades) and intricate roller-skating patterns.
• Robustness and Resilience Testing: Evaluate and enhance robot performance in the presence of simulated hardware imperfections like gear backlash, command delays, and varying friction.
• Sim2Real Transfer Learning: Serve as a foundational toolkit and demonstration of effective techniques for transferring learned policies from simulation to real-world robotic systems.