Overview

Miles is a powerful, enterprise-grade reinforcement learning framework meticulously engineered for the post-training of large-scale Language Models (LLMs) and Vision-Language Models (VLMs). Forked from and co-evolving with the slime project, Miles delivers the precision, stability, and observability essential for RL at trillion-parameter scale. It seamlessly integrates with SGLang for high-throughput rollouts and Megatron-LM for scalable training, while also offering a PyTorch FSDP2 backend for users who prefer working with HuggingFace implementations.

Key Features

• Fully Asynchronous RL: Decouples rollout and training workers with configurable on/off-policy schedules, optimized pipelines, and flexible async rollout/evaluation modes for maximum efficiency.
• Fast Agentic Rollout: Leverages SGLang, powered by a request-routing engine, to distribute loads, preserve per-request metadata, and ensure fleet health for demanding multi-turn agentic workloads.
• Rapid Weight Updates: Achieves near-instantaneous propagation of new weights (seconds) to engines, even for trillion-parameter models, with P2P RDMA as a high-speed path for disaggregated setups.
• Low-Precision Training: Supports advanced training formats like MXFP8 and NVFP4 with numerically stable recipes to combat precision-induced divergence, alongside FP8, INT4 QAT, BF16, and FP16.
• Efficient LoRA & Multi-LoRA: Enables training of frontier-scale models on significantly fewer GPUs using Low-Rank Adapters, which load directly into SGLang for seamless inference.
• Token-In-Token-Out (TITO): Eliminates detokenize/retokenize round-trips between rollout and training for all supported models and black-box harnesses, ensuring data integrity.
• Rollout Routing Replay (R3): Replays expert routing from rollout into the trainer's forward pass, mitigating MoE routing mismatches and stabilizing large-scale runs with overlapped compute and communication for cost efficiency.
• Robust Fault Tolerance: Gracefully recovers from SGLang engine failures, resuming runs in place without restarts or pauses.
• Day-0 Model Support: Guarantees immediate compatibility with cutting-edge models upon their release, alongside extensive hardware support for both NVIDIA and AMD platforms.
• Broad Recipe & Environment Support: Offers a wide range of RL algorithms (GRPO, GSPO, PPO, REINFORCE++), SFT, on-policy distillation, and connectors for diverse agentic environments and task sandboxes.

Typical Use Cases

Miles is designed for organizations pushing the boundaries of AI through large-scale model optimization. Its enterprise-grade features make it ideal for:

• Fine-tuning Foundational Models: Enhancing the capabilities of massive LLMs and VLMs for specific downstream tasks or improved performance characteristics using reinforcement learning.
• Developing Advanced AI Agents: Training sophisticated agents for complex interactions in simulated or real-world environments, leveraging its robust agentic rollout and environment connectors.
• Rapid Prototyping and Deployment: Quickly integrating and post-training new frontier models with its day-0 support and efficient training mechanisms.
• Research and Development at Scale: Enabling researchers and engineers to experiment with novel RL algorithms and distributed training strategies on massive parameter counts.
• Optimizing Inference and Training Costs: Utilizing low-precision training and efficient weight update mechanisms to reduce computational resources and accelerate the training lifecycle.