Inside Miles v0.1: The Production-Grade RL Post-Training Architecture for Trillion-Parameter MoE Models

In August 2026, the artificial intelligence landscape experienced a quiet but monumental shift: while frontier labs release massive open-weight models weekly, the real engineering battleground has shifted from pre-training to post-training Reinforcement Learning (RL). Aligning 700B+ Mixture-of-Experts (MoE) models via multi-turn terminal feedback, compiler checks, and self-play reasoning requires running asynchronous policy updates across hundreds of GPUs.

However, training massive MoE models with RL has historically suffered from a fatal systems flaw: training-inference divergence. Subtle numerical discrepancies between the high-throughput inference engine (rollout) and the distributed training framework cause expert routing decisions to flip during backward passes, distorting gradient updates and destabilizing policy optimization.

To solve this, LMSYS Org and RadixArk released Miles v0.1—a production-grade reinforcement learning post-training framework that seamlessly unifies SGLang for async multi-turn rollouts, NVIDIA Megatron-LM (and PyTorch FSDP2) for distributed training, and Ray for cluster orchestration.

Here is an architectural deep dive into Miles v0.1, its Rollout Routing Replay (R3) mechanism, and how it enables asynchronous RL on trillion-parameter models.

1. The Miles RL Post-Training Architecture

Miles restructures the traditional RL loop into an asynchronous pipeline optimized for long-horizon agentic trajectories:

1. Rollout (SGLang): Generates multi-turn agent interactions in sandboxed environments, leveraging RadixAttention to achieve over 96% prefix-cache hit rates.
2. R3 Capture: Records the exact expert gating choices made by SGLang for every token.
3. Training (Megatron-LM): Consumes trajectory batches, replays the exact recorded routes, and computes policy updates without routing divergence.
4. Weight Sync: Streams updated policy weights back to the SGLang fleet asynchronously without stalling in-flight agent sessions.

2. Solving MoE Divergence: Rollout Routing Replay (R3) & TITO

In dense model RL, slight numerical variations between rollout and training engines are often tolerated. In sparse MoE models, however, a tiny difference in router logits flips the top-$k$ expert assignment:

• Gradient Misallocation: Gradients intended for Expert 4 get applied to Expert 12.
• Catastrophic Policy Drift: The training engine updates expert weights that were never used during the rollout pass.

A. Rollout Routing Replay (R3)

Miles records SGLang's expert routing choices during the forward rollout pass and replays them directly in Megatron-LM, ensuring bit-exact alignment between the model that experienced the reward and the model receiving gradients.

B. Token-In-Token-Out (TITO)

Miles preserves raw token IDs generated by the model rather than de-tokenizing to text. This eliminates chat-template re-serialization bugs that alter token boundaries between agent turns.

3. Mathematical Optimization: Memory Scaling & NVMe Streaming

Training a 744B-parameter MoE asynchronously across multiple nodes creates massive GPU High-Bandwidth Memory (HBM) pressure:

For a 744B model in FP8, Adam optimizer states alone consume over 2.9 Terabytes of memory.

Miles implements Asynchronous NVMe Bucket Streaming:

• Optimizer states are offloaded to node-local NVMe SSDs or host CPU memory.
• During each optimizer step, parameter buckets are streamed to GPU SRAM on-demand and flushed back immediately:

In reference benchmarks, this optimization saves 30+ GB of HBM per GPU, allowing an asynchronous training engine for a 744B model to fit on just 32 NVIDIA GB300 GPUs.

4. Reference Benchmark: Training GLM-5.2 (744B MoE) on Terminal Tasks

LMSYS verified Miles v0.1 by training a 744B-parameter GLM-5.2 model on terminal coding workflows across a 64x NVIDIA GB300 cluster (32 GPUs for SGLang rollout, 32 GPUs for Megatron-LM training):

5. Frequently Asked Questions (FAQ)

What is the relationship between Miles and slime?

Miles v0.1 was developed as a production fork of THUDM's slime, restructuring the codebase for trillion-parameter MoE scale, SGLang rollout integration, and low-precision Blackwell recipes.

Can Miles run on AMD Instinct GPUs?

Yes. Miles includes native AMD ROCm support (MI300X and MI355X) via HIP and RCCL, running the identical training and SGLang rollout pipeline verified under continuous integration.

6. Conclusion

As frontier artificial intelligence advances toward self-improving reasoning agents, the infrastructure that powers post-training reinforcement learning is just as critical as pre-training superclusters. Miles v0.1 by LMSYS and RadixArk establishes the open-source industry standard for scaling stable, verified, and low-precision RL to trillion-parameter models.

(Cover Image Courtesy: Unsplash / High-Density Datacenter Supercomputing & GPU Interconnect)