Miles: Large-scale LLM RL post-training with SGLang and Megatron-LM
An RL post-training framework for model teams that connects SGLang rollouts with Megatron-LM in an async pipeline.
GitHub radixark/miles Updated 2026-09-05 Branch main Stars 2.6K Forks 445
Python RL post-training SGLang Megatron-LM LLM/VLM NVIDIA and AMD GPUs

🧭 Decision Guide

Try it if you

  • You need SGLang for high-throughput rollouts and Megatron-LM for trillion-parameter model training
    The README About section states that SGLang handles rollouts and Megatron-LM handles scalable training, with the largest models using Megatron-LM.
  • You need low-precision RL training such as MXFP8, NVFP4, or INT4 QAT
    The README Performance section lists MXFP8, NVFP4, FP8, INT4 QAT, BF16, and FP16.
  • Your models or environments include DeepSeek-V4, Kimi-K3, Inkling, Nemotron, or AMD MI355X
    The README What Miles runs and News sections list these models and DeepSeek-V4 Flash RL on AMD Instinct MI355X.
  • You need multi-turn agentic rollouts, TITO, or in-place recovery after an SGLang engine failure
    The README Performance and Correctness and resilience sections list agentic rollout, TITO, and fault tolerance.

Skip it if you

  • You want the largest-model training path to use only the HuggingFace implementation with PyTorch FSDP2
    The README About section says FSDP2 is available, but the recipes, parallelism, and largest models are all on Megatron-LM.
  • Your hardware is outside the NVIDIA or AMD GPU families listed in the README
    The README What Miles runs section lists only GB300, GB200, B300, B200, H200, H100, A100, MI300X, MI325, MI350, and MI355X.
  • You require a mature multi-release ecosystem while the project is at v0.1 with one release
    Project metadata shows the latest version is v0.1 and the release count is 1; the README News marks Miles v0.1 as released in 2026/08.

Requirements

  • SGLang is required as the rollout component; the README About section says it “pairs SGLang for high-throughput rollout.”
  • Megatron-LM is required for the largest models and main parallel training recipes; the README says “the recipes, the parallelism, and the largest models all live on Megatron-LM.”
  • Hardware must be checked per GPU in Installation hardware requirements; the README lists NVIDIA GB300, GB200, B300, B200, H200, H100, A100 and AMD MI300X, MI325, MI350, MI355X.
  • Available training backends include Megatron-LM and PyTorch FSDP2; the README provides a Training Backends section.
  • For low-precision paths, the available choices include MXFP8, NVFP4, FP8, INT4 QAT, BF16, and FP16.

Watch out

  • Do not treat FSDP2 as the main largest-model path; the README explicitly places the largest models and parallelism recipes on Megatron-LM.
    README About section: FSDP2 is for training the HuggingFace implementation as-is, while the recipes, parallelism, and largest models are on Megatron-LM.
  • GPU compatibility cannot be determined by vendor alone; check the per-GPU status and container image in Installation.
    The README What Miles runs section directs users to Installation for per-GPU status and the container image.
  • If R3 is not enabled, the README's Rollout Routing Replay behavior is not automatically available for MoE training.
    The README Correctness and resilience section describes R3 as a distinct capability that records and replays expert routing.

Alternatives

  • slime:Use slime when you need to adopt the upstream project from which Miles was forked directly.
    README Acknowledgment
  • PyTorch FSDP2:Use the FSDP2 backend listed in the README when you want to train the HuggingFace implementation as-is instead of following the Megatron-LM path for the largest models.
    README About

Not stated in the README

  • The README materials do not provide an installation command, Python version, or CUDA/ROCm version.
  • The README materials do not provide memory requirements, minimum GPU count, or exact parallel configurations for each GPU model.
  • The README materials do not provide a version compatibility matrix for SGLang, Megatron-LM, and PyTorch FSDP2.
  • The README materials do not provide v0.1 performance benchmarks, throughput numbers, or quantitative comparisons with slime.
  • The README materials do not explain the responsibilities, issue response time, or production support period associated with the 10 contributors.
  • The README materials do not include a copyable command from Quick Start.

💡 Deep Analysis

7
No I have only one consumer GPU, normally use Hugging Face Trainer for ordinary SFT, and do not need SGLang rollout, Megatron-LM, or agent sandboxes. Is Miles worth adopting?
For: A model developer with only one consumer GPU who mainly uses Hugging Face Trainer for ordinary SFT and does not maintain distributed rollout, agent sandboxes, or a Megatron-LM cluster

No. Miles targets enterprise-grade, large-scale LLM/VLM post-training, while your workload does not need its distributed rollout-and-training loop.

  • The About section positions the project as “enterprise-ready” for “large-scale model post-training,” centered on SGLang rollout and Megatron-LM training.
  • Although Miles provides PyTorch FSDP2, SFT, and a path to train Hugging Face implementations as-is, FSDP2 is still a backend within a large training framework rather than a lightweight single-GPU SFT tool.
  • Agentic rollout, P2P RDMA, asynchronous scheduling, and fault tolerance address multi-worker, cluster-communication, and long-running RL problems; they offer little corresponding value for ordinary single-GPU SFT.
  • The README explicitly says that the largest models, main recipes, and parallelism live on Megatron-LM.

Adopting Miles would therefore likely add installation, dependency, and debugging overhead instead of shortening your SFT workflow.

  • README title: "Enterprise-Grade Reinforcement Learning for Large-Scale Model Post-Training"
  • About: "enterprise-ready reinforcement learning framework for large-scale model post-training"
  • About: "A PyTorch FSDP2 backend is available"
  • About: "the recipes, the parallelism, and the largest models all live on Megatron-LM"
Not stated in the README:The README does not specify whether there is a supported single-GPU installation or minimum hardware configuration;The README provides no resource-overhead comparison between ordinary Hugging Face SFT and Miles' FSDP2 backend
Yes I need to train coding agents and computer-use agents using Harbor, HUD, or NeMo Gym, with tasks running in AgentENV, Daytona, E2B, or Modal sandboxes. Can Miles cover multi-turn rollout and environment integration?
For: A research team training coding agents and computer-use agents with Harbor, HUD, or NeMo Gym connectors and AgentENV, Daytona, E2B, or Modal sandboxes

Yes. The README directly lists multi-turn agentic rollout, environment connectors, and task sandboxes as supported capabilities.

  • What Miles runs says coding and computer-use agents can be trained through Harbor, HUD, NeMo Gym, OpenEnv, and Verifiers connectors.
  • The same section lists AgentENV, Daytona, E2B, and Modal as task sandboxes, indicating an integration path between environment execution and rollout.
  • Performance says the router behind SGLang distributes requests, preserves per-request metadata, and health-checks the engine fleet, with tuning for multi-turn agentic workloads.
  • Correctness and resilience states that TITO is supported for “every model and every black-box harness,” reducing detokenize/retokenize issues between rollout and training.

Miles provides the training-loop and environment-integration infrastructure, not the quality of your reward function, tools, or task definitions. The README does not describe the maturity or feature differences of each connector.

  • What Miles runs: "Train coding and computer-use agents through connectors for Harbor, HUD, NeMo Gym, OpenEnv, Verifiers"
  • What Miles runs: "task sandboxes on AgentENV, Daytona, E2B, or Modal"
  • Performance: "preserves per-request metadata" and "Tuned for multi-turn agentic workloads"
  • Correctness and resilience: "Supported for every model and every black-box harness"
Not stated in the README:The README does not state which tool calls, state persistence, or concurrency modes are supported by Harbor, HUD, and NeMo Gym individually;The README does not describe deployment cost, network isolation, timeout behavior, or failure-reporting formats for each sandbox;The README provides no end-to-end metrics for coding-agent or computer-use-agent workloads
Yes I use Megatron-LM to train an MoE model and already face expert-routing mismatches between rollout and training, along with token-boundary changes caused by repeated tokenization of multi-turn samples. Can Miles directly address these correctness risks?
For: A Megatron-LM training engineer responsible for MoE RL stability and dealing with expert-routing mismatch and token-boundary issues between rollout and training

Yes. Miles’ TITO and R3 directly target token consistency and MoE routing consistency, respectively.

  • The Correctness and resilience section says TITO preserves token-level data between rollout and training, avoiding the detokenize/retokenize round trip, and claims support for every model and every black-box harness.
  • The same section says R3 records expert routing during rollout and replays it in the trainer’s forward pass to remove MoE routing mismatch.
  • The README also says R3 overlaps computation and communication to reduce the added cost of consistency handling.
  • These capabilities are aligned with the Megatron-LM training path and large-scale MoE RL correctness requirements.

However, “supported” does not guarantee seamless behavior with a custom tokenizer, special stop conditions, or a custom harness. The README does not list covered MoE architectures or provide a TITO validation example for your multi-turn message format.

  • Correctness and resilience: "no detokenize/retokenize round-trip between rollout and training"
  • Correctness and resilience: "Expert routing recorded during rollout is replayed in the trainer's forward pass"
  • Correctness and resilience: "Supported for every model and every black-box harness"
  • Correctness and resilience: R3 uses "compute and communication overlapped"
Not stated in the README:The README does not list the specific MoE architectures, expert counts, or parallelism combinations supported by R3;The README does not specify TITO interface constraints for custom tokenizers, stop conditions, or multi-turn message assembly;The README provides no stability or performance comparison before and after enabling R3/TITO
Yes I use Megatron-LM to train Kimi-K2.6 on an NVIDIA H100 cluster. I need to decouple SGLang rollout workers from training workers and rapidly synchronize trillion-parameter weights in a disaggregated deployment. Is Miles suitable for this production pipeline?
For: An infrastructure engineer training Kimi-K2.6 or other trillion-parameter models with Megatron-LM on an NVIDIA H100 cluster, requiring separate rollout and training resources and fast weight synchronization

Yes. Miles is specifically designed for large-scale, disaggregated RL post-training.

  • The About section pairs SGLang for high-throughput rollout with Megatron-LM for scalable training and states that rollout and training workers can be decoupled.
  • The Performance section says new weights can reach inference engines in-loop and identifies P2P RDMA as the fast path for disaggregated deployments. It also cites Kimi-K2.6 as a trillion-parameter example.
  • Fully async RL provides configurable on-policy and off-policy scheduling, which can reduce pipeline bubbles caused by rollout and training waiting on each other.

Whether it reaches the desired throughput still depends on GPU topology, RDMA networking, containers, drivers, and parallelism settings. The README does not provide end-to-end measurements for your exact cluster size.

  • About: "pairs SGLang for high-throughput rollout with Megatron-LM for scalable training"
  • Performance: "P2P RDMA as the fast path for disaggregated setups"
  • Performance: "even on a trillion-parameter model such as Kimi-K2.6"
  • Project data: latest release is v0.1.0, with only 1 formal release
Not stated in the README:The README does not specify the recommended GPU count, network bandwidth, RDMA configuration, or actual weight-update time for an H100 cluster;The README does not specify Kimi-K2.6 stability or throughput with your Megatron-LM parallelism configuration
Yes I run multi-day SGLang rollout and RL training on an NVIDIA H200 cluster. My main concern is that a single inference-engine failure will restart the entire job. Can Miles' fault-tolerance mechanism meet this constraint?
For: An enterprise RL platform engineer running SGLang rollout on an NVIDIA H200 cluster for multi-day jobs and concerned that inference-engine failures will restart the entire training run

Yes, at least for the SGLang engine-failure scenario described in the README, Miles provides in-place recovery instead of restarting the entire run.

  • The Correctness and resilience section explicitly says that when an SGLang engine dies, Miles recovers it and resumes the run in place, with “no restart, no pause.”
  • The Performance section says rollout requests pass through a router that health-checks the engine fleet, providing architectural support for detecting unhealthy instances.
  • NVIDIA H200 is listed as supported hardware, and Fully async RL decouples rollout, training, and evaluation workers for long-running distributed pipelines.
  • This capability targets inference-engine failures; it does not imply automatic recovery from training-worker, RDMA-network, GPU, or sandbox failures.

Miles is therefore closely aligned with your failure constraint. The README does not specify recovery time, recoverable failure classes, state persistence, or behavior when multiple engines fail simultaneously.

  • Correctness and resilience: "When an SGLang engine dies, Miles recovers it and resumes the run in place: no restart, no pause"
  • Performance: the router "health-checks the fleet"
  • What Miles runs: the hardware list includes NVIDIA H200
  • Performance: "Rollout and training workers are decoupled"
Not stated in the README:The README does not specify average or worst-case SGLang engine recovery time or whether spare instances are required;The README does not define how training checkpoints, rollout state, and request metadata are persisted during recovery;The README does not state whether training-worker, GPU, RDMA, or agent-sandbox failures receive the same level of automatic recovery
It depends I maintain Hugging Face model implementations, and my team wants to use PyTorch FSDP2 directly for RL post-training instead of migrating to Megatron-LM. Is Miles suitable if we need GRPO, SFT, or on-policy distillation?
For: A model engineer maintaining Hugging Face model implementations who does not want to rewrite them for Megatron-LM and plans to use PyTorch FSDP2 for LLM post-training

It depends. Miles offers an FSDP2 path, but the README explicitly places its largest models, main recipes, and parallelism capabilities on the Megatron-LM path.

  • The About section says FSDP2 is available for runs that want to “train the HuggingFace implementation as-is,” reducing the need to rewrite the model.
  • The same section states that the recipes, parallelism, and largest models live on Megatron-LM; FSDP2 should not be treated as a fully equivalent backend.
  • What Miles runs lists GRPO, GSPO, PPO, REINFORCE++, SFT, and on-policy distillation, but does not map each capability to FSDP2 support.

FSDP2 is reasonable if the model size is moderate and the target recipes are covered by that path. For the largest models or Megatron-specific parallelism, it is not the safe default. The README lacks a detailed recipe-by-backend compatibility matrix.

  • About: "A PyTorch FSDP2 backend is available for runs that would rather train the HuggingFace implementation as-is"
  • About: "the recipes, the parallelism, and the largest models all live on Megatron-LM"
  • What Miles runs: "GRPO, GSPO, PPO, and REINFORCE++ ... plus SFT and on-policy distillation"
  • Project data: the latest project version is v0.1.0
Not stated in the README:The README does not list per-recipe support for GRPO, SFT, and on-policy distillation on the FSDP2 backend;The README does not state whether the target Hugging Face model has an FSDP2 example, checkpoint format, or performance baseline
It depends I train DeepSeek-V4 with ROCm on an AMD Instinct MI355X cluster and want to use Miles' low-precision RL capabilities. Is the project's AMD MI355X and MXFP8/NVFP4 support sufficient for adoption?
For: A hardware platform engineer training DeepSeek-V4 with ROCm on an AMD Instinct MI355X cluster and evaluating MXFP8, NVFP4, and conventional BF16

It depends. The README explicitly demonstrates AMD MI355X and DeepSeek-V4 support, but it does not establish equal stability for every low-precision combination on AMD.

  • The News section records “DeepSeek-V4 Flash RL training” on AMD Instinct MI355X with Miles, which is directly relevant evidence for the hardware and model.
  • What Miles runs lists AMD MI300X, MI325, MI350, and MI355X through ROCm.
  • Performance lists MXFP8, NVFP4, FP8, INT4 QAT, BF16, and FP16. However, NVFP4 is presented in a broader Blackwell/NVIDIA context elsewhere, so full MI355X support cannot be inferred.
  • The project has only one formal release, v0.1.0, so stability across specific hardware, ROCm, kernels, and precision combinations remains uncertain.

There is clear evidence for AMD MI355X plus DeepSeek-V4. If MXFP8 or NVFP4 is mandatory, the README is insufficient for a procurement or production decision.

  • News: "DeepSeek-V4 Flash RL training comes to AMD Instinct MI355X with Miles"
  • What Miles runs: "AMD MI300X, MI325, MI350, and MI355X via ROCm"
  • Performance: "MXFP8 and NVFP4 ... FP8, INT4 QAT, BF16, and FP16 are also supported"
  • Project data: release_count is 1 and latest_release is v0.1.0
Not stated in the README:The README does not provide per-feature support status for MXFP8 and NVFP4 on MI355X/ROCm;The README does not specify the required ROCm, driver, container, or kernel versions for the AMD path;The README provides no low-precision throughput, memory, or training-stability data for DeepSeek-V4 on MI355X

✨ Highlights

  • Pairs SGLang rollouts with Megatron-LM training for trillion-parameter models
  • Supports MXFP8, NVFP4, FP8, and INT4 QAT training
  • TITO covers every model and black-box harness, avoiding re-tokenization
  • Provides day-0 support for DeepSeek-V4, Kimi-K3, and Inkling
  • It has only one release, v0.1, and 10 contributors

🔧 Engineering

  • Routes multi-engine rollouts with SGLang and performs scalable training with Megatron-LM
  • Provides async RL, P2P RDMA weight transfer, and SGLang engine recovery
  • Covers GRPO, GSPO, PPO, REINFORCE++, SFT, and on-policy distillation

⚠️ Risks

  • The largest models and parallelism recipes rely on Megatron-LM rather than the FSDP2 path
  • The project is at v0.1 with only one release and 10 contributors
  • GPU support must be checked per device in Installation; the README gives no single hardware threshold

👥 For who?

  • Teams running large-scale LLM post-training with SGLang and Megatron-LM
  • RL infrastructure teams using NVIDIA GB200/H100 or AMD MI355X hardware
  • Teams training frontier models such as Kimi-K3, DeepSeek-V4, or Inkling