SageMaker AI Outlines Multi-turn Reinforcement Learning Best Practices

AWS launched multi-turn reinforcement learning on Amazon SageMaker AI on June 3, 2026, a serverless model-customization technique that trains models against a full agent trajectory (a complete multi-step task) rather than single-turn outputs, so models learn which decisions earlier in a task actually mattered. Agents can run on Amazon Bedrock AgentCore Runtime for fully managed hosting, or on Amazon EKS, Amazon EC2, AWS Fargate, or any framework; SageMaker AI manages the full training loop, from rollout orchestration and trajectory collection to training and checkpoint management, with built-in MLflow tracking and evaluation jobs that report reward, pass@k, and trajectory metrics. It runs as a fully serverless, pay-per-token capability with no infrastructure to provision. Supported models at launch include Qwen 3.6 27B, Nova Lite 2.0, GPT-OSS-20B, and Gemma 31B, available in US East and US West regions. AWS illustrates the workflow's evaluation methodology using SOP-Bench, its own benchmark of complex, multi-step standard operating procedures spanning roughly a dozen business and industrial domains including healthcare, logistics, and finance, used to test whether agents can complete realistic, tool-using workflows.

SageMaker AI's algorithm support spans standard PPO, the more recent CISPO loss (from the MiniMax-M1 line of work), and several methods for computing group-based advantages, including GRPO, RLOO, and variants tuned for sparse-reward tasks. AWS's published hyperparameter reference gives unusually specific tuning guidance: PPO's clipping range (0.8 to 1.2) is described as the safe default for a first run, while CISPO requires much wider, asymmetric clipping (1.0 low, 6.0 high) because it lets bad-action probabilities decrease freely and relies only on the upper bound for stability; CISPO-based runs are noted to be prone to collapse between steps 40 and 80. The guide also details how sparse-reward tasks, where many rollouts in a group score identically, dilute gradient signal and require a lower learning rate or larger batch/group size, and how off-policy staleness in asynchronous training should be set to zero when diagnosing training collapse.

The published tuning guide is a genuinely actionable checklist: start with PPO and default clipping for a first run before moving to CISPO; size the per-turn token cap so that max turns times tokens-per-turn plus expected tool output plus prompt length stays within the model's context window, since truncated rollouts teach the model to associate incomplete attempts with bad outcomes; monitor response-length metrics to catch silent truncation; and treat a rollout permanent-failure rate above roughly 1 percent as an environment bug rather than a retry-count problem. AWS frames pass@1 as the headline evaluation metric and pass@G (G equal to group size) as a sanity check for whether prompts are calibrated to the right difficulty.

Track community adoption of the adapter model for connecting external agent frameworks, whether SOP-Bench or similar externalized evaluation suites become a standard for multi-turn agent benchmarking, and how AWS's specific numeric guidance (clipping ranges, staleness thresholds) compares with practices at other RL infrastructure providers as more teams publish their own tuning results.

This is AWS's own documentation and benchmark, describing its own product; the specific numeric defaults and thresholds reflect Amazon's internal experience but have not been independently validated across other agent environments or model families.