Clockwork Deploys TorchPass To Avoid Training Restarts

According to a March 11, 2026 press release, Palo Alto startup Clockwork.io made its TorchPass capability generally available as a software fault-tolerance layer for distributed training. TorchPass uses what the company calls "Live GPU Migration" to move workloads off failing GPUs or nodes without a checkpoint restart. Clockwork CEO Suresh Vasudevan said, "We built TorchPass to fundamentally reject that premise. Instead of treating failure as inevitable and restarting after the fact, TorchPass makes infrastructure faults invisible to the workload - training continues through failures transparently, in software." Per the company's figures, a 2,048-GPU H200 deployment could recover over $6 million a year in compute otherwise lost to failure-driven restarts. Fierce Network reports TorchPass completes failure recovery in under two minutes versus multi-hour recompute after a typical checkpoint rollback, and that Clockwork has raised over $40 million, with customers including Nscale, DCAI, and Nebius.

Live migration for GPU-accelerated training combines fast state transfer of model weights and optimizer state, tight clock synchronization across devices to preserve distributed state, and early failure detection to avoid corrupted checkpoints. Coverage credits Clockwork's nanosecond-level clock synchronization, which the company traces to Stanford research, with underpinning both its failure detection and its migration triggers.

SemiAnalysis has noted AWS publishes a faster checkpointless-recovery figure - about 1 minute 45 seconds, versus roughly 15 minutes for a standard checkpoint restart - putting Clockwork's under-two-minute claim in a competitive but not clearly leading position among vendors solving the same problem. Teams running multi-thousand-GPU clusters have historically treated periodic job rollbacks as a fixed operational cost; software that cuts recovery from hours to minutes changes that assumption regardless of which vendor is fastest.

Engineers and SREs evaluating this class of tool should weigh the complexity of adding a migration layer against simply tuning checkpoint frequency and preemption tolerance, and should ask vendors, Clockwork included, for reproducible, framework-specific benchmarks - PyTorch DDP, ZeRO, and similar - rather than relying on marketing figures alone.

Independent, reproducible recovery-latency benchmarks; whether customers publish real utilization gains; and how TorchPass's claims compare once AWS and other checkpointless-recovery vendors publish their own head-to-head numbers. The savings and recovery figures in this story originate from Clockwork's own press materials and have not yet been independently validated.