Case Studyllmtext to speechh100mlops
SAIL Optimizes Orpheus-TTS For Higher Throughput
8.9
SAIL evaluated the publicly available Orpheus-TTS deployment (served via Baseten) and applied system-level optimizations to characterize and improve real-time inference performance. Baseline sustained about 24 concurrent real-time streams per NVIDIA H100 GPU, and after optimizations sustained 216 streams (~10×), reducing equivalent annual accelerator spend from about $1.4M to $140k for a 100-GPU capacity.
Key Points
- 1Demonstrates ~10× throughput increase: 24 to 216 concurrent real-time streams per H100 GPU
- 2Highlights system-level optimizations (scheduling, pipeline coupling) as dominant over model-level tweaks for latency
- 3Enables identical service capacity using ~10 H100 GPUs versus 100, cutting annual accelerator spend by 90%
Scoring Rationale
Strong empirical 10× throughput improvement across production-like TTS pipelines; limited by single-source internal evaluation and lack of external replication.
Sources
Public references used for this report.
Practice with real Logistics & Shipping data
90 SQL & Python problems · 15 industry datasets
Used by DS/ML engineers at top companies
High-Value Overnight OrdersEasyDelivered International ShipmentsMediumOn-Time Delivery Rate by CarrierHard
250 free problems · No credit card
See all Logistics & Shipping problems
