Goodput Reframes LLM Serving Performance Benchmarks
For practitioners, LLM capacity tests that optimize only requests or tokens per second can obscure whether interactive workloads are meeting user-facing latency objectives. CNCF contributor Graziano Casto argues that goodput, completed requests that satisfy latency targets, is a more useful serving metric than raw throughput alone. The article describes tests serving Qwen2.5-7B with vLLM on a single NVIDIA A10G in Amazon EKS, with Prometheus, Grafana, and NVIDIA DCGM metrics. It identifies time to first token (TTFT) and time per output token (TPOT) as central SLOs. Supporting material from Anyscale and Hao AI Lab likewise distinguishes interactive workloads, where latency matters, from batch workloads that may prioritize aggregate throughput.
For practitioners, goodput provides a closer link between infrastructure utilization and application-level service quality than raw requests per second. Industry-pattern observations: a serving stack can complete more requests while failing a larger share of latency objectives as queues, batching, and contention increase. That distinction matters most when teams benchmark chat, code-assistant, or other streaming applications under concurrent load.
CNCF contributor Graziano Casto defines goodput as the number of completed requests per second that also meet latency targets. In the July 20 post, Casto argues that throughput alone can rise while perceived service quality deteriorates, because completed requests that breach latency targets still count toward throughput. The post identifies time to first token (TTFT) and time per output token (TPOT) as the key targets for LLM serving.
Casto's reported setup served `Qwen2.5-7B` through `vLLM` on one NVIDIA A10G GPU with 24 GB of memory in an Amazon EKS cluster. The benchmarks used Prometheus and Grafana for monitoring, plus NVIDIA's DCGM exporter for GPU-level metrics. Casto notes that the repository, including Kubernetes manifests, is available on GitHub for reproduction.
Why the metric changes benchmark design
Editorial analysis
goodput makes an SLO an explicit condition of performance rather than a separate dashboard panel. A useful test should therefore record both aggregate capacity and the fraction of requests satisfying workload-specific thresholds. Reporting only tokens per second, requests per second, or HTTP success rates can conceal an overloaded interactive service when late responses remain technically successful.
Anyscale's LLM serving documentation similarly distinguishes interactive applications from batch workloads. It states that chatbots and coding assistants benefit most from low TTFT and low inter-token latency, while large-scale batch generation may favor high throughput and cost efficiency. The documentation also notes that prompt length, queueing under load, autoscaling behavior, and the compute-intensive prefill stage can affect TTFT.
Hao AI Lab's 2024 DistServe post describes goodput as completed requests per second that adhere to service-level objectives. It argues that applications have different latency requirements, citing chat interactions that need rapid initial responses and code completion that needs fast end-to-end generation. The post presents prefill-decode disaggregation, which assigns prefill and decode work to separate GPUs, as one approach evaluated for tighter latency constraints.
Applying goodput to production tests
For practitioners
benchmark suites commonly need workload classes rather than one global latency target. Prompt and output length distributions, streaming behavior, concurrency, queue depth, and traffic mix can each change the balance between TTFT, TPOT, end-to-end latency, and throughput. A batch summarization job and an interactive coding assistant can therefore produce very different conclusions from the same server configuration.
A practical reporting set can include:
- •Raw throughput, in requests or tokens per second
- •Goodput, defined against explicit TTFT and TPOT SLOs
- •Latency percentiles for TTFT, inter-token latency, and end-to-end completion
- •SLO compliance segmented by workload class, prompt length, and concurrency
Industry context
prefill and decode place different demands on accelerators and memory. Systems research and vendor material have consequently examined scheduling, resource partitioning, and prefill-decode separation as ways to improve SLO-compliant capacity, not merely total output. The central operational question is not whether a server can produce more tokens, but how much traffic it can serve within the latency budgets that the application actually promises.
Key Points
- 1Goodput counts completed LLM requests meeting latency SLOs, exposing service degradation that raw throughput and HTTP success rates can hide.
- 2CNCF's example uses vLLM, Qwen2.5-7B, EKS, and an A10G to illustrate SLO-aware benchmarking with observability tooling.
- 3Interactive and batch LLM workloads require different performance targets, so industry practice favors segmented SLOs over one aggregate capacity metric.
Scoring Rationale
The article addresses a practically important benchmarking distinction for teams operating LLM inference workloads. It is an explanatory post rather than a new serving system, model release, or broad infrastructure announcement, which limits its industry-wide impact.
Sources
Primary source and supporting public references used for this report.
Practice interview problems based on real data
1,625 SQL & Python problems across 15 industry datasets — the exact type of data you work with.
Try 250 free problems
