Google Details Qwen3.5 Serving Optimizations for Ironwood TPUs
Google's TPU performance team reports a new serving stack for Qwen3.5-397B-A17B on one Ironwood host. The model has 397 billion total parameters and activates 17 billion per token. Between April and June 2026, Google measured approximately 3.1x higher decode-heavy performance and 4.7x higher prefill-heavy performance at the 512-concurrency tier after changing attention sharding, expert routing, memory layouts, and custom JAX/Pallas kernels. The engineering work is unusually well documented through linked open-source pull requests, but the headline gains remain a vendor comparison across evolving internal stacks. The report does not provide independent reproduction, price-normalised cost, end-user latency distributions, or a comprehensive quality evaluation, so teams should treat it as a reproducible optimization playbook rather than a universal hardware ranking.
Google has published a detailed systems-engineering report on serving Qwen3.5-397B-A17B with Ironwood TPUs. The central idea is not one magic kernel: it is a modular stack that separates attention, mixture-of-experts routing, recurrent-state updates, communication, and memory layout so each bottleneck can be measured and replaced independently.
The model has 397 billion total parameters and activates 17 billion per token. That sparsity reduces arithmetic work, but the server still has to place and move a roughly 400 GB weight footprint while coordinating attention state and routed experts across devices.
What Google changed
The test host contained 4 physical Ironwood chips exposed as 8 logical devices. Qwen's small number of key-value heads made conventional tensor sharding inefficient, so the team replicated attention across data-parallel ranks while distributing experts across the device mesh. Tokens then move between the attention and expert layouts through custom collective operations.
| Systems layer | Optimization | Intended effect |
|---|---|---|
| Attention | Data-parallel attention and coarser paged-cache access | Avoid fractional head sharding and indexing stalls |
| Expert routing | Expert parallelism with packed routing metadata | Reduce redundant communication and padded work |
| Communication | Hierarchical reduce-scatter kernels | Overlap transfers with local reduction |
| Recurrent attention | Fused convolution and state-update kernels | Remove intermediate memory round trips |
| Prefill | Ragged sequence handling and grouped matrix kernels | Reduce padding and expert-load stragglers |
Between April and June 2026, Google measured approximately 3.1x higher decode-heavy performance and 4.7x higher prefill-heavy performance at the 512-concurrency tier. These are improvements over earlier versions of Google's own stack, not comparisons with current GPU systems or independently tuned alternatives.
Reading the throughput numbers correctly
At the 64-concurrency tier, Google reports 3,707 tokens per second per chip for an 8K-input and 1K-output workload, or 82.4% of its discounted 4,500-token roofline. For the 1K-input and 8K-output workload, it reports 677 tokens per second per chip, or 79.6% of an 850-token roofline. The denominator is a vendor model of the hardware ceiling, while the numerator combines input and output tokens. Neither figure is a request-latency service-level objective.
The linked merged pull request provides useful audit evidence. It records launch commands, scheduling details, data-parallel changes, and reviewer discussion about whether batch-token settings created a fair comparison. The authors explain that the scheduling capacity differs across data-parallel ranks and argue that decode results are driven primarily by cache capacity rather than that setting.
The same pull request reports an MMLU-Pro score of 0.8129 for the data-parallel path versus 0.8271 for the tensor-parallel baseline with a limit of 50 examples, describing the difference as within one standard deviation. That is a small regression spot check, not a comprehensive output-equivalence or application-quality study. The blog separately describes numerical verification for selected low-level kernels, which should not be interpreted as proof that every end-to-end response is unchanged.
LDS analysis: the reusable part is the measurement loop
The strongest contribution is the workflow: build a roofline model, profile each subsystem, publish the configuration, connect performance claims to merged code, and isolate changes by workload shape. That makes the report more actionable than an unexplained throughput chart.
Teams evaluating the stack should add four missing comparisons. First, replay representative production prompts and report time to first token, inter-token latency, and tail latency. Second, run a larger quality and tool-use suite after every precision, routing, or scheduler change. Third, report power and price-normalised cost at matched service objectives. Fourth, reproduce the same configuration on another host and disclose variance across runs.
The published gains show substantial progress for this model and hardware pairing. They do not establish that Ironwood is universally faster or cheaper, and they do not remove the need to validate quality after aggressive kernel, precision, and scheduling changes.
Key Points
- 1Google decomposed attention, expert routing, communication, and recurrent updates into reusable hardware-aware modules rather than tuning one monolithic server.
- 2The published gains compare successive Google software stacks, so they cannot establish a universal cross-hardware performance or cost ranking.
- 3A production evaluation still needs tail latency, broader quality checks, matched cost, power measurements, and independent repeat runs.
Scoring Rationale
The report offers unusually concrete open implementation details for frontier-scale inference, while vendor-only measurements and incomplete cost, latency, and quality evidence limit broader conclusions.
Sources
Primary source and supporting public references used for this report.
Practice interview problems based on real data
1,625 SQL & Python problems across 15 industry datasets — the exact type of data you work with.
Try 250 free problems
