Kimi K3 does not fit.
That is the whole problem, stated the way an infrastructure engineer would state it. Moonshot's model carries 2.8 trillion total parameters with 104 billion active and a 1-million-token context window, which works out to more than 1.5 terabytes of GPU memory for the weights alone, before a single byte is allocated to the key-value cache that inference actually needs. An eight-GPU Nvidia B200 node has 192 GB per card. Do the arithmetic and the model spills off the box.
So when the San Francisco startup Wafer went to serve one of the largest open-weight models yet released, the first question was not which GPU is fastest. It was which GPU has enough memory to avoid talking to a second machine.
The answer surprised people, and it was not made by Nvidia.
The Numbers Split Cleanly Down the Middle
Wafer published its results on July 31, 2026, in a technical post by technical staff member Ian Ye titled "Is memory the moat?" The benchmark used a 1,024-token input and a 400-token output. Three configurations, one model.
| Configuration | Decode per stream | Peak aggregate | Per GPU | Per dollar |
|---|---|---|---|---|
| 8x AMD MI355X (TP8) | 118 tok/s | 952 tok/s | 119 tok/s | 48 tok/s/$ |
| Nvidia B300 (TP8+DCP8) | 172 tok/s | 1,568 tok/s | 196 tok/s | 33 tok/s/$ |
| 2x8 Nvidia B200 (TP16) | 90 tok/s | 498 tok/s | 31 tok/s | 7 tok/s/$ |
Read the raw throughput column and Nvidia wins convincingly. The B300 delivered 65% more aggregate throughput than the AMD node and a faster single stream. Read the last column and the ranking inverts, because AMD's MI355X is about 2.4 times cheaper per GPU than a B300.
The hourly rental rates Wafer assumed are the load-bearing part of that conclusion, and worth remembering before anyone quotes the 48 figure in a procurement deck.
| GPU | Assumed hourly rate |
|---|---|
| AMD MI355X | $2.50 |
| Nvidia B300 | $6.00 |
| Nvidia B200 | $4.25 |
The B200 result deserves a footnote rather than a headline. Kimi K3 plus a 1-million-token cache will not fit on one eight-GPU B200 node, so Wafer had to span two nodes across sixteen GPUs. That configuration pays a cross-node all-reduce on the decode critical path, and the 498 tokens per second is the total across all sixteen cards, roughly 249 per node.
That penalty is not a footnote to the argument. It is the argument. The MI355X carries 288 GB of HBM3E per GPU, giving an eight-card node about 2.3 TB, enough to hold Kimi K3 and its cache on one machine. A nominally faster accelerator that forces you across a network boundary can lose to a slower one that does not.
Neither Fix Required Writing a Custom Kernel
AMD's reputation problem in inference has been the software layer rather than the silicon, with kernels landing late and day-zero framework support thinner than on CUDA. Wafer's own summary is that the performance-per-dollar result came "relatively out of the box," and that unlike its earlier GLM5.2 work, this one "certainly did not require custom kernels." Two bugs still stood in the way.
The first fix involved speculative decoding, the technique where a small draft model proposes several tokens at once and the large model verifies them in a single pass. Kimi K3 ships no draft tensors of its own, so Wafer used RadixArk's external Kimi-K3-DSpark draft. It ran on CUDA immediately. On ROCm, AMD's software stack, the serving framework crashed because a top-k renormalization routine that the CUDA build imports from a compiled kernel library was simply undefined in the ROCm build.
The repair was a few lines of PyTorch: sort the probability vector, keep the highest entries, zero the rest, rescale to sum to one. As Wafer put it, the reflex on ROCm is to assume you need a custom kernel, but this was a missing definition, not a missing kernel. That single change bought roughly 2.2x on single-stream performance, about 1.7x per stream at moderate load, and 18% on peak aggregate throughput. It also pushed peak throughput out to much higher concurrency.
The second fix targeted prefill, the stage where the model reads your input before emitting anything. A 172,000-token cold prefill took about 51 seconds on the MI355X against roughly 23 seconds on the B300, which on a million-token model is the difference between a usable fleet and a fleet spinning uselessly.
Almost all of that gap traced to one kernel refusing to load. Kimi K3 at eight-way tensor parallelism produces twelve attention heads per rank, and AMD's fast attention path is built for four, eight, or multiples of sixteen. Wafer padded twelve up to sixteen, ran the fast kernel, and discarded the extra outputs. Steady-state prefill went from roughly 4,000 to 7,000 tokens per second up to about 13,000.
Wafer is explicit that this second fix does not move the headline numbers. It is a time-to-first-token lever, and decode is unchanged. The distinction matters, because time-to-first-token is the latency a user actually feels, and it is the metric Wafer argues gets overlooked in favor of decode throughput.
Nobody wrote a new kernel. Somebody read a stack trace carefully.
The Company Publishing the Benchmark Sells the Conclusion
Wafer is a Y Combinator Summer 2025 company with seven people, founded by Emilio Andere and Steven Arellano, who met in their first year at the University of Chicago and became roommates. Andere studied mathematics and worked on weather models at Argonne National Laboratory. Arellano studied computer science and worked on high-performance computing and AI infrastructure at Google, Two Sigma, and Sei Labs. The company raised a $4 million seed round on April 14, 2026, led by Fifty Years with participation from Liquid 2 and Y Combinator.
What Wafer sells is software agents that profile a workload and search across models, inference engines, kernels, and hardware configurations so a customer never has to commit to one chip vendor in advance.
Which means this benchmark is also a sales argument, and every caveat that follows is one a buyer should apply.
The results are self-reported and produced on Wafer's own test setup. RuntimeWire, which covered the post on August 2, said so plainly. The rental prices are assumptions, and GPU spot pricing moves. The B200 comparison is structurally unfavorable in a way that flatters the headline multiple. Arellano's claim on X of "3.8x higher throughput and 71% lower cost" is measured against that two-node B200 setup, not the B300.
And the B300 genuinely won the tests that many workloads care about most: more raw throughput, a faster single stream, and a cold prefill that took less than half the time before Wafer patched the attention kernel.
The honest summary is narrower than the headline. Under one company's configurations and one company's assumed prices, a single AMD node served one very large open model more cheaply per unit of throughput.
What Changes for Practitioners
Two things, and neither requires believing Wafer's price assumptions.
- Memory capacity is now a first-order deployment constraint. Open-weight models have grown past what most nodes can hold. When weights plus cache exceed one node, the architecture decision is made for you before performance enters the conversation.
- The CUDA software moat is measurably thinner than a year ago. AMD shipped day-zero support for Kimi K3. The two blockers Wafer hit were a missing function definition and a head-count shape mismatch, neither of which required new kernel code. That is a very different situation from writing assembly kernels from scratch.
Anthropic's decision to run Claude on AMD silicon and use Claude to improve ROCm points at the same trend from the other direction, and Zyphra's 8B reasoning model trained entirely on AMD chips showed it on the training side months ago.
The Bottom Line
For three years the benchmark that sold GPUs was raw throughput. When a single open model needs 1.5 TB just to load, the binding question becomes whether it fits on one box and what that box costs per hour. Those are two different questions, and Nvidia does not automatically win the second one.
Nvidia still makes the faster chip. That was never in dispute, and Wafer's own numbers confirm it twice over.
What is in dispute is whether faster is the metric. Inference buyers do not purchase tokens per second. They purchase tokens, and they pay in dollars, and the conversion rate between those two things now depends on a memory spec sheet and a handful of framework patches that a seven-person startup could write in PyTorch.
Ye ended the post with a question rather than a claim, which is the appropriate posture for a single self-run benchmark: "SOTA on AMD is imminent. Is the CUDA moat dead?"
Not dead. But for the first time, somebody published a number where it did not matter.
Sources
- Is memory the moat? Running Kimi K3 at ~952 tok/s/node (Jul 31, 2026)
- Wafer says AMD's MI355X beats Nvidia B300 on Kimi K3 cost efficiency (Aug 2, 2026)
- Kimi K3: Open Frontier Intelligence (technical report) (2026)
- AMD Instinct MI350 Series accelerators (2026)
- RadixArk/Kimi-K3-DSpark on Hugging Face (2026)
- Wafer: Announcing our Seed Round (Apr 14, 2026)
- Wafer on Y Combinator (2026)
- Performance per dollar is getting faster and cheaper (GLM5.2 on MI355X) (Jul 3, 2026)
- AI News Briefs Bulletin Board for August 2026 (Aug 4, 2026)