Dual DGX Spark Testbed Trains NanoChat Across 200 Gb/s Link
A new proof-of-concept report describes distributed NanoChat pretraining across two NVIDIA DGX Spark systems linked by dedicated 200 Gb/s fiber and administered through Tailscale. The authors report processing about 653 million tokens over four days at roughly 1,890 tokens per second, while explicitly presenting the setup as a reproducibility testbed rather than a matched scaling-efficiency benchmark.
An August 7 preprint by Vasanth Iyer documents a two-node NVIDIA DGX Spark testbed for distributed language-model training and cybersecurity fine-tuning. The setup used one GB10 Grace Blackwell system-on-chip and 128 GB of unified memory in each node, with two separate networking planes: Tailscale for remote administration and a direct 200 Gb/s QSFP56 fiber link for NCCL training traffic.
How the testbed was configured
The reported NanoChat run used PyTorch torchrun, DistributedDataParallel and NCCL with one process per node. A depth-20 model, local batch size of 32 per node and 2,048-token context produced a global batch of 131,072 tokens per step.
The authors report a step time of about 69.4 seconds and throughput near 1,890 tokens per second. Across four days, the run processed about 653 million tokens. The companion GitHub repository publishes a sanitized deployment runbook, network topology, diagnostic scripts and troubleshooting notes, including the separation between the management network and the high-throughput training link.
The report also records a practical failure mode: a step-zero evaluation bug caused NCCL timeouts. Documenting that issue, along with interface binding and checkpointing choices, makes the work more useful as an engineering reference than a headline performance comparison.
What the results do and do not show
For a second experiment, the team built a cyber-threat-intelligence dataset from 77 CISA advisories, producing 338 training and 37 validation conversations. On a 17-question held-out evaluation scored by an Ollama-hosted LLM judge, the reported overall score moved from 2.06 to 2.29 on a 0-10 scale. Cybersecurity-specific categories improved, while general-knowledge categories regressed.
Those results are preliminary. The evaluation is small and model-judged, and the paper says its single-node throughput comparison was estimated rather than measured under matched conditions. The evidence therefore supports feasibility and reproducibility, not a claim that two DGX Spark nodes deliver a particular scaling efficiency.
For practitioners, the most transferable contribution is the operational pattern
keep remote access separate from the NCCL data path, publish the failure modes and measure both task-specific gains and regressions before treating a fine-tuned checkpoint as an improvement.
Key Points
- 1The testbed separates Tailscale-based remote administration from a dedicated 200 Gb/s fiber link used for distributed training traffic.
- 2The authors report about 1,890 tokens per second and roughly 653 million tokens processed over four days, but no matched single-node scaling measurement.
- 3Cyber-threat-intelligence fine-tuning improved task-specific categories in a small model-judged evaluation while general-knowledge categories regressed.
- 4A public companion repository provides a sanitized runbook, scripts and troubleshooting notes for reproducing the two-node setup.
Scoring Rationale
The report provides reproducible engineering evidence for a compact two-node AI training testbed and unusually explicit limitations, but its throughput comparison is not matched and its fine-tuning evaluation is small.
Sources
Primary source and supporting public references used for this report.
Practice interview problems based on real data
1,625 SQL & Python problems across 15 industry datasets — the exact type of data you work with.
Try 250 free problems