Benchmarking Reveals H100 SXM is Most Cost-Effective for Training Nanochat Despite Higher Hourly Rates

Summary

Developer Nikhil Kasukurthi conducted comprehensive benchmarks comparing three NVIDIA H100 GPU variants—PCIe, SXM, and NVL—for training Nanochat, an open-source language model inspired by Andrej Karpathy's work. The study evaluated configurations across cloud providers Runpod and Vast.ai, measuring step times, NCCL communication overhead, and total training costs. Despite having the highest hourly rate, the H100 SXM configuration proved most economical, completing the training run in approximately 3 hours for $37—making it 2x cheaper than PCIe and 3x cheaper than NVL options.

The benchmark focused on understanding how network interconnect performance impacts distributed training, particularly for the Zero-2 optimizer pattern used in Nanochat. The model employs a combined Muon + AdamW optimizer that requires frequent gradient synchronization across GPUs through reduce_scatter and all_gather operations. The superior NVLink 4.0 interconnect in SXM configurations significantly reduced communication overhead compared to the PCIe-based alternatives, despite communication representing a relatively small portion of overall training time for this model size.

The research also uncovered several practical pitfalls in GPU cluster utilization, including CPU starvation from improper NUMA socket pinning, spot instance preemption during profiling runs, broken nodes throwing CUDA errors, and NCCL connection issues on NVL configurations. These findings highlight that raw GPU compute power represents only part of the equation—interconnect bandwidth, system configuration, and operational reliability significantly impact both performance and cost-effectiveness for distributed training workloads.

• The cost to train GPT-2-level performance has dropped dramatically from $43,000 in 2019 to under $100 in 2026 using modern hardware and techniques
• Network interconnect choice becomes increasingly important as models scale, even when communication represents a small fraction of total training time

Editorial Opinion

This benchmark provides valuable real-world data that challenges the simplistic "cheapest per-hour" mentality in cloud GPU selection. The finding that faster interconnects can more than compensate for higher hourly rates has broad implications for the AI training market, particularly as developers increasingly train models on spot instances. The detailed documentation of operational pitfalls—from NUMA configuration to spot preemption—represents the kind of practitioner knowledge that's often missing from vendor marketing materials but crucial for actual cost optimization.