High-Bandwidth Flash Could Scale GPU Memory to Multiple Terabytes
Key Takeaways
- ▸HBF achieves 512GB per module—over 14x current HBM4 capacity—enabling terabyte-scale GPU memory
- ▸Stacked NAND flash design delivers 1.6 TB/s bandwidth (approaching HBM3e) with potential path to 3.2 TB/s
- ▸Write endurance and latency limitations prevent full HBM replacement but suit read-heavy inference workloads
Summary
SanDisk and SK Hynix are developing high-bandwidth flash (HBF), an emerging memory technology that could increase GPU memory capacity from hundreds of gigabytes to terabytes. Built by stacking NAND flash layers—similar to how HBM stacks DRAM—the first-generation HBF modules promise read bandwidths up to 1.6 TB/s (comparable to HBM3e) with potential to reach 3.2 TB/s in future versions. Most significantly, each 16-layer module can achieve 512GB capacity, over 14 times the capacity of current HBM4 memory used in NVIDIA and AMD's latest accelerators, and without expected power or price premiums.
However, the technology comes with significant trade-offs. Because HBF uses NAND flash rather than DRAM, it suffers from finite write endurance and microsecond-level access latencies versus nanoseconds for DRAM—making it unsuitable as a direct HBM replacement. Instead, developers propose using HBF to supplement HBM during AI inference, leveraging the architectural difference between prefill (write-intensive) and decode (read-intensive) phases. This tiered approach could substantially reduce inference costs by storing large model weights in high-capacity HBF while keeping performance-sensitive operations on fast HBM.
- Proposed hybrid architecture: HBM for prefill (write-intensive), HBF for decode (read-intensive token generation)
- Superior bits-per-dollar economics with no expected power penalty could reshape inference cost structures
Editorial Opinion
High-bandwidth flash elegantly solves one of AI infrastructure's most pressing constraints—memory capacity—by leveraging the architectural asymmetries within inference workloads. While write endurance prevents it from displacing HBM entirely, the 512GB modules at price parity open a compelling path for hyperscalers to serve larger models more cost-effectively. The tiering strategy is particularly clever: it sidesteps NAND's latency limitations where they matter most (prefill compute) while exploiting its capacity advantage where bandwidth, not latency, dominates (decode). If performance and reliability projections hold, this could reshape accelerator ROI economics within the next generation of hardware.
