Why Jensen Huang Is Wrong About Memory Gluts and Right About the Wrong Thing

Why Jensen Huang Is Wrong About Memory Gluts and Right About the Wrong Thing

Predictable? No. Profitable for the seller? Always.

When Jensen Huang dismisses fears of a 2029 memory glut with a breezy "You can always use more," half of Wall Street nods like a bobblehead and the other half panics about inventory cycles. Both sides are staring at the wrong meter. Asking whether memory will be in glut or shortage four years out assumes memory behaves like wheat or crude oil—a simple ratio of barrels pumped to barrels burned. It does not. Memory is a moving bottleneck disguised as a commodity.

The lazy consensus in coverage of Huang's remarks splits into two tired camps. Camp A treats the CEO of Nvidia as an oracle whose word settles supply-and-demand math years ahead of fab depreciation schedules. Camp B treats memory makers—SK Hynix, Micron, Samsung—as perpetual boom-and-bust gamblers fated to overbuild cleanrooms every few years until prices collapse.

Both camps miss the real friction. In AI infrastructure, raw gigabytes rarely sit idle in a vacuum waiting for a "glut" to clear. What gets stranded is usable memory: capacity that matches the bandwidth wall, thermal envelope, packaging constraints, and software scheduling of actual production workloads. You can have plenty of silicon on paper and still starve a cluster because the data cannot cross the interconnect fast enough to feed tensor cores.

Defining What "Memory" Actually Means Before You Bet a Cent

Half the arguments in semiconductor comment sections fall apart in thirty seconds because people conflate DRAM, High Bandwidth Memory (HBM), and NAND flash as if they come out of the same spigot. They do not.

  • Standard DDR/LPDDR DRAM handles host system memory, lower-tier inference servers, and general compute where cost per gigabyte and capacity density trump peak pin bandwidth.
  • High Bandwidth Memory (HBM) stacks DRAM dies vertically and connects them via through-silicon vias (TSVs) and advanced packaging (such as CoWoS or equivalent interposer flows) right next to the compute die. It buys massive parallel bandwidth at the cost of higher unit price, stricter thermal limits, and tighter wafer-to-package yields.
  • NAND flash provides persistent storage—checkpoints, training corpora, KV-cache offloading tiers—that is orders of magnitude slower than DRAM, even with modern PCIe Gen5/Gen6 NVMe controllers.

When someone says "You can always use more," they are usually talking about context windows, KV-cache footprints, mixture-of-experts (MoE) parameter residency, and multi-agent state. Longer prompts and persistent reasoning traces expand the working set held in fast memory during inference. From a pure software-appetite standpoint, Huang has a point: models gladly eat whatever fast memory you hand them.

Where the consensus gets sloppy is assuming infinite software appetite automatically prevents a localized oversupply—or guarantees perpetual pricing power for every memory vendor. Software appetite is conditional on unit economics. If adding another 80 GB or 192 GB of HBM per accelerator pushes token cost above what paying customers will tolerate for a marginal quality gain, buyers throttle back or change model architecture.

A model will gladly consume every byte of HBM you give it. A CFO will only pay for the bytes that move gross margin in the right direction.

The Myth of the Four-Year Crystal Ball

Four years in semiconductor manufacturing is an eternity wrapped in multi-billion-dollar commitments. Lithography tools—extreme ultraviolet (EUV) and high-NA EUV systems from ASML—carry long lead times. Cleanroom shells take years to permit, build, and qualify. Wafer start ramps depend on yield learning curves that stubbornly refuse to read executive press releases.

I have watched infrastructure teams blow millions locking in multi-year supply commitments on a single memory tier, only to discover eighteen months later that quantization advances, speculative decoding, or leaner MoE routing shifted where the bottleneck lived. Over-allocating budget to one scarce component while under-investing in networking fabric or power delivery leaves you with expensive parts sitting in racks running below peak utilization.

The flip side happens just as often. Memory suppliers historically expand capacity when margins look irresistible—because CFOs reward growth when cash flows are fat—and cleanrooms commissioned during peak euphoria have a nasty habit of coming online right as buyers optimize their stacks or digest prior capex waves. Calling 2029 "glut-free" with certainty ignores how nonlinear software efficiency gains and architectural shifts can bend demand curves faster than fab construction schedules can bend supply.

Consider a thought experiment. Imagine a scenario where by 2028, standard inference workloads shift heavily toward aggressive FP4/int4 quantization, structured KV-cache compression, and disaggregated prefill/decode pools that push long-context state onto cheaper tiered memory pools. Total gigabytes shipped across the industry might still rise year over year, yet the specific high-margin HBM stack configuration vendors bet their peak capex on could face softer pricing than bulls expect. Growth in volume is not immunity from margin compression.

What People Actually Ask—And Why the Questions Are Crooked

Search behavior around this topic clusters around a few predictable questions. Most of them smuggle in false assumptions.

"Will there be an AI memory shortage or glut by 2029?"

Binary framing makes for punchy headlines and terrible capital allocation. The industry routinely experiences simultaneous shortage and glut across different grades and generations. Last-generation parts or non-optimal configurations can pile up while leading-edge stacked memory or advanced packaging capacity stays tight. Asking "glut or shortage?" in 2026 about 2029 is like asking in 2022 whether 2026 will have "too much or too little compute"—it flattens a multidimensional engineering and pricing puzzle into a coin flip.

"Why does Jensen Huang say you can always use more memory?"

Because from an accelerator vendor's vantage point, memory capacity and bandwidth are primary levers that let larger models and longer contexts run on fewer chips or with higher throughput per rack. If customers keep scaling reasoning length, batch sizes, and concurrent streams, memory demand scales right alongside compute demand. That is true as a demand ceiling observation. It is not proof that supply, pricing, and architecture will stay locked in his favor forever.

"Should companies buy memory early or wait for prices to drop?"

Neither blindly. Buying early locks in availability during tight allocation windows, at the risk of paying peak prices or getting stuck with yesterday's form factor when a new architecture lands. Waiting for price drops works great during true oversupply—until a surprise demand spike leaves you begging brokers for allocation at triple the spot rate.

The Heavy Hitters and the Physics That Govern Them

You do not need rumor mills to see where real constraints live; look at the people and physics defining the stack. Work coming out of research groups and hardware leaders—from David Patterson and John Hennessy's classic framing of the memory wall (the widening gap between processor speed and memory access latency/bandwidth) to modern systems architects designing disaggregated inference engines and high-bandwidth interconnects—points to one unforgiving truth: compute is useless if data starves on the way to the arithmetic units.

At the same time, memory titans like SK Hynix, Micron, and Samsung operate under brutal physical trade-offs. Stacking more layers in HBM increases heat density and warpage risk while demanding tighter thermal dissipation and power delivery. Every step up the stacking ladder tests yields and packaging throughput.

Even Huang himself—whose company sits at the center of data-center rack design with NVLink and high-speed fabric ecosystems—knows that system performance is a coupled equation of compute, memory bandwidth, interconnect bisection bandwidth, and power per rack. Memory is huge, critical, and expensive. It is not the sole god of the data center.

Where My Contrarian Take Can Burn You

Honesty earns trust faster than bravado. Betting against a "perpetual shortage" narrative—or assuming software efficiency will save buyers from paying up for memory—carries real risks.

First, software efficiency often triggers Jevons-style rebound demand. When models get cheaper per token to run, teams run longer chains of thought, wider parallel rollouts, and heavier agent loops. Efficiency lowers unit cost; it does not automatically shrink total memory appetite.

Second, under-committing during a real squeeze hurts worse than overpaying a bit. If your product depends on low-latency, high-throughput inference and you cheap out on memory bandwidth or allocation, degraded user experience and missed revenue can dwarf a few percentage points of hardware savings.

Third, packaging and yield bottlenecks can create artificial scarcity even when raw wafer capacity looks decent on a spreadsheet. A single chokepoint in advanced packaging or test time can keep prices elevated longer than textbook economics predicts.

Unconventional Moves That Actually Work

If you run infrastructure, build AI products, or allocate capital in this space, stop treating memory as a passive bill-of-materials line item. Do this instead:

  1. Engineer for memory tiering from day one. Design your serving stack so hot active states live in fast memory while cold or older context tiers degrade gracefully into slower, cheaper pools. Never hardcode your product's survival around unlimited ultra-expensive memory.
  2. Benchmark cost per useful token at target latency, not raw GB per GPU. A slightly smaller memory footprint paired with smarter KV-cache management and batching often beats brute-force capacity on real unit economics.
  3. Hedge contract terms, not just volume. Mix committed allocation for baseline production traffic with flexible capacity for spiky or experimental workloads. Avoid locking 100% of your forecast into rigid multi-year assumptions when model architectures are still shifting fast.
  4. Watch packaging and interconnect roadmaps as closely as DRAM. Memory speed in a system is capped by how fast data moves across dies, interposers, and nodes. Track the bottlenecks around the memory chip, not just inside it.

Ignore the 2029 fortune-telling contest. Build systems that stay profitable whether memory gets tighter, looser, or weirder.

CC

Caleb Chen

Caleb Chen is a seasoned journalist with over a decade of experience covering breaking news and in-depth features. Known for sharp analysis and compelling storytelling.