SAN FRANCISCO, December 26, 2024 – The quest for faster artificial intelligence is hitting a thermal wall. Researchers at Imec discovered that directly stacking high-bandwidth memory (HBM) on top of a GPU—a technique aimed at boosting performance—initially doubles the chip’s operating temperature, rendering it unusable. This finding, presented at the 2025 IEEE International Electron Device Meeting (IEDM), underscores the complex engineering challenges of squeezing more power out of ever-smaller spaces.
The Heat Is On: Stacking Memory and Processors
Table of Contents
The future of AI computing hinges on overcoming bottlenecks in data transfer, and 3D stacking of memory and processors offers a potential solution—but not without significant hurdles.
- Directly stacking HBM on a GPU initially creates unsustainable heat levels.
- Removing redundant silicon layers and optimizing HBM stack design can significantly reduce temperatures.
- Slowing the GPU clock speed, coupled with increased memory bandwidth, can improve performance while managing heat.
- Further research is needed to determine the optimal configuration—HBM-on-GPU or GPU-on-HBM.
Today’s most advanced AI systems, like those from AMD and Nvidia, rely on a 2.5D arrangement where the GPU sits alongside HBM chips on a shared substrate called an interposer. This minimizes the distance data needs to travel, crucial for AI workloads. But what if you could eliminate that distance altogether by stacking the memory directly on top of the processor? Imec’s team, led by James Myers, investigated this possibility using detailed thermal simulations.
The initial results weren’t promising. A straightforward stacking approach pushed the GPU temperature to a scorching 140 °C, far exceeding its typical 80 °C limit. However, the team didn’t stop there. They identified several optimizations that could potentially bring the temperature back down to acceptable levels.
Imec’s simulations began with a model of a GPU and four HBM dies in a standard 2.5D configuration. This setup consumes 414 watts for the GPU and around 40 watts for the memory, with liquid cooling systems—increasingly common in AI data centers—removing the heat. Yukai Chen, a senior researcher at Imec, explained to engineers at IEDM that while this approach works now, it doesn’t scale well for future designs, particularly as it limits connections between GPUs within a package. “The 3D approach leads to higher bandwidth, lower latency… The most important improvement is the package footprint,” Chen said.
Rethinking the Stack
The first step toward cooling the stacked design involved eliminating a redundant layer of silicon. HBM consists of stacks of up to 12 high-density DRAM dies, connected by tiny solder balls. These stacks are then connected to a “base die” that manages data flow to the GPU. But with the HBM directly on top, this base die—and its data-multiplexing function—becomes unnecessary.
“Bits can flow directly into the processor without regard for how many wires happen to fit along the side of the chip,” Myers explained, adding that moving the memory control circuits from the base die into the GPU should be feasible without significantly altering the processor’s design.
Removing this intermediary layer reduced the temperature by a modest 4 °C. More significantly, it opened the door to another optimization: slowing down the GPU. Large language models are “memory-bound,” meaning their performance is limited by memory bandwidth. The team estimated that 3D stacking would quadruple bandwidth, allowing them to reduce the GPU clock speed by 50 percent while still achieving a performance gain and lowering temperatures by over 20 °C. In practice, a smaller reduction in clock speed—increasing the frequency to 70 percent—only raised the GPU temperature by 1.7 °C.
Optimizing HBM for Heat Dissipation
Further temperature reductions came from optimizing the HBM stack itself. This included merging the four stacks into two wider stacks to eliminate heat-trapping regions, thinning the top die of the stack, and filling the surrounding space with silicon to improve heat conduction. These changes brought the temperature down to around 88 °C.
Finally, adding cooling to the underside of the stack, in addition to the standard top-side cooling, dropped the temperature another 17 °C, bringing it close to the original 70 °C.
While the research suggests HBM-on-GPU is possible, Myers cautioned that it isn’t necessarily the best approach. “We are simulating other system configurations to help build confidence that this is or isn’t the best choice,” he said. “GPU-on-HBM is of interest to some in industry,” because it places the GPU closer to the cooling system, but it would likely be a more complex design.
