At this year’s IEEE International Electron Device Meeting, [IEDM] (December in San Francisco) SK hynix gave an interesting look at “Advanced Packaging Technology in Memory Applications for Future Generative AI Era”.
[Generative AI creates new content through the use of machine learning models such as generative adversarial networks. Such frameworks use algorithms and neural nets to generate content from scratch.
With AI demanding memory with higher bandwidth, higher capacity, and better power efficiency, SK hynix is looking at the challenges that HBM stacked memory will be facing in the future.
As processor performance increases, DRAM performance becomes a bottleneck in improving system performance, making it important to place memory closer to logic and connect it through more I/O paths.
Figure 1 shows the fundamental structure of SK hynix HBM, consisting of a base logic die at the bottom and stacked DRAM die vertically connected by TSV and microbumps.
SK hynix has recently announced a 24GB HBM with a 12-die high stack.
Table 1 shows the bandwidth and capacity trends for each HBM generation. The pitch of the TSV and microbumps must be continuously reduced to meet such requirements.
As the number of stacked chips increases, more power TSVs are required and power delivery issues must be resolved. SK hynix has found that power drop improves dramatically when power is supplied evenly by adding power TSVs to the bank area, and when the power TSVs are increased in size.
Thermal management is also very important. To this end, the thermal conductivity of the gap-fill material between the layers should be as high as possible and a clear path should be created to dissipate heat.
HBM4 requires a bandwidth corresponding to an IO speed of over 12Gbps with 16 or more high DRAM stacks. For such solutions, Hynix expects that C2W hybrid bonding is a better solution since it will stack more die in a given Z-height by removing gap fill material and better controlling thermal dissipation.
Another advantage of hybrid bonding is to reduce the interconnect pitch since micro bump joining cannot go below 20um.
AiM and PiM Memory Packaging
- Processing-In-Memory (PIM): A next-generation technology that adds computational capabilities to semiconductor memories to solve the problem of data movement congestion in AI and big data processing.
- Accelerator in Memory (AiM): SK hynix’s PIM semiconductor product name, which includes GDDR6-AiM.
Existing GPU systems take up to 10s of seconds to generate AI answers of five to ten sentences. Sk hynix notes that PiM and AiM can be solutions to overcome long processing times and improve overall performance
GDDR6-AiM is a type of memory chip that adds computational functions to GDDR6 memory chips, which process data at 16Gbps. It is a next-generation technology that adds computational capabilities to semiconductor memories to solve the problem of data movement congestion in AI and big data processing. SK Hynix recently unveiled and demonstrated a prototype of AiMX 1, a generative AI accelerator card based on GDDR6-AiM, at the AI Hardware & Edge AI Summit 2023 held in Santa Clara, California.
The first application of PIM is called GDDR6-AiM. According to SK hynix’s testing, GDDR6-AiM can speed AI processing up to 16x and reduce power consumption by 80%.
This first PIM product is going to be targeting machine learning, high-performance computing, and big data computation and storage applications. One of the first applications earmarked to adopt GDDR6-AiM is AI processing acceleration, using chips developed in collaboration with SAPEON Inc., which recently spun off from SK Telecom (South Korea’s national telco).
The potential benefits include the fact that GDDR6-AiM operates at 1.25V, lower than existing GDDR5 modules (1.35V). Thanks to this improvement, and the reduction of data movement between memory and the CPU or GPU, SK Hynix claims a power consumption savings of up to 80% can be achieved.
For all the latest in Advanced packaging stay linked to IFTLE…………………..
