Major CSPs Aggressively Constructing AI Servers and Boosting Demand for AI Chips and HBM
June 22, 2023 | TrendForceEstimated reading time: 2 minutes
TrendForce reports that explosive growth in generative AI applications like chatbots has spurred significant expansion in AI server development in 2023. Major CSPs including Microsoft, Google, AWS, as well as Chinese enterprises like Baidu and ByteDance, have invested heavily in high-end AI servers to continuously train and optimize their AI models. This reliance on high-end AI servers necessitates the use of high-end AI chips, which in turn will not only drive up demand for HBM during 2023~2024, but is also expected to boost growth in advanced packaging capacity by 30~40% in 2024.
TrendForce highlights that to augment the computational efficiency of AI servers and enhance memory transmission bandwidth, leading AI chip makers such as Nvidia, AMD, and Intel have opted to incorporate HBM. Presently, Nvidia’s A100 and H100 chips each boast up to 80 GB of HBM2e and HBM3. In its latest integrated CPU and GPU, the Grace Hopper Superchip, Nvidia expanded a single chip’s HBM capacity by 20%, hitting a mark of 96 GB. AMD’s MI300 also uses HBM3, with the MI300A capacity remaining at 128 GB like its predecessor, while the more advanced MI300X has ramped up to 192 GB, marking a 50% increase. Google is expected to broaden its partnership with Broadcom in late 2023 to produce the AISC AI accelerator chip TPU, which will also incorporate HBM memory, in order to extend AI infrastructure.
TrendForce predicts that AI accelerator chips that primarily utilize HBM (including Nvidia’s H100 and A100, AMD’s MI200 and MI300, and Google’s TPU) will have a total HBM capacity of 290 million GB in 2023—a nearly 60% growth rate. This momentum is projected to sustain at a rate of 30% or more into 2024.
Additionally, there is escalating demand for advanced packaging technology in AI and HPC domains, with TSMC’s Chip on Wafer on Substrate (CoWoS) being the predominant choice for AI server chips. CoWoS packaging technology involves two stages: CoW and oS. The CoW phase merges various Logic ICs (CPUs, GPUs, and ASICs) and HBM memory; the oS stage uses solder bumps to connect the CoW assembly and attach it onto a substrate. This is subsequently integrated into the PCBA, forming the principal computational unit of the server’s motherboard, thereby creating a comprehensive AI server system with other components such as networks, storage, power supply units (PSUs), and other I/O.
TrendForce notes that due to the robust demand for high-end AI chips and HBM, TSMC’s monthly CoWoS capacity is projected to hit 12K by the end of 2023. The demand for CoWoS has surged by almost 50% since the onset of 2023, driven by the needs of Nvidia’s A100 and H100 and associated AI servers. CoWoS capacity is likely to be strained in 2H23 as demand for high-end AI chips from AMD, Google, and other companies intensifies. This strong demand is anticipated to continue into 2024, with a projected growth of 30~40% in advanced packaging capacity, given the readiness of related equipment.
TrendForce underscores that in light of the urgent demand for AI, whether it’s producing HBM or CoWoS, monitoring of support technologies is critical. This includes considering when technologies like Through Silicon Via (TSV) packaging, Interposer circuit boards, and related equipment (i.e., wet process equipment) are implemented, along with factoring in lead times. Should strong demand for AI persist, it’s plausible that Nvidia may explore other advanced packaging alternatives for the CoWoS process—such as those provided by Amkor or Samsung—to address potential supply shortages.
Suggested Items
Indium Corporation to Showcase Precision Gold Solder Solutions at MEDevice Silicon Valley 2024
11/18/2024 | Indium CorporationIndium Corporation® will feature its high-reliability AuLTRA® MediPro gold solder solutions at MEDevice Silicon Valley, taking place on November 20-21 in Silicon Valley, California. AuLTRA® MediPro is a family of high-performance, precision gold solder solutions for critical medical applications.
AIM to Highlight NC259FPA Ultrafine No Clean Solder Paste at SMTA Silicon Valley Expo & Tech Forum
11/14/2024 | AIMAIM Solder, a leading global manufacturer of solder assembly materials for the electronics industry, is pleased to announce its participation in the upcoming SMTA Silicon Valley Expo & Tech Forum taking place on December 5 at the Fremont Marriott Silicon Valley in Fremont, California.
Data-driven Precision in PCBA Manufacturing
11/13/2024 | Julie Cliche-Dubois, CogiscanThe intricacies involved in electronics manufacturing require more than just expensive equipment and skilled technicians; they necessitate an accurate understanding of the entire production flow, informed and driven by access and visibility to reliable data.
Rehm Thermal Systems Mexico Wins the Mexico Technology Award 2024 in the Category Convection Soldering
11/13/2024 | Rehm Thermal SystemsRehm Thermal Systems Mexico has won the Mexico Technology Award in the category convection soldering with the patented mechatronic curtain for convection soldering systems.
SMTA: Capital Chapter & Connecticut Chapter Joint Technical Webinar
11/12/2024 | SMTAThe SMTA Capital Chapter is co-hosting a free webinar for SMTA members with the SMTA Connecticut Chapter on Tuesday, November 19 at 11:00 a.m. EST.