AI Racks Cross 100kW as Memory Scarcity Reshapes GPUs and Data-Center Power and Cooling
Rack-scale AI hardware is repricing the rest of the data centre: Meta's infrastructure chief said 10kW racks are giving way to 100kW-plus racks needing a second high-speed fibre network, liquid cooling and swing-tolerant power, while Nvidia redirected RTX 5090-class silicon into an 84GB workstation part as memory, not logic, sets the ceiling.
Rack density is now the binding design constraint. Speaking at the AI Infra Summit in Santa Clara on 16 September, Meta's vice president of infrastructure Manish Kadakia said racks that once drew about 10 kilowatts are being replaced by AI racks pulling 100 kilowatts or more, and that all four core data-centre systems, servers, networks, power and cooling, are changing at once. A 100kW rack also produces 100kW of heat, while the power distribution and cooling in most buildings were designed for about a tenth of that per rack. Training adds a second, back-end network linking processors to one another, which Kadakia said requires a humongous amount of fibre and cannot tolerate delay, because a processor waiting on another's results sits idle and stretches the training run.
Power quality is emerging as a distinct problem. Thousands of processors can ramp up and down together, sending demand sharply higher and lower; David Skeans, senior vice president of sales at electrical equipment maker Legrand, said those swings can resemble the electrical faults backup systems are designed to detect, so equipment built to ride through outages must now also absorb abrupt changes caused by the servers themselves. On the cooling side, liquid now circulates through metal plates attached to processors, placing it close to expensive electronics, and operators have placed sandbags around racks alongside leak sensors. Monitoring vendors warn that chemistry changes drive corrosion, worn parts shed metal particles and microbes can grow in the loop, clogging passages and reducing heat-transfer capacity.
Memory scarcity is reshaping the hardware itself. Nvidia quietly added the RTX Pro 5500 Blackwell Workstation Edition on 14 September, built on the same GB202 die and 21,760 CUDA cores as the GeForce RTX 5090 but carrying 84GB of ECC GDDR7, about 2.6 times the 5090's 32GB, on a 448-bit bus with up to 1,398GB/s of bandwidth and a 600W board limit. The card supports Multi-Instance GPU partitioning into one 84GB instance or two 42GB instances, a configuration aimed squarely at AI workloads rather than gaming. Pricing and release date were not announced. The move signals that in a sold-out memory market, scarce DRAM capacity is being steered toward higher-margin professional and AI parts first.
Infrastructure vendors are redesigning around the same constraint. Huawei unveiled what it calls the world's first 3D data centre on 15 September in Wuhu, Anhui, stacking cooling, IT, power and backup power vertically across four layers rather than spreading them horizontally, with more than 90% of mechanical and electrical systems prefabricated. The company said the approach cut delivery time for those systems from about six months to three and is aimed at clusters of more than 100,000 AI accelerator cards. Across the industry, air-cooled and liquid-cooled systems will have to coexist during the transition, and each hardware upgrade now forces a decision on how much surrounding infrastructure must change and how to do the work without interrupting running compute.