AI Racks and 1.6T Optics Tighten as Memory Costs Reshape Server Economics

Rack-scale architecture is becoming the organising principle of AI hardware. d-Matrix said on 10 September it will license Nvidia's NVLink Fusion interconnect and build on MGX rack reference designs for its Raptor inference accelerators, targeting systems of up to 144 accelerators on a single all-to-all NVLink fabric by end-2027. The company is pairing its chips with Nvidia Vera CPUs, NVSwitch appliances, BlueField and ConnectX NICs and SpectrumX Ethernet. MediaTek joined Marvell, Qualcomm, Arm, Fujitsu and AWS in adopting NVLink Fusion, with Nvidia investing $3.5 billion in MediaTek and $2 billion in Marvell as inducements.

Memory, not logic, is setting the cost curve. Nvidia has told major customers that systems built on Grace Blackwell and Vera Rubin will cost more than 15% from early 2027, according to press reports the company has not publicly confirmed. A component-level teardown of a Vera Rubin NVL72 rack puts 72 Vera GPUs near $3.96 million, 36 Vera CPUs near $180,000, memory and storage near $2.0 million and networking, PCB, power and cooling near $1.34 million - about $7.8 million in total, roughly double today's GB300 NVL72 rack - with Bernstein revising to about $9.1 million after HBM4 and LPDDR5X moved together. Memory's share of the bill of materials rose from 9% to 26%.

Optical interconnect is the other bottleneck. At CIOE in Shenzhen on 9-11 September, exhibitors across modules, chips, fibre, connectors and test equipment described tight supply with some orders already booked into 2027; 800G remains the volume product for 2026 while 1.6T moves into limited mass production at leading vendors. Upstream pressure is spreading to optical chips, MPO and FAU assemblies, specialty fibre and high-speed electrical chips, and 8x200G 1.6T designs still depend on scarce DSP and driver silicon. Higher rates are also forcing architecture choices among LPO, NPO and CPO.

Power is now a first-order design constraint. Current 800G modules run below 12W and 1.6T modules below 21W, but aggregate networking draw in a large cluster is measured in megawatts, and every microsecond of added latency extends training job completion and leaves accelerators idle. AI traffic now accounts for roughly 30% of backbone network utilisation, up from under 1% two years ago, and Ethernet is projected to dominate a $100 billion AI back-end switch market by 2030. Fresh funding for optical switching startups and the OIF's 1.6T implementation agreement point the same way.