Challenges for AI and Data Centers
AI's Data Center Solution Requirements
As AI technology advances, the demands on data center components are increasing exponentially. Every critical element in the compute chain must rise to the challenge by delivering enhanced performance whilst consuming less power, providing greater bandwidth, achieving higher density, and doing so at a reduced cost.
The Power Challenge
McKinsey predicts that data center power consumption will represent ~12% of total US demand in 2030.
Training large language models (LLMs) requires immense computational demands, often taking weeks or even months to complete. As these models continue to grow in size and complexity, the required resources also increase. Hence, it is essential to develop new technologies that reduce power consumption across the entire compute chain.
Source: Global Energy Perspective 2023, McKinsey, October 18, 2023; McKinsey analysis
AI and LLMs Limited by Memory Bandwidth
LLM parameter count grew >100,000 faster than GPU HBM memory bandwidth over the last six years
LLMs, such as GPT-4 (est. >1 trillion parameters) require more data to flow between memory and xPUs. Each parameter represents a bias that is updated during training, so larger models require more data retrieval, processing and writing, increasing the bandwidth needs for efficient execution.
Therefore, there is a need for low power, high bandwidth, low cost, reliable interconnect solutions to overcome these challenges.
µLED Interconnects Deliver Best-in-Class Energy Efficiency and Bandwidth Density
While electrical interconnects work well over short distances of a few millimeters, they face fundamental limits in terms of reach, size and power efficiency, especially at higher data rates and longer distances. Avicena’s LightBundle™ interconnects use arrays of microLEDs connected via multi-core fiber bundles to Photo Detectors on CMOS ICs. This enables ultra-low power links of < 1pJ/bit with up to 10m reach.
LightBundle™ — Using microLEDs to “move data”
The parallel nature of the LightBundle interconnect is well matched to the wide internal bus architecture of ICs like CPUs, GPUs or switch ASICs, therefore eliminating the need for a power intensive SerDes interface. In addition, Avicena’s LightBundle is a 2D µLED array making it possible for the IO to cover the entire IC area, thereby increasing the BW density further.
LightBundle Achieving Class Leading Bandwidth Density
The figure illustrates how bandwidth density scales with per-lane data rates. With the current 4 Gbps lane data rate and 50 µm pitch , the LightBundle™ achieves bandwidth densities > 2 Tbps/mm.
The next generation of devices, featuring a reduced 25 µm pitch, will enable bandwidth densities exceeding 10 Tbps/mm.
Shoreline Bandwidth Density vs Per-lane data rate for existing and next generation LightBundle Transceivers
LightBundle™: 10x Better Power and Density
The performance of an interconnect can be expressed in terms of a figure of merit (FoM) combining bandwidth density on the edge of chip (Gbps/mm) with energy efficiency (pJ/bit). Avicena’s LightBundle™ optical interconnects can achieve power and density FoMs which are >10x Better than Existing Solutions.
The chart shows the LightBundle interconnects match the performance of electrical links whilst increasing the reach up to ~10m, allowing more efficient scale-up AI architectures.
Product Roadmap for AI/ML Applications
Avicena’s On-Board Optics (OBO) solution consists of an array of 1.6Tbps OBO transceivers that can be placed close to the xPU, significantly reducing power consumption, boosting bandwith density, and facilitating xPU scale-up in data centers.
OBO LightBundle Solution
CPO LightBundle Solution
Avicena’s OBOs are an essential step towards Co-Packaged Optics (CPO), which further enhance bandwidth density and efficiency by directly integrating optical components within the processing units. This enables higher data throughput and energy efficiency for data-intensive applications like AI and high-performance computing.
Learn more about different applications for Avicena’s LightBundle Technology.