{"id":10308,"date":"2025-09-26T17:25:24","date_gmt":"2025-09-26T09:25:24","guid":{"rendered":"https:\/\/ascentoptics.com\/blog\/?p=10308"},"modified":"2026-07-09T11:38:18","modified_gmt":"2026-07-09T03:38:18","slug":"infiniband-ndr-xdr-for-ai-and-hpc-data-centers","status":"publish","type":"post","link":"https:\/\/ascentoptics.com\/blog\/infiniband-ndr-xdr-for-ai-and-hpc-data-centers\/","title":{"rendered":"InfiniBand NDR\/XDR for AI and HPC Data Centers"},"content":{"rendered":"

InfiniBand technology continues to evolve as a cornerstone for high-performance computing (HPC), artificial intelligence (AI), and data center networking. With the advent of NDR (Next Data Rate) and XDR (eXtreme Data Rate) specifications, InfiniBand delivers unprecedented bandwidth, low latency, and scalability to meet the demands of modern workloads.<\/p>\n

 <\/p>\n

What is InfiniBand NDR\/XDR Technology?<\/strong><\/h2>\n

InfiniBand NDR<\/a> operates at 400 Gb\/s per port, utilizing PAM4 modulation and advanced SerDes technology to achieve high-speed data transfer. It supports configurations like 64 x 400 Gb\/s ports or 128 x 200 Gb\/s ports in switches, making it ideal for large-scale clusters. XDR advances this further, providing 800 Gb\/s per port and enabling up to 1.6 Tb\/s throughput in aggregated setups, with features like enhanced in-network computing for AI acceleration.<\/p>\n

NDR builds on previous generations like HDR (200 Gb\/s), doubling speeds and improving efficiency, while XDR represents the next leap, targeting exascale computing and massive AI models by 2025 standards.<\/p>\n

 <\/p>\n

Key Benefits for AI and HPC Applications<\/strong><\/p>\n

NDR and XDR InfiniBand solutions offer ultra-low latency (sub-microsecond), high bandwidth for parallel processing, and in-network computing via technologies like NVIDIA SHARP, which accelerates collective operations by up to 32 times. These benefits are crucial for AI training, where data movement between GPUs must be seamless, reducing bottlenecks in large-scale models. In HPC, they enable simulations with over a million nodes, supporting applications in climate modeling, drug discovery, and scientific research. Additionally, features like self-healing networks and adaptive routing ensure reliability, while power efficiency improvements make them suitable for sustainable data centers.<\/p>\n

 <\/p>\n

\"Key<\/p>\n

 <\/p>\n

 <\/p>\n

InfiniBand Switches<\/strong><\/h2>\n

InfiniBand switches<\/a> form the backbone of these networks, providing scalable interconnects for high-density environments.<\/p>\n

 <\/p>\n

NVIDIA Quantum\u2122-2 InfiniBand Switch (NDR)<\/strong><\/p>\n

The NVIDIA Quantum-2 switch family, including models like MQM9790-NS2F and MQM9700-NS2F, delivers 64 x 400 Gb\/s ports or 128 x 200 Gb\/s ports in a compact 1U form factor using 32 OSFP connectors. It incorporates third-generation SHARP for in-network aggregation, MPI acceleration, and advanced congestion control. Compared to predecessors, it triples port density and quintuples system capacity, supporting Dragonfly+ topologies for networks exceeding one million nodes.<\/p>\n

 <\/p>\n

NVIDIA Quantum-X800 InfiniBand Switch (XDR)<\/strong><\/p>\n

The Quantum-X800 series, such as the Q3200-RA and Q3400-RA, offers 144 x 800 Gb\/s ports over OSFP cages, powered by the Quantum-3 ASIC. It features fourth-generation SHARP v4, ultra-low latency, self-healing, and silicon photonics for enhanced performance. The dual-switch enclosure in models like Q3200-RA provides 72 x 800 Gb\/s ports, ideal for AI clusters requiring 1.6 Tb\/s aggregated bandwidth.<\/p>\n

 <\/p>\n

\"Quantum-2<\/p>\n

 <\/p>\n

 <\/p>\n

Optical Modules and Transceivers<\/strong><\/h2>\n

Optical modules and transceivers are core components enabling high-speed, low-loss data transmission in InfiniBand networks. With the advancement of NDR and XDR technologies, these modules adopt cutting-edge modulation schemes (such as PAM4) and packaging formats (such as OSFP and OSFP-XD), supporting bandwidths from 400 Gb\/s up to 1.6 Tb\/s. They are well-suited for AI data centers and high-performance computing (HPC) environments. In recent years, innovations like silicon photonics and Linear-Drive Optics (LPO) have further reduced power consumption and costs while enhancing performance and integration.<\/p>\n

 <\/p>\n

OSFP-SR8-800G InfiniBand Optical Module Technology Overview (NDR\/XDR)<\/strong><\/h3>\n

The OSFP-SR8-800G<\/a> is a dual-port 2\u00d7400 Gb\/s (total 800 Gb\/s) multimode module, utilizing 100G-PAM4 modulation across 8 channels. It supports transmission distances up to 50 meters over MPO-12\/APC fibers and integrates dual optical engines for high-density switching. Leveraging silicon photonics technology, the module ensures low insertion loss and excellent signal-to-noise ratio, making it ideal for short-reach data center interconnects. In practical deployment, it is seamlessly compatible with NVIDIA Quantum-2 or Quantum-X800 switches, while supporting hot-pluggable operation and Digital Diagnostic Monitoring (DDM) functions for real-time monitoring of temperature, power, and signal integrity.<\/p>\n

\"OSFP-SR8-800G\"<\/p>\n

 <\/p>\n

1.6T OSFP<\/a> Optical Transceivers (XDR-Specific)<\/strong><\/h3>\n

XDR-dedicated 1.6T OSFP transceivers, such as 2\u00d7DR4\/DR8 variants, operate over 500m single-mode fiber at 1310nm, powered by Broadcom\u2019s 5nm DSP to achieve \u226430W power consumption. They enable dual-port 800 Gb\/s configurations, which are critical for 1.6 Tb\/s links in AI data centers. These transceivers also support extended-reach variants such as FR4 (up to 2km) and LR4 (up to 10km), using CWDM or DWDM wavelength-division multiplexing to accommodate different data center scales. In addition, integrated LPO (Linear-Drive Optics) technology can further reduce power consumption to below 20W while maintaining low-latency performance, making them well-suited for large-scale AI training clusters.<\/p>\n

\"1.6T<\/p>\n

 <\/p>\n

InfiniBand OSFP-400G-SR4 Optical Module (NDR)<\/strong><\/h3>\n

The OSFP-400G-SR4<\/a> is a single-port 400 Gb\/s multimode transceiver for NDR, supporting up to 50m transmission over MPO-12 fiber. It is designed for switch-to-HCA connections, ensuring a low BER (bit error rate <10^-15) and full compatibility with the Quantum-2 ecosystem. In deployment, this module is often used for short-reach server-to-switch links and supports breakout cable configurations, such as 400G to 4\u00d7100G, for backward compatibility with EDR\/HDR systems. Certified versions are available through NVIDIA\u2019s LinkX series, ensuring zero compatibility issues and long-term stability.<\/p>\n

\"OSFP-400G-SR4\"<\/p>\n

 <\/p>\n

 <\/p>\n

Other Common InfiniBand Optical Module Types<\/strong><\/h2>\n

OSFP-DR8-800G and FR4 Variants<\/strong><\/p>\n

The OSFP-DR8-800G<\/a> module is designed for single-mode fiber, supporting 500m transmission at a 1310nm wavelength with PAM4 modulation, making it suitable for medium-reach data center interconnects. The FR4 variant extends the reach up to 2km, supports CWDM multiplexing, and keeps power consumption under 25W. These modules are particularly useful in XDR environments to handle data traffic across large-scale GPU clusters.<\/p>\n

 <\/p>\n

Silicon Photonics Integrated Modules<\/strong><\/p>\n

Silicon photonics technology was a highlight at the 2025 OFC Conference, enabling significant reductions in module size and cost by integrating lasers, modulators, and detectors onto a silicon chip. For example, NVIDIA\u2019s latest silicon photonics transceivers achieve optimized power consumption at 800G transmission and support Co-Packaged Optics (CPO) architectures, further shortening signal paths and reducing latency. Such modules are ideal for AI accelerators like DGX systems, offering higher integration density.<\/p>\n

 <\/p>\n

LPO and CPO Innovations<\/strong><\/p>\n

Linear-Drive Optics (LPO) transceivers eliminate traditional DSPs and rely on host-side signal processing, further reducing power consumption to below 15W and accelerating adoption in NDR\/XDR deployments. Co-Packaged Optics (CPO) integrates optical engines directly alongside ASICs and is expected to become mainstream after 2025, targeting exascale computing. These innovations address the thermal management and power challenges of AI workloads.<\/p>\n

 <\/p>\n

Selection Considerations for Optical Modules<\/strong><\/p>\n

When choosing InfiniBand optical modules, several factors need to be taken into account, including transmission distance, power budget, compatibility, and cost. Short-reach SR modules are ideal for in-rack connections, while DR\/FR modules are better suited for inter-rack or inter-floor connectivity. It is also essential to ensure that modules comply with NVIDIA\u2019s interconnect requirements, including cable types and BER testing.<\/p>\n

Furthermore, trends in 2025 indicate that growing AI demand is driving supply chain optimization for optical modules, with production capacity expected to double in order to meet the expansion of global data centers.<\/p>\n

 <\/p>\n

 <\/p>\n

Typical Application Scenarios of NDR\/XDR Optical Connectivity<\/strong><\/h2>\n

(1) In-Rack GPU Node Interconnection<\/strong><\/p>\n

Solution:<\/strong> DAC (Direct Attach Copper)<\/p>\n

Features:<\/strong> Ultra-low latency, 0\u20133m reach, and low power consumption; ideal for direct connections between GPUs and switches within the same rack.<\/p>\n

 <\/p>\n

(2) Cross-Rack GPU Cluster Interconnection<\/strong><\/p>\n

Solution:<\/strong> AOC (Active Optical Cable)<\/p>\n

Features:<\/strong> Supports 10\u2013100m reach, flexible cabling; suitable for GPU cluster deployments across adjacent or multiple racks.<\/p>\n

 <\/p>\n

(3) Intra-Data Center TOR\/Leaf-Spine Networking<\/strong><\/p>\n

Solution:<\/strong> Multimode\/Single-mode Optical Patch Cords<\/p>\n

Features:<\/strong> Supports 100\u2013500m transmission, with typical breakout rates such as 800G to 4\u00d7200G or 400G to 4\u00d7100G; widely used for switch-to-switch interconnects or room-to-room connections.<\/p>\n

 <\/p>\n

(4) AI Supercomputing Backbone Interconnect<\/strong><\/p>\n

Solution:<\/strong> Single-mode Fiber with Long-Reach Optical Modules (e.g., DR4, FR4, LR4)<\/p>\n

Features:<\/strong> Covers 500m\u201310km distances, ensuring high-speed connectivity for large-scale GPU clusters and multi-floor\/campus-wide AI factories.<\/p>\n

 <\/p>\n

Typical Connectivity Solution Examples<\/strong><\/h3>\n

800G GPU to 4\u00d7200G HCA Connection:<\/strong> One 800G optical patch cord split into 4\u00d7200G channels via breakout cables, enabling a single GPU to connect to multiple servers.<\/p>\n

Switch Spine\u2013Leaf Architecture Interconnect:<\/strong> Using OSFP-800G DR4\/FR4 modules with single-mode fibers to achieve 500m\u20132km Spine\u2013Leaf connections, ensuring low latency and high bandwidth for large-scale clusters.<\/p>\n

High-Density In-Rack Direct Attach:<\/strong> 1.5m DAC cables for GPU-to-GPU or GPU-to-switch links within the same rack, delivering high bandwidth with minimal latency.<\/p>\n

 <\/p>\n

\"NDR\/XDR<\/p>\n

 <\/p>\n

 <\/p>\n

Future Outlook<\/strong><\/h2>\n

As AI factories and exascale supercomputing continue to expand, the demand for ultra-high bandwidth and energy-efficient interconnects will intensify. Silicon photonics, Co-Packaged Optics (CPO), and Linear-Drive Optics (LPO) are expected to play a pivotal role in addressing power and thermal challenges while enabling higher integration density.<\/p>\n

Looking ahead, InfiniBand will remain at the core of next-generation data centers, evolving from NDR to XDR and eventually to GDR (1.6 Tb\/s). This progression will not only support massive AI training clusters but also drive innovations in in-network computing, real-time data analytics, and HPC workloads at unprecedented scales.<\/p>\n

 <\/p>\n

 <\/p>\n

Conclusion<\/strong><\/h2>\n

InfiniBand NDR and XDR are key technologies for building high-performance AI and HPC networks. NDR provides mature 400G connectivity for current GPU clusters and supercomputing workloads, while XDR extends performance to 800G and 1.6T for next-generation AI data centers.<\/p>\n

When choosing an InfiniBand solution, it is important to consider bandwidth requirements, transmission distance, switch and adapter compatibility, power consumption, thermal design, and cable or optical module type. DAC and AOC are suitable for short-distance connections, while SR, DR, FR, and LR optical modules support wider deployment scenarios from in-rack links to data center backbone networks.<\/p>\n

As AI workloads continue to grow, technologies such as silicon photonics, LPO, and CPO will further improve power efficiency and bandwidth density. From NDR to XDR and future GDR networks, InfiniBand will remain a critical foundation for scalable, low-latency AI and HPC infrastructure.<\/p>\n

 <\/p>\n

 <\/p>\n

FAQs About InfiniBand NDR\/XDR<\/strong><\/h2>\n

1. What is the main difference between InfiniBand NDR and XDR?<\/h3>\n

The main difference is port speed and scalability. NDR provides 400Gb\/s per port, while XDR increases the speed to 800Gb\/s per port and can support up to 1.6Tb\/s aggregated connectivity in certain configurations. NDR is suitable for current high-performance AI and HPC clusters, while XDR is designed for larger next-generation AI data centers and exascale computing environments.<\/p>\n

2. When should I choose XDR instead of NDR?<\/h3>\n

You should consider XDR when your network requires higher GPU-to-GPU bandwidth, larger AI training clusters, 800G port speeds, or 1.6T aggregated links. XDR is also better suited for future-ready AI factories, exascale HPC systems, and environments where network bottlenecks may limit GPU utilization.<\/p>\n

Which switches are used for NDR and XDR InfiniBand networks?<\/h3>\n

For NDR deployments, NVIDIA Quantum-2 InfiniBand switches are commonly used. They support 64 \u00d7 400G ports or 128 \u00d7 200G breakout configurations. For XDR deployments, NVIDIA Quantum-X800 switches are used, offering up to 144 \u00d7 800G ports and enhanced capabilities such as SHARP v4, self-healing network intelligence, and support for massive AI clusters.<\/p>\n

What network adapters are used with NDR and XDR?<\/h3>\n

NDR networks typically use NVIDIA ConnectX-7 InfiniBand adapters, which support 400Gb\/s throughput and PCIe 5.0\/4.0 interfaces. XDR networks use ConnectX-8 SuperNICs, which support 800Gb\/s throughput, PCIe 6.0, and enhanced acceleration features for large-scale AI workloads.<\/p>\n

What optical modules are commonly used in NDR\/XDR InfiniBand networks?<\/h3>\n

Common options include 400G OSFP SR4 modules for NDR, 800G OSFP SR8 or DR4 modules for NDR\/XDR environments, and 1.6T OSFP transceivers for XDR high-bandwidth deployments. SR modules are typically used for short-distance multimode links, while DR, FR, and LR modules are used for longer single-mode fiber connections.<\/p>\n

What is the difference between DAC, AOC, and optical transceiver solutions?<\/h3>\n

DAC is a copper cable solution used for very short in-rack connections, usually 0\u20133m. It offers low latency and low cost. AOC is an active optical cable used for longer rack-to-rack connections, typically 10\u2013100m. Optical transceiver solutions with fiber patch cords are used for longer distances, such as 100\u2013500m, 2km, or even 10km depending on whether SR, DR, FR, or LR modules are selected.<\/p>\n

What fiber type should be used for NDR\/XDR optical links?<\/h3>\n

For short-reach multimode connections, OM4 or OM5 fiber is commonly used with SR modules. For medium- and long-reach links, OS2 single-mode fiber is used with DR, FR, or LR modules. The correct fiber type depends on transmission distance, module type, connector interface, and network design.<\/p>\n

How should I choose between SR, DR, FR, and LR optical modules?<\/h3>\n

Choose based on distance and fiber infrastructure. SR modules are best for short multimode links inside or between nearby racks. DR modules are suitable for around 500m single-mode connections. FR modules are commonly used for up to 2km links. LR modules are used for longer 10km-class connections, such as campus or large data center backbone links.<\/p>\n

Is InfiniBand better than Ethernet for AI and HPC?<\/h3>\n

For AI training and HPC environments that require extremely low latency, RDMA, lossless transport, and high-performance collective communication, InfiniBand often provides stronger performance than traditional Ethernet. Ethernet remains widely used in general data center networking, but InfiniBand is especially valuable in GPU clusters and supercomputing workloads where network efficiency directly affects compute performance.<\/p>\n

 <\/p>\n

 <\/p>\n