InfiniBand is a high-performance, low-latency interconnect fabric dominating AI and HPC clusters. Evolving through generations—from SDR to the latest NDR and XDR—it doubles bandwidth per generation. Optical transceivers (QSFP, OSFP) enable long-reach connectivity while maintaining signal integrity. InfiniBand provides credit-based flow control, native RDMA capabilities, and highly predictable communication performance. In some platforms, in-network computing technologies can further accelerate collective operations.
InfiniBand is a switched-fabric interconnect standard for connecting servers, storage, and network equipment. The InfiniBand Trade Association (IBTA) describes it as an input/output architecture built on switched, point-to-point channels that can scale to thousands of nodes.
The term actually covers two related things. First, it is a physical link-layer protocol that defines how data moves across the fabric. Second, it is the InfiniBand Verbs API, an implementation of Remote Direct Memory Access (RDMA). This split matters because it explains why InfiniBand feels different from Ethernet: the protocol and the programming interface are designed together.
The measurable result is latency around 600 nanoseconds end-to-end, according to the IBTA. That number, combined with near-lossless delivery, is why InfiniBand remains the default choice for the largest AI training clusters.
InfiniBand’s performance comes from deliberate architecture choices, not a single trick. Three mechanisms do most of the work.
Traditional network data transfer is inefficient. A packet arrives, the operating system kernel receives it, figures out which application it belongs to, wakes that application, then copies the data from kernel memory into the application’s buffer. Each step adds CPU load and latency.
RDMA removes that path. The network adapter places incoming data directly into the target application’s memory. RDMA enables the network adapter to transfer data directly between application memory regions with minimal operating-system and CPU involvement, reducing data copies and software overhead. For a training job that passes gradients between thousands of GPUs thousands of times per second, that reduction in CPU overhead compounds quickly.
The trade-off is that RDMA cannot use the standard Berkeley Sockets API. Applications can access InfiniBand RDMA capabilities through APIs such as the InfiniBand Verbs interface, either directly or through higher-level communication libraries and frameworks.
Traditional Ethernet networks can rely on packet loss and retransmission mechanisms such as TCP. RoCEv2-based AI networks, however, use mechanisms such as PFC, ECN, and congestion-control algorithms to reduce packet loss and manage congestion for RDMA traffic. Under heavy load, this creates tail latency: a few slow packets stall an entire distributed job.
InfiniBand takes the opposite approach. A sender transmits only when the receiver has advertised buffer credits, so packets are not dropped under normal congestion. This credit-based flow-control mechanism is designed to prevent buffer overflow under normal operating conditions, providing a lossless or near-lossless fabric for supported traffic. Reliability and congestion management are built into the protocol layer rather than bolted on, which keeps latency predictable as a cluster scales.
InfiniBand fabrics are managed by a centralized subnet manager rather than a distributed Ethernet-style control plane. Fat-tree and related Clos-based topologies are widely used in large InfiniBand clusters because they provide predictable bandwidth and scalable connectivity. Most deployments use a fat-tree topology: leaf switches connect to servers, spine switches interconnect the leaves, and the hierarchy is sized so bandwidth stays consistent between any two nodes.
This is precision engineering with a learning curve. Mis-cabled links and uneven port usage are common causes of reduced bisection bandwidth, which is why InfiniBand deployments reward careful planning and monitoring.
Each InfiniBand generation roughly doubles the effective bandwidth of the last. Understanding the ladder is the first step toward choosing the right optics.
| Generation | Full Name | Data Rate | Typical Lane Configuration | Typical Form Factor |
| EDR | Enhanced Data Rate | 100 Gbps | 4 × 25 Gb/s NRZ | QSFP28 |
| HDR | High Data Rate | 200 Gbps | 4 × 50 Gb/s PAM4 | QSFP56 |
| NDR | Next Data Rate | 400 Gbps | 4 × 100 Gb/s PAM4 | OSFP |
| XDR | Extreme Data Rate | 800 Gbps | 8 × 100 Gb/s PAM4 | OSFP |
Earlier generations such as SDR (10 Gbps), DDR (20 Gbps), QDR (40 Gbps), and FDR (56 Gbps) are legacy today. New deployments almost always start at HDR or NDR.
HDR uses four 50 Gbps PAM4 lanes for 200 Gbps total. NDR doubles that with four 100 Gbps PAM4 lanes for 400 Gbps, which is why NDR switches to the larger OSFP form factor to handle the higher signal integrity and power requirements.
Latency improves with each step. EDR sits around 110 to 120 nanoseconds, HDR around 80 to 90, and NDR around 70 to 75. Those differences sound small, but in a collective operation run across hundreds of nodes, they translate into meaningfully faster job completion times. NDR also supports split-lane modes such as NDR200, which runs two 100 Gbps lanes for 200 Gbps.
The loudest question in AI networking right now is whether to build on InfiniBand or Ethernet. The honest answer is that the gap has narrowed, but it has not closed.
InfiniBand’s advantages are well documented. It delivers about 0.6 microseconds of latency, native RDMA, a lossless fabric, and predictable scaling, which makes it the gold standard for large-scale training clusters of 1,000 or more GPUs. NVIDIA owns the ecosystem through its Mellanox acquisition, so InfiniBand is deeply optimized for NCCL-based training.
Ethernet is fighting back. RoCEv2, which runs RDMA over converged Ethernet, now reaches 85 to 95 percent of InfiniBand throughput in well-tuned deployments. Well-engineered RoCEv2 Ethernet fabrics can deliver competitive performance for many AI workloads, although the actual performance difference depends on topology, NICs, switches, congestion control, and workload characteristics. Ethernet hardware costs roughly half as much, avoids single-vendor lock-in, and leverages skills most engineers already have. The Ultra Ethernet Consortium, backed by AMD, Arista, Broadcom, Cisco, and Meta, is standardizing Ethernet for AI and HPC.
Ethernet has gained significant momentum in AI networking, while InfiniBand remains an important choice for large-scale AI training and HPC deployments. InfiniBand is not disappearing, but it is increasingly the specialist’s choice for latency-sensitive training rather than the default.
A practical framework for 2026:
| Workload | Recommended Choice |
| LLM training over 1,000 GPUs | InfiniBand |
| Inference serving or enterprise AI | Ethernet |
| Mixed training and inference | Hybrid |
| HPC and scientific computing | InfiniBand |
| Clusters under 100 GPUs | Ethernet |
The decision comes down to scale, latency tolerance, and how much you value an open ecosystem. For the optical layer, however, the same fundamental question applies either way: which transceiver and cable get the signal where it needs to go.
This is where InfiniBand content usually stops, and where real deployments often stumble. The protocol layer gets all the attention, but the physical layer determines whether a cluster works at all.
InfiniBand transceivers map to specific form factors by generation. EDR 100G uses QSFP28, the same pluggable format used for 100G Ethernet. HDR 200G moves to QSFP56, which carries 50 Gbps per lane. OSFP provides a larger thermal and mechanical envelope suitable for high-speed NDR optical and copper connectivity.
Within each generation, distance variants match the optics to the physical reach. Short-reach (SR) modules use 850 nm multimode fiber with MPO-12 connectors and reach about 100 meters, which covers most intra-rack and adjacent-rack links. For longer runs, single-mode variants such as DR, FR, and LR extend reach from 500 meters to 2 kilometers or more using duplex LC or MPO/APC connectors.
The naming can trip up engineers coming from Ethernet, where “SR” and “LR” mean something slightly different at each speed. For InfiniBand specifically, an HDR SR4 QSFP56 module is a 4x50G PAM4 part over multimode fiber, while an NDR FR4 OSFP module uses 1310 nm single-mode fiber. Matching the distance variant to the cabling plant is as important as matching the form factor.
Not every InfiniBand link needs pluggable optics. Direct attach copper (DAC) cables work well for very short runs, up to about 3 meters for NDR and 5 meters for HDR. Active optical cables (AOC) extend that reach to tens of meters with integrated transceivers on each end. Pluggable optical transceivers remain the choice for longer distances and structured cabling.
DAC cables are commonly used for short intra-rack and adjacent-rack connections where low cost and low power consumption are priorities. The trade-off is reach and flexibility.
A simple checklist keeps most deployments on track. First, confirm the generation and data rate. Second, confirm the form factor: QSFP28, QSFP56, or OSFP. Third, match the distance variant to your fiber type and reach. Fourth, verify compatibility with the switch or adapter vendor. Finally, choose the cable type: DAC, AOC, or pluggable optics.
Get the first two right, and most of the rest follows. Get them wrong, and you get a shipping delay instead of a working link.
InfiniBand earns its place at the top of high-performance computing through deliberate design: RDMA removes kernel overhead, credit-based flow control keeps the fabric lossless, and each generation doubles bandwidth while trimming latency. Ethernet is closing the gap on cost and openness, which means the right choice now depends on your workload, scale, and tolerance for vendor lock-in.
Three takeaways to carry forward. First, match the speed generation to your collective-operation scale, not just the port label. Second, confirm the form factor, QSFP28, QSFP56, or OSFP, before you order anything. Third, choose the distance variant and cable type based on the actual reach of your links.
InfiniBand is primarily used for high-performance computing (HPC), AI and machine learning clusters, scientific computing, and other distributed workloads that require high bandwidth and low latency. It is particularly suitable for large-scale GPU clusters where efficient communication between compute nodes can have a significant impact on application performance.
InfiniBand is a purpose-built high-performance interconnect with native RDMA and credit-based flow control. Ethernet is a general-purpose networking technology that can support RDMA through RoCEv2. InfiniBand is widely used in AI training and HPC, while Ethernet offers broader multi-vendor interoperability and can support AI, inference, storage, cloud, and conventional data center workloads.
Yes. RDMA is a fundamental capability of InfiniBand. It allows data to be transferred directly between memory regions on different systems with minimal CPU and operating-system involvement, reducing software overhead and improving communication efficiency.
HDR, NDR, and XDR are successive generations of InfiniBand technology. HDR provides 200G-class connectivity, NDR provides 400G-class connectivity, and XDR represents the move toward 800G-class connectivity. Each generation increases bandwidth and improves the performance available for large-scale AI and HPC networks.
Common InfiniBand form factors include QSFP28, QSFP56, and OSFP. QSFP28 is commonly associated with 100G EDR, QSFP56 with 200G HDR, and OSFP with 400G NDR and newer 800G-class InfiniBand platforms. The correct form factor must always be verified against the specific switch or adapter.
A DAC (Direct Attach Copper) uses copper conductors and is typically used for short connections where low cost and low power consumption are important. An AOC (Active Optical Cable) integrates optical transceivers and fiber into a single cable assembly, allowing longer reach while maintaining a simple installation. The appropriate choice depends on link distance, power requirements, and deployment architecture.
The supported distance depends on the InfiniBand generation and optical module. SR modules are generally designed for short-reach multimode fiber connections, while DR, FR, and LR configurations provide longer single-mode fiber reach. Actual distances can range from tens of meters to several kilometers or more depending on the module specification and fiber infrastructure.
Yes. InfiniBand can use both optical transceivers with separate fiber cables and integrated optical cables such as AOCs. Optical connectivity is particularly useful for longer links and large AI or HPC clusters where copper cabling cannot provide the required reach.
Not necessarily. InfiniBand and Ethernet may use the same mechanical form factor or similar optical architectures, but the module must support the required protocol, signaling, firmware, and platform specifications. A QSFP56, OSFP, or other physically compatible module should not be assumed to work interchangeably between InfiniBand and Ethernet equipment.
Neither technology is universally better. InfiniBand is particularly well suited to large-scale AI training and HPC workloads that prioritize predictable low latency, native RDMA, and tightly integrated fabric management. Ethernet with RoCEv2 can be attractive when multi-vendor interoperability, infrastructure reuse, and integration with broader data center networks are important.
No. InfiniBand is used in a broad range of HPC and accelerated computing environments. However, NVIDIA has a particularly strong InfiniBand ecosystem following its acquisition of Mellanox, and InfiniBand is closely integrated with NVIDIA networking and GPU software technologies.
Start by confirming the InfiniBand generation and port speed, then verify the required form factor, optical reach, fiber type, connector, and platform compatibility. Finally, determine whether DAC, AOC, or pluggable optical transceivers are most appropriate for the link distance and deployment environment.
An InfiniBand upgrade should consider more than the switch port speed. Network operators should evaluate the switch and adapter generation, transceiver form factor, lane rate, optical reach, fiber infrastructure, connector type, power consumption, thermal requirements, breakout architecture, and compatibility with the existing network.
InfiniBand is expected to remain important in AI training, HPC, and other high-performance computing applications as networks move toward higher bandwidth. The transition from HDR to NDR and XDR is accompanied by advances in high-speed optical transceivers, electrical interfaces, thermal management, and network architectures. At the same time, Ethernet and RoCEv2 continue to evolve as alternatives for AI and accelerated computing.
