An InfiniBand switch is a high-bandwidth, low-latency network device that forms the backbone of an InfiniBand fabric. It routes data between compute nodes, storage systems, and accelerators using remote direct memory access (RDMA) and credit-based flow control, delivering switching latency measured in hundreds of nanoseconds rather than microseconds.<\/p>\n
Unlike a traditional Ethernet switch, an InfiniBand switch operates in a fabric that is centrally managed by a Subnet Manager. This central control is what lets InfiniBand guarantee in-order, lossless delivery, a property that matters enormously when thousands of GPUs must exchange gradients without a single dropped packet stalling a training job.<\/p>\n
<\/p>\n
The key technical characteristics that set an InfiniBand switch apart include:<\/p>\n
- \n
- \u2022<\/span>Sub-microsecond port-to-port latency<\/strong>, typically cited around 0.6 \u00b5s for current generations<\/li>\n
- \u2022<\/span>Credit-based flow control<\/strong>, which prevents packet loss at the link layer rather than relying on retransmission<\/li>\n
- \u2022<\/span>Native RDMA support<\/strong>, allowing one node to write directly into another node’s memory without involving the CPU<\/li>\n
- \u2022<\/span>Very high radix<\/strong>, with a single switch exposing 64 to 144 high-speed ports to keep hop counts low<\/li>\n<\/ul>\n
<\/p>\n
These properties are why InfiniBand, and not general-purpose Ethernet, became the default fabric for the world’s largest supercomputers and AI training clusters.<\/p>\n
<\/p>\n
<\/p>\n
NVIDIA InfiniBand Switch Generations: HDR, NDR, and XDR<\/strong><\/h2>\n
NVIDIA, which acquired the technology through Mellanox, ships three active InfiniBand switch generations<\/u><\/a>. Understanding the difference between them is the first step in sizing a fabric correctly.<\/p>\n
<\/p>\n
\n\n
\n Generation<\/b><\/strong><\/td>\n Representative Switch<\/b><\/strong><\/td>\n Maximum Ports<\/b><\/strong><\/td>\n Per-Port Speed<\/b><\/strong><\/td>\n Physical Interface<\/b><\/strong><\/td>\n<\/tr>\n \n HDR \/ Quantum<\/td>\n QM8700<\/td>\n Up to 40<\/td>\n 200 Gb\/s<\/td>\n QSFP56<\/td>\n<\/tr>\n \n NDR \/ Quantum-2<\/td>\n QM9700 \/ QM9790<\/td>\n 64<\/td>\n 400 Gb\/s<\/td>\n 32 twin-port OSFP cages<\/td>\n<\/tr>\n \n XDR \/ Quantum-X800<\/td>\n Q3400<\/td>\n 144<\/td>\n 800 Gb\/s<\/td>\n 72 twin-port OSFP cages<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n <\/p>\n
The workhorse for most current deployments is the\u00a0Quantum-2 NDR<\/strong>\u00a0platform. The QM9700 is a 1U switch with 64 ports of 400 Gb\/s InfiniBand across 32 OSFP connectors. It delivers 51.2 Tb\/s of aggregate bidirectional throughput and processes more than 66.5 billion packets per second. The QM9790 is the unmanaged sibling, intended for fabrics where the Subnet Manager runs on external hosts.<\/p>\n
Two details worth noting about NDR. First, the QM9700 supports\u00a0NDR200 port splitting<\/strong>, which breaks each 400 Gb\/s port into two 200 Gb\/s ports for a total of 128 ports. Second, QM9700 has an on-board Subnet Manager that can manage fabrics of roughly 2,000 nodes, while QM9790 leaves that job to a separate management node.<\/p>\n
At the high end, the\u00a0Quantum-X800 XDR<\/strong>\u00a0line doubles the per-port rate to 800 Gb\/s. The Q3400 packs 144 XDR ports into a 4U chassis for 115.2 Tb\/s of throughput, while the Q3200 fits two independent 36-port switches into a single 2U chassis. These are the switches shipping into the largest Blackwell-era GPU clusters, where a two-level fat-tree can connect more than 10,000 accelerators.<\/p>\n
<\/p>\n
![\"HDR](\"https:\/\/ascentoptics.com\/blog\/wp-content\/uploads\/2026\/08\/HDR-NDR-XDR-Switch-Evolution.png\")
<\/p>\n<\/p>\n
<\/p>\n
<\/p>\n
How an InfiniBand Switch Works<\/strong><\/h2>\n
Several architectural mechanisms distinguish InfiniBand from a conventional Ethernet fabric.<\/p>\n
<\/p>\n
Subnet Manager<\/strong><\/h3>\n
The Subnet Manager is responsible for discovering and configuring the InfiniBand subnet. It assigns LIDs, calculates routes, manages topology information, configures partitions, and distributes fabric parameters.<\/p>\n
This centralized fabric-management model means InfiniBand does not rely on protocols such as Spanning Tree or BGP to establish ordinary intra-fabric forwarding paths.<\/p>\n
Production environments can deploy redundant Subnet Managers or UFM-based management architectures to avoid relying on a single management instance.<\/p>\n
<\/p>\n
Virtual Lanes and QoS<\/strong><\/h3>\n
InfiniBand supports multiple Virtual Lanes to separate traffic classes over the same physical link. Up to 15 Virtual Lanes can carry normal traffic, while VL15 is reserved for subnet-management traffic.<\/p>\n
Packets carry a Service Level (SL), and switches map SL values into Virtual Lanes according to configured QoS policies.<\/p>\n
Virtual Lanes do not magically create additional physical bandwidth, but they allow administrators to prioritize and isolate different classes of traffic and reduce head-of-line blocking.<\/p>\n
<\/p>\n
Adaptive Routing and Congestion Control<\/strong><\/h3>\n
Modern NVIDIA Quantum switches support adaptive routing and telemetry-driven congestion-control mechanisms.<\/p>\n
Instead of treating every path as permanently fixed regardless of conditions, the fabric can use multiple available paths and react to congestion.<\/p>\n
Together with credit-based flow control and QoS, these capabilities help maintain predictable throughput under the synchronized traffic patterns common in distributed AI training.<\/p>\n
<\/p>\n
SHARP In-Network Computing<\/strong><\/h3>\n
One of NVIDIA InfiniBand’s most distinctive features is SHARP \u2014 Scalable Hierarchical Aggregation and Reduction Protocol<\/strong>.<\/p>\n
SHARP allows collective reduction operations to be partially processed inside the network rather than sending all intermediate data back to CPUs or GPUs.<\/p>\n
For operations such as AllReduce, this can reduce the amount of data traversing the fabric and lower endpoint processing overhead. Quantum-2 implements SHARPv3, while Quantum-X800 introduces SHARPv4.<\/p>\n
NVIDIA states that Quantum-X800 delivers up to 9\u00d7 higher in-network computing performance<\/strong>\u00a0than the previous generation, and SHARPv4 adds support for FP8 operations used in modern AI workloads.<\/p>\n
Actual application-level improvement depends on the workload, collective algorithm, message size, topology, NCCL\/MPI implementation, and cluster configuration.<\/p>\n
<\/p>\n
![\"How](\"https:\/\/ascentoptics.com\/blog\/wp-content\/uploads\/2026\/08\/How-an-InfiniBand-Switch-Works.png\")
<\/p>\n<\/p>\n
<\/p>\n
<\/p>\n
InfiniBand vs Ethernet for AI Clusters<\/strong><\/h2>\n
There is no universal GPU-count threshold at which Ethernet stops working and InfiniBand becomes mandatory. Modern AI networks can be built successfully with either InfiniBand or properly engineered RoCE Ethernet. The architectural differences are more useful than simple latency or cost numbers.<\/p>\n
<\/p>\n
\n\n
\n Dimension<\/b><\/strong><\/td>\n InfiniBand<\/b><\/strong><\/td>\n AI Ethernet \/ RoCE<\/b><\/strong><\/td>\n<\/tr>\n \n Fabric model<\/td>\n InfiniBand-native switched fabric<\/td>\n Ethernet-based<\/td>\n<\/tr>\n \n Flow control<\/td>\n Native credit-based InfiniBand flow control<\/td>\n RoCE designs use Ethernet congestion and flow-control mechanisms<\/td>\n<\/tr>\n \n RDMA<\/td>\n Native InfiniBand transports<\/td>\n RoCE<\/td>\n<\/tr>\n \n Routing \/ congestion<\/td>\n Adaptive routing and InfiniBand congestion control<\/td>\n Depends on switch architecture; modern AI Ethernet supports advanced multipath and congestion control<\/td>\n<\/tr>\n \n In-network computing<\/td>\n NVIDIA SHARP on Quantum<\/td>\n Platform dependent<\/td>\n<\/tr>\n \n Standards ecosystem<\/td>\n Standardized by IBTA; vendor ecosystem relatively concentrated<\/td>\n Broad multi-vendor Ethernet ecosystem<\/td>\n<\/tr>\n \n Typical strength<\/td>\n Tightly coupled AI training and HPC<\/td>\n AI training, inference, cloud and converged Ethernet infrastructure<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n <\/p>\n
InfiniBand remains the performance leader for large-scale training. The two technologies are converging, though. The Ultra Ethernet Consortium released its UEC 1.0 specification in mid-2025, adding multipath delivery and credit-based flow control to Ethernet. NVIDIA’s own Spectrum-X platform brings RDMA and adaptive routing to Ethernet as well.<\/p>\n
The practical guidance: for training clusters above 1,000 GPUs, InfiniBand is usually the right call. For inference serving or sub-100-GPU setups, Ethernet is often sufficient and cheaper.<\/p>\n
<\/p>\n
![\"InfiniBand](\"https:\/\/ascentoptics.com\/blog\/wp-content\/uploads\/2026\/08\/InfiniBand-vs-Ethernet-for-AI-Clusters.png\")
<\/p>\n<\/p>\n
<\/p>\n
InfiniBand Optical Interconnect Options<\/strong><\/h2>\n
InfiniBand links can use DAC, active copper, AOC, or pluggable optical transceivers<\/strong>\u00a0depending on distance, power, cabling, and serviceability. Passive DAC<\/a> is best for very short in-rack connections, active copper<\/a> extends copper reach, AOC<\/a> is suitable for short-to-medium optical links, while pluggable transceivers are preferred for structured cabling and longer distances.<\/p>\n
For NDR 400G<\/strong><\/a>, NVIDIA Quantum-2 switches use twin-port OSFP<\/strong>, with each switch-side OSFP carrying two independent 400G ports. ConnectX-7 endpoints may use either OSFP or QSFP112<\/strong>, so the two ends of the link do not necessarily use the same form factor. Common optical options include SR4 for short multimode links, DR4 for single-mode links up to approximately 500 m, and twin-port 2\u00d7DR4 or 2\u00d7FR4 options on the switch side.<\/p>\n
- \u2022<\/span>Credit-based flow control<\/strong>, which prevents packet loss at the link layer rather than relying on retransmission<\/li>\n
![\"NDR](\"https:\/\/ascentoptics.com\/blog\/wp-content\/uploads\/2026\/08\/NDR-vs-XDR-Optical-Interconnect-Architecture2.png\")
<\/p>\n