NVIDIA InfiniBand switches form the scale-out network behind many of today’s high-performance AI and HPC clusters. The technology has evolved from 200 Gb/s HDR through 400 Gb/s NDR to 800 Gb/s XDR, increasing not only link bandwidth but also switch radix, telemetry, congestion-control capabilities, and in-network computing performance.
The current NDR generation is built around the NVIDIA Quantum-2 platform. The newer Quantum-X800 platform, based on the NVIDIA Quantum-3 switch ASIC, raises the port rate to 800 Gb/s. NVIDIA has also introduced Quantum-X InfiniBand Photonics. Unlike the pluggable-optics Q3200 and Q3400, the Q3450-LD integrates co-packaged silicon photonics and exposes optical connectivity directly through front-panel MPO interfaces.
Understanding these differences is important because the switch, network adapter, cable, and optical transceiver must be treated as one complete interconnect system.
An InfiniBand switch is a high-bandwidth, low-latency network device that forms the backbone of an InfiniBand fabric. It routes data between compute nodes, storage systems, and accelerators using remote direct memory access (RDMA) and credit-based flow control, delivering switching latency measured in hundreds of nanoseconds rather than microseconds.
Unlike a traditional Ethernet switch, an InfiniBand switch operates in a fabric that is centrally managed by a Subnet Manager. This central control is what lets InfiniBand guarantee in-order, lossless delivery, a property that matters enormously when thousands of GPUs must exchange gradients without a single dropped packet stalling a training job.
The key technical characteristics that set an InfiniBand switch apart include:
These properties are why InfiniBand, and not general-purpose Ethernet, became the default fabric for the world’s largest supercomputers and AI training clusters.
NVIDIA, which acquired the technology through Mellanox, ships three active InfiniBand switch generations. Understanding the difference between them is the first step in sizing a fabric correctly.
| Generation | Representative Switch | Maximum Ports | Per-Port Speed | Physical Interface |
| HDR / Quantum | QM8700 | Up to 40 | 200 Gb/s | QSFP56 |
| NDR / Quantum-2 | QM9700 / QM9790 | 64 | 400 Gb/s | 32 twin-port OSFP cages |
| XDR / Quantum-X800 | Q3400 | 144 | 800 Gb/s | 72 twin-port OSFP cages |
The workhorse for most current deployments is the Quantum-2 NDR platform. The QM9700 is a 1U switch with 64 ports of 400 Gb/s InfiniBand across 32 OSFP connectors. It delivers 51.2 Tb/s of aggregate bidirectional throughput and processes more than 66.5 billion packets per second. The QM9790 is the unmanaged sibling, intended for fabrics where the Subnet Manager runs on external hosts.
Two details worth noting about NDR. First, the QM9700 supports NDR200 port splitting, which breaks each 400 Gb/s port into two 200 Gb/s ports for a total of 128 ports. Second, QM9700 has an on-board Subnet Manager that can manage fabrics of roughly 2,000 nodes, while QM9790 leaves that job to a separate management node.
At the high end, the Quantum-X800 XDR line doubles the per-port rate to 800 Gb/s. The Q3400 packs 144 XDR ports into a 4U chassis for 115.2 Tb/s of throughput, while the Q3200 fits two independent 36-port switches into a single 2U chassis. These are the switches shipping into the largest Blackwell-era GPU clusters, where a two-level fat-tree can connect more than 10,000 accelerators.
Several architectural mechanisms distinguish InfiniBand from a conventional Ethernet fabric.
The Subnet Manager is responsible for discovering and configuring the InfiniBand subnet. It assigns LIDs, calculates routes, manages topology information, configures partitions, and distributes fabric parameters.
This centralized fabric-management model means InfiniBand does not rely on protocols such as Spanning Tree or BGP to establish ordinary intra-fabric forwarding paths.
Production environments can deploy redundant Subnet Managers or UFM-based management architectures to avoid relying on a single management instance.
InfiniBand supports multiple Virtual Lanes to separate traffic classes over the same physical link. Up to 15 Virtual Lanes can carry normal traffic, while VL15 is reserved for subnet-management traffic.
Packets carry a Service Level (SL), and switches map SL values into Virtual Lanes according to configured QoS policies.
Virtual Lanes do not magically create additional physical bandwidth, but they allow administrators to prioritize and isolate different classes of traffic and reduce head-of-line blocking.
Modern NVIDIA Quantum switches support adaptive routing and telemetry-driven congestion-control mechanisms.
Instead of treating every path as permanently fixed regardless of conditions, the fabric can use multiple available paths and react to congestion.
Together with credit-based flow control and QoS, these capabilities help maintain predictable throughput under the synchronized traffic patterns common in distributed AI training.
One of NVIDIA InfiniBand’s most distinctive features is SHARP — Scalable Hierarchical Aggregation and Reduction Protocol.
SHARP allows collective reduction operations to be partially processed inside the network rather than sending all intermediate data back to CPUs or GPUs.
For operations such as AllReduce, this can reduce the amount of data traversing the fabric and lower endpoint processing overhead. Quantum-2 implements SHARPv3, while Quantum-X800 introduces SHARPv4.
NVIDIA states that Quantum-X800 delivers up to 9× higher in-network computing performance than the previous generation, and SHARPv4 adds support for FP8 operations used in modern AI workloads.
Actual application-level improvement depends on the workload, collective algorithm, message size, topology, NCCL/MPI implementation, and cluster configuration.
There is no universal GPU-count threshold at which Ethernet stops working and InfiniBand becomes mandatory. Modern AI networks can be built successfully with either InfiniBand or properly engineered RoCE Ethernet. The architectural differences are more useful than simple latency or cost numbers.
| Dimension | InfiniBand | AI Ethernet / RoCE |
| Fabric model | InfiniBand-native switched fabric | Ethernet-based |
| Flow control | Native credit-based InfiniBand flow control | RoCE designs use Ethernet congestion and flow-control mechanisms |
| RDMA | Native InfiniBand transports | RoCE |
| Routing / congestion | Adaptive routing and InfiniBand congestion control | Depends on switch architecture; modern AI Ethernet supports advanced multipath and congestion control |
| In-network computing | NVIDIA SHARP on Quantum | Platform dependent |
| Standards ecosystem | Standardized by IBTA; vendor ecosystem relatively concentrated | Broad multi-vendor Ethernet ecosystem |
| Typical strength | Tightly coupled AI training and HPC | AI training, inference, cloud and converged Ethernet infrastructure |
InfiniBand remains the performance leader for large-scale training. The two technologies are converging, though. The Ultra Ethernet Consortium released its UEC 1.0 specification in mid-2025, adding multipath delivery and credit-based flow control to Ethernet. NVIDIA’s own Spectrum-X platform brings RDMA and adaptive routing to Ethernet as well.
The practical guidance: for training clusters above 1,000 GPUs, InfiniBand is usually the right call. For inference serving or sub-100-GPU setups, Ethernet is often sufficient and cheaper.
InfiniBand links can use DAC, active copper, AOC, or pluggable optical transceivers depending on distance, power, cabling, and serviceability. Passive DAC is best for very short in-rack connections, active copper extends copper reach, AOC is suitable for short-to-medium optical links, while pluggable transceivers are preferred for structured cabling and longer distances.
For NDR 400G, NVIDIA Quantum-2 switches use twin-port OSFP, with each switch-side OSFP carrying two independent 400G ports. ConnectX-7 endpoints may use either OSFP or QSFP112, so the two ends of the link do not necessarily use the same form factor. Common optical options include SR4 for short multimode links, DR4 for single-mode links up to approximately 500 m, and twin-port 2×DR4 or 2×FR4 options on the switch side.
For XDR 800G, Quantum-X800 Q3200 and Q3400 switches use 1.6T twin-port OSFP, providing two independent 800G XDR ports per cage. ConnectX-8 endpoints typically use 800G single-port OSFP. A representative optical link is therefore:
1.6T 2×DR4 Twin-Port OSFP → 2 × 800G DR4 Links → 2 × 800G OSFP ConnectX-8 Endpoints
| Interconnect | Typical Use | Key Point |
| DAC | Very short in-rack links | Lowest power and cost |
| Active Copper | Short links beyond passive DAC reach | Extends copper distance |
| AOC | Short-to-medium optical links | Simple integrated optical cable |
| Optical Transceiver | Structured cabling and longer reach | Flexible and field-replaceable |
| NDR Optics | Quantum-2 / ConnectX-7 | Twin-port OSFP switch side; OSFP or QSFP112 endpoint |
| XDR Optics | Quantum-X800 / ConnectX-8 | 1.6T twin-port OSFP switch side; 800G OSFP endpoint |
When selecting an interconnect, verify the complete link rather than the nominal speed alone:
Switch → Switch-Side Optic/Cable → Fiber → Endpoint Optic/Cable → NIC
Form factor, cooling design, optical specification, connector, fiber type, reach, and platform compatibility must all match the exact NVIDIA switch and adapter.
Compatibility is the top pain point for engineers buying third-party optics. NVIDIA programs its own transceivers with vendor-specific coding, and a stock module will often be rejected by the switch unless it carries the correct identifier.
When selecting an InfiniBand optical transceiver, verify these points before ordering:
The InfiniBand Trade Association has standardized the physical layer in its Volume 2 Release 2.0 specification, which covers 800 Gb/s links and adds CMIS 5.3 transceiver management. Sticking to transceivers built to these specs reduces the risk of a mismatch.
The switch chassis is only one part of the InfiniBand network budget.
A realistic budget should include the switch systems, server-side adapters, DAC/AOC or optical transceivers, fiber assemblies, patch panels, cable management, power, cooling, management software, installation labor, and operational spares.
Optical costs become especially important at NDR and XDR speeds because high-radix switches can require dozens of high-speed interconnect devices per chassis.
For example, a single QM9700 exposes 64 × 400G NDR ports, while a Q3400 exposes 144 × 800G XDR ports.
At that density, even a relatively small difference in cost, power, or failure rate per optical link becomes significant when multiplied across an entire cluster.
For third-party optics, the goal should therefore not simply be to find the lowest-cost module.
The more important requirement is to obtain a module that is correctly coded, thermally appropriate, optically compliant, firmware-compatible, and validated on the exact NVIDIA switch or adapter platform.
A failed compatibility decision can cost much more in GPU downtime than the price difference between two transceivers.
An NVIDIA InfiniBand switch is the backbone of a serious AI training fabric, but it does not work alone. The takeaways from this guide:
Your next step is to map your switch ports to the interconnects they require. If you are building an InfiniBand fabric, getting that mapping right is the difference between a cluster that lights up on day one and one that sits dark for a week.
An NVIDIA InfiniBand switch connects GPU servers, storage, and other InfiniBand endpoints in a high-bandwidth, low-latency RDMA fabric. Modern platforms include Quantum HDR, Quantum-2 NDR, and Quantum-X800 XDR.
NDR provides up to 400 Gb/s per port, while XDR increases this to 800 Gb/s per port. XDR also provides higher switch radix and newer SHARP in-network computing capabilities.
The QM9700 provides 64 × 400G NDR ports through 32 twin-port OSFP cages. Each physical OSFP cage carries two independent 400G ports.
Both provide 64 × 400G NDR ports. The QM9700 includes onboard management and Subnet Manager capability, while the QM9790 is designed for externally managed fabrics.
Both are available depending on the adapter model and system design. Therefore, always confirm the exact ConnectX-7 interface before selecting cables or optical transceivers.
Quantum-X800 switches can use 1.6T twin-port OSFP modules carrying two 800G XDR links, while ConnectX-8 endpoints typically use 800G single-port OSFP modules.
Not automatically. Even if the speed and form factor match, electrical mapping, firmware coding, optical specification, and platform qualification must also be compatible with the InfiniBand switch or NIC.
No. InfiniBand is highly optimized for tightly coupled AI training and HPC, while modern RoCE Ethernet can also support large GPU clusters. The choice depends on workload, scale, congestion-control requirements, operations, and total cost.
