{"id":13213,"date":"2026-08-31T17:51:37","date_gmt":"2026-08-31T09:51:37","guid":{"rendered":"https:\/\/ascentoptics.com\/blog\/?p=13213"},"modified":"2026-08-31T17:51:37","modified_gmt":"2026-08-31T09:51:37","slug":"infiniband-troubleshooting","status":"publish","type":"post","link":"https:\/\/ascentoptics.com\/blog\/infiniband-troubleshooting\/","title":{"rendered":"InfiniBand Troubleshooting: A Step-by-Step Guide"},"content":{"rendered":"

As the core interconnect technology for high-performance computing and AI clusters, InfiniBand’s<\/u><\/a>\u00a0stability directly impacts overall training efficiency. Troubleshooting goes beyond simple link checks; it requires a systematic analysis integrating physical layer signal integrity, subnet manager status, routing strategies, and congestion control mechanisms. Common issues\u2014such as link speed degradation, symbol errors, or timeout alerts\u2014often mask underlying causes like thermal management problems, cable aging, or configuration drift. Bottlenecks can be rapidly identified by employing a layered diagnostic approach that progresses from port counters and Subnet Manager (SM) logs to end-to-end latency testing. Mastering this systematic methodology significantly reduces unplanned downtime and ensures the long-term, stable operation of large-scale distributed tasks.<\/p>\n

 <\/p>\n

 <\/p>\n

How InfiniBand Troubleshooting Differs from Ethernet<\/strong><\/h2>\n

Before touching a single cable, it helps to understand why diagnosing an InfiniBand fabric is not the same as debugging Ethernet. The two interconnects look similar at the port level, but they behave differently under the hood.<\/p>\n

InfiniBand employs an addressing mechanism based on Local Identifiers (LIDs) and Global Identifiers (GIDs). An InfiniBand fabric cannot initialize properly without a Subnet Manager (typically OpenSM or a vendor-supplied equivalent) running on the network\u2014a requirement that does not exist for Ethernet. Because InfiniBand utilizes kernel bypass technology to achieve Remote Direct Memory Access (RDMA), an interface may appear “down” in the output of the `ip a` command while actually remaining fully operational; unless IPoIB is enabled, the interface itself is not assigned an IP address.<\/p>\n

InfiniBand is also natively lossless, using credit-based flow control.\u00a0RoCEv2 runs over Ethernet and typically relies on mechanisms such as Priority Flow Control (PFC) and Explicit Congestion Notification (ECN) to manage congestion and reduce packet loss. InfiniBand uses link-level flow control and its own congestion-management mechanisms. As a result, troubleshooting priorities differ: InfiniBand investigations often focus on link state, Subnet Manager behavior, fabric configuration, and port counters, while RoCE troubleshooting also requires careful examination of Ethernet QoS, PFC, ECN, and congestion-control configuration.<\/p>\n

This matters for troubleshooting. When a link fails on InfiniBand, the fix is usually physical or subnet-manager related. When a link fails on RoCE, the fix is more often a lossless-Ethernet configuration problem. Knowing which interconnect you are on changes where you start looking.<\/p>\n

 <\/p>\n

\"InfiniBand<\/p>\n

 <\/p>\n

 <\/p>\n

Start at the Physical Layer: Cables, Optics, and Connectors<\/strong><\/h2>\n

Here is the single most important rule of InfiniBand troubleshooting: check the physical layer first. Cables, transceivers, and connectors are cheap to swap and easy to misdiagnose. They are also, by a wide margin, the most common source of intermittent link problems.<\/p>\n

 <\/p>\n

DAC vs AOC vs Pluggable Transceiver: Failure Symptoms<\/b><\/strong><\/h3>\n

InfiniBand links run over three main cable types, and each fails in a slightly different way.<\/p>\n

A direct attach copper (DAC) cable is passive and inexpensive, with a reach of roughly 0.5\u00a0to 3 meters. It draws almost no power and is immune to many optical issues, but it is sensitive to bend radius and signal integrity at higher speeds. A 400G or 800G DAC<\/a> pushed half a meter past its rated reach can begin producing pre-FEC errors that never show up as a hard link down.<\/p>\n

An active optical cable (AOC)<\/a> converts electrical signals to light inside the connector, extending reach to around 3 to 100 meters, and up to 150 meters on HDR fabrics. AOC failures tend to be intermittent. The link trains, drops, and retrains as a transceiver or fiber inside the assembly degrades. For example, an intermittent link may appear healthy during a quick inspection but repeatedly flap under sustained traffic.<\/p>\n

Pluggable optical transceivers<\/a> paired with fiber provide greater flexibility for longer links. Depending on the transceiver type and fiber, InfiniBand optical links can support reaches from short data-center connections to several kilometers.\u00a0Transceiver failures often surface as hard link-down events or as rising error counters rather than slow degradation.<\/p>\n

The takeaway: match the failure pattern to the cable type. A hard down with no negotiation points to a broken DAC or transceiver. An intermittent link points to a marginal AOC or fiber connection.<\/p>\n

 <\/p>\n

Reading the Link-Diagnostic Codes<\/b><\/strong><\/h3>\n

Modern InfiniBand switches, including the NVIDIA Quantum and Spectrum families, expose coded link-diagnostic values that tell you exactly why a port will not come up. These codes fall into two bands.<\/p>\n

Physical-layer indications, numbered 0 through 1023, cover auto-negotiation failure, link training failure, bad signal integrity, remote fault, and speed degradation. Management indications, numbered 1024 through 2047, cover problems such as cable unplugged, bad or unsupported EEPROM, unsupported cable, module thermal shutdown, power budget exceeded, and shorted cable.<\/p>\n

Some InfiniBand platforms provide vendor-specific link-diagnostic codes that can help identify why a port fails to establish a link. For example, NVIDIA platforms expose diagnostic information covering physical-layer and management-related conditions. The exact code ranges and meanings are vendor- and platform-specific, so administrators should consult the documentation for the specific switch or adapter model.<\/p>\n

 <\/p>\n

How to Isolate a Bad Cable or Optic vs. a Bad Port<\/b><\/strong><\/h3>\n

Use this sequence to separate a physical fault from a config problem:<\/p>\n

    \n
  1. 1. Reseat the cable at both ends and check the connector for debris or bent pins.<\/li>\n
  2. 2. Swap in a known-good cable of the same type and recheck the port state.<\/li>\n
  3. 3. Move the suspect link to a known-good port on the same switch. If it comes up, the original port is bad.<\/li>\n
  4. 4. Move a known-good link onto the original port. If that fails too, the port or its configuration is the problem.<\/li>\n<\/ol>\n

    IBM documents this same isolation procedure for its InfiniBand switches, and it remains the fastest way to localize a fault without specialized test gear.<\/p>\n

     <\/p>\n

     <\/p>\n

    Diagnosing Port State: Why Your Link Won’t Come Up<\/strong><\/h2>\n

    In InfiniBand troubleshooting, the single most useful command is `ibstat`. It reports the status, physical state, link-layer information, and speed of each local HCA port. Once you learn to interpret the command’s output, you can resolve most link issues within minutes.<\/p>\n

     <\/p>\n

    The Port State Reference Table<\/strong><\/h3>\n

    Ports operating normally should show an “Active” status, with a physical status of “LinkUp.” Any other status may indicate a potential issue.<\/p>\n

     <\/p>\n\n\n\n\n\n\n\n
    Port State<\/b><\/strong><\/td>\nPhysical State<\/b><\/strong><\/td>\nWhat It Means<\/b><\/strong><\/td>\nWhat to Do<\/b><\/strong><\/td>\n<\/tr>\n
    Active<\/td>\nLinkUp<\/td>\nThe port is operational and has been configured by the Subnet Manager<\/td>\nNo action required<\/td>\n<\/tr>\n
    Down<\/td>\nPolling<\/td>\nThe port has not established a link<\/td>\nCheck cable, optic, port configuration, and physical connectivity<\/td>\n<\/tr>\n
    Init<\/td>\nLinkUp<\/td>\nThe physical link is established, but the port has not completed subnet initialization<\/td>\nCheck Subnet Manager and fabric configuration<\/td>\n<\/tr>\n
    Armed<\/td>\nLinkUp<\/td>\nThe port has completed initialization but has not transitioned to Active<\/td>\nCheck SM configuration, partitioning, and related port state information<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n

     <\/p>\n

    At the low level, these states correspond to numerical values that can be read via the sysfs path `\/sys\/class\/infiniband\/*\/ports\/*\/state`. A value of 4 indicates the “Active” state; 1 indicates “Down,” 2 indicates “Init” (Initializing), and 3 indicates “Armed” (Ready). The corresponding `phys_state` file should show a value of 5, indicating “LinkUp”; a value of 3 indicates the “Polling” state, signifying that the port is attempting to establish a link but has not succeeded.<\/p>\n

     <\/p>\n

    \"InfiniBand<\/p>\n

     <\/p>\n

     <\/p>\n

    The Subnet Manager: InfiniBand’s Single Point of Failure<\/strong><\/h2>\n

    Because the Subnet Manager controls fabric initialization and management, SM availability is critical to InfiniBand fabric operation. A properly configured redundant SM design can reduce the impact of an SM failure.<\/p>\n

    Multiple Subnet Managers can coexist in an InfiniBand fabric, but only one normally operates as the Master SM at a time. SM priorities are used to determine which instance becomes the Master, while standby SMs can provide failover. Troubleshooting should therefore focus on SM election, priority, reachability, and configuration conflicts rather than simply counting the number of running SM processes.<\/p>\n

    SM restarts also cause brief outages. Rebooting a node that hosts the SM can interrupt IPoIB communication for several seconds while a new master takes over. For critical fabrics, run the SM on dedicated management hosts rather than on compute nodes that reboot during jobs.<\/p>\n

    There are two other noteworthy signals: recurring “heavy sweep” entries in `\/var\/log\/opensm.log` indicate physical-layer link flapping or a node stuck in a reboot loop; meanwhile,\u00a0If two ports are Active but applications still cannot communicate, check the P_Key\/partition configuration first, along with routing, MTU, application configuration, and other fabric policies. For OpenSM-based deployments, the partition configuration can be reviewed in the relevant partition configuration file.\u00a0In the default partition settings, `0xffff` represents full membership, while `0x7fff` represents limited membership.<\/p>\n

     <\/p>\n

     <\/p>\n

    Reading Error Counters to Find Marginal Links<\/strong><\/h2>\n

    Not every bad link shows up as a port-down event. Marginal links hide in the error counters, slowly degrading performance until a job finally fails.<\/p>\n

    The most critical counters include `SymbolErrors`, `LinkDowned`, `LinkErrorRecovery`, and `RcvErrors`. A non-zero counter does not necessarily indicate a fault by itself. The more useful signal is whether an error counter increases consistently over time or during a representative workload.\u00a0You can read these counter values using the `perfquery` command or via the sysfs path\u00a0`\/sys\/class\/infiniband\/*\/ports\/*\/counters\/`.<\/p>\n

    There is a subtlety worth knowing. Link Level Retransmission (LLR) on FDR and faster links automatically corrects CRC errors at the link level. This is good for data integrity, but it can mask a degrading physical link by hiding symbol errors. The way to catch it is to monitor the LLR retransmission rate.\u00a0Monitoring link-level retransmissions can also help identify marginal links. A sustained increase in retransmissions, especially when correlated with rising error counters or application performance degradation, should be investigated.<\/p>\n

    A practical workflow: clear the counters, run a representative workload or benchmark, then re-read the counters. A rapidly increasing error counter is a useful indicator that further investigation is required. Depending on the counter, possible causes include physical-layer degradation, link instability, congestion-related events, or link speed\/width issues.\u00a0\u00a0Speaking of width, a common silent failure is a link that trains at 2x width instead of 4x. A 64-GPU NDR cluster running at 38% of expected all-reduce bandwidth was traced to exactly this: a single port negotiated down to half width.<\/p>\n

     <\/p>\n

     <\/p>\n

    The InfiniBand Diagnostic Toolkit<\/strong><\/h2>\n

    These are commands that every InfiniBand engineer should master. They belong to the `infiniband-diags` package.<\/p>\n

    1\u3001ibstat and ibstatus display port status, physical status, and link-layer information.
    \n2\u3001sminfo is used to identify the master SM (Subnet Manager) and its priority.
    \n3\u3001ibnetdiscover and ibnodes are used to map the fabric topology.
    \n4\u3001ibdiagnet is the primary diagnostic tool for fabric health; it discovers the fabric topology, checks links, and flags faulty links as well as instances where error thresholds are exceeded.
    \n5\u3001perfquery is used to read port counters and error counters.
    \n6\u3001ibping uses vendor-specific MADs (Management Datagrams) to verify connectivity between two nodes.
    \n7\u3001ibtracert is used to trace the path between nodes.
    \n8\u3001ibclearerrors and ibclearcounters are used to reset counters before performing diagnostics.<\/p>\n

     <\/p>\n

    Bring these together into a repeatable seven-step sequence:<\/p>\n

      \n
    1. 1. Run `ibstat` to check the status and link layer of each port.<\/li>\n
    2. 2. Run `sminfo` to confirm the presence of an active master SM (Subnet Manager).<\/li>\n
    3. 3. Run `ibnetdiscover` to confirm that the topology matches your design.<\/li>\n
    4. 4. Use `ibclearcounters` to clear the counters.<\/li>\n
    5. 5. Generate traffic through actual benchmarking or MPI jobs.<\/li>\n
    6. 6. Use `perfquery` to re-check the counters and observe whether the values \u200b\u200bincrease.<\/li>\n
    7. 7. Run `ibdiagnet` to detect any rate or link-width mismatches across the network fabric.<\/li>\n<\/ol>\n

       <\/p>\n

      \"7-Step<\/p>\n

       <\/p>\n

       <\/p>\n

      Third-Party Transceiver and Cable Compatibility<\/strong><\/h2>\n

      One question comes up in nearly every InfiniBand deployment: will these third-party optics work in my switch and adapters? It is the highest-stakes compatibility question in the ecosystem, and the answer is more nuanced than “buy the vendor’s cables.”<\/p>\n

      InfiniBand and Ethernet share the same QSFP<\/u><\/a>, QSFP-DD<\/u><\/a>, and OSFP<\/u><\/a>\u00a0form factors, but the two are not fully interchangeable. A QSFP28<\/u><\/a>\u00a0module validated for Ethernet may not work in an InfiniBand port, and vice versa, because of minor signaling and EEPROM differences. QSFP56<\/u><\/a>\u00a0for HDR is electrically related to QSFP28 but not backward compatible at the signal level. OSFP is not mechanically compatible with QSFP at all, so an OSFP module will not even seat in a QSFP cage.<\/p>\n

       <\/p>\n

       <\/p>\n

      Conclusion<\/strong><\/h2>\n

      InfiniBand troubleshooting rewards a disciplined, bottom-up approach. Most fabrics fail at the physical layer first, in a cable, optic, or connector, long before the subnet manager or the drivers deserve blame. Start there, and you will save hours of misdirected effort.<\/p>\n

      When addressing the next service interruption, keep the following three points in mind: check the `ibstat` output and port status table before modifying configurations; verify that only one master Subnet Manager is running on the system; and treat rising error counts as an early warning sign of poor link performance, rather than merely as insignificant noise.<\/p>\n

      When the problem does turn out to be a cable or transceiver, the fix should be quick and painless. Choose modules that are verified for your exact switch and adapter, and check compatibility before you buy rather than after a fabric fails to come up.<\/p>\n

       <\/p>\n

       <\/p>\n

      FAQ<\/strong><\/h2>\n

      1. What is the first step in InfiniBand troubleshooting?<\/b><\/strong><\/h3>\n

      Start with the physical layer. Check the cable, transceiver, connector, and port status before investigating software or configuration issues.<\/p>\n

      2. Which command is commonly used to check InfiniBand port status?<\/b><\/strong><\/h3>\n

      ibstat\u00a0is one of the most useful commands. It shows the port state, physical state, link layer, and supported link speed.<\/p>\n

      3. What does an InfiniBand port in the Init state mean?<\/b><\/strong><\/h3>\n

      The physical link is established, but the port has not completed subnet initialization. Check the Subnet Manager and fabric configuration.<\/p>\n

      4. How can I tell if an InfiniBand cable or transceiver is faulty?<\/b><\/strong><\/h3>\n

      Replace the suspected cable or optic with a known-good one and test the link again. Moving the link to another known-good port can also help isolate the fault.<\/p>\n

      5. Why is my InfiniBand port Active but communication still fails?<\/b><\/strong><\/h3>\n

      Check P_Key\/partition settings, routing, MTU, and application configuration. An Active port does not always guarantee end-to-end application connectivity.<\/p>\n

      6. Which InfiniBand counters should I monitor?<\/b><\/strong><\/h3>\n

      Commonly useful counters include SymbolErrors, LinkDowned, LinkErrorRecovery, and RcvErrors. Pay particular attention to counters that increase during traffic.<\/p>\n

      7. How do I check the health of an InfiniBand fabric?<\/b><\/strong><\/h3>\n

      Tools such as ibdiagnet, ibnetdiscover, iblinkinfo, and perfquery\u00a0can help check topology, link status, errors, and port counters.<\/p>\n

      8. Are third-party optical transceivers compatible with InfiniBand?<\/b><\/strong><\/h3>\n

      Not necessarily. Physical form-factor compatibility does not guarantee interoperability. Always verify the module’s data rate, reach, optical specifications, and compatibility with the specific InfiniBand switch or adapter.<\/p>\n

       <\/p>\n

       <\/p>\n