Network Speed IB Alert - Troubleshooting Guide

KB 021273 - Network Speed IB Alert - Troubleshooting Guide

Purpose

This document provides a standardized approach for handling and troubleshooting the Network Speed – IB alert, including L1 and L2 actions, escalation process, and resolution guidelines.

Alert Name

Network Speed IB

Alert Description

This alert is triggered when the InfiniBand (IB) link operates below the expected bandwidth. It indicates a mismatch or degradation in IB link speed on the head node, which can impact high-performance computing (HPC) environments.

Severity

P2 – High Impact

Possible Causes

  1. InfiniBand link speed mismatch
  2. Faulty or incompatible IB cables
  3. Defective SFP/transceiver modules
  4. Switch port configuration issues
  5. Hardware limitations or degradation
  6. Issues in IB fabric health

L1 Engineer Actions

Step 1 – Log the Alert

  1. Log in to the affected node
  2. Review alert summary and details

Step 2 – Create Ticket

  1. Monitor the alert for 20 minutes for auto-resolution
  2. If issue persists:
    1. Create an internal ticket
    2. Include observations and alert details

Step 3 – Escalation (If required)

  1. Inform L2 team after 20 minutes if unresolved
  2. Share all relevant findings

Step 4 – Documentation

After confirmation with L2, update Issue Master Sheet/ Issue Tracker Sheet.

L2 Engineer Actions

Step 1 – Verify InfiniBand Link Speed

ibstat

  1. Check current IB link speed
  2. Compare with expected bandwidth

Step 2 – Check IB Fabric Health

  1. Validate overall InfiniBand fabric status
  2. Identify any inconsistencies or degraded links

Step 3 – Inspect Transceivers (SFPs)

  1. Check on both- Server side and Switch side
  2. Ensure proper seating and compatibility

Step 4 – Switch Side Validation (If Applicable)

  1. Perform port flapping- Shutdown/ No Shutdown
  2. Validate switch configuration for IB ports

Step 5 – Hardware Validation

  1. Reseat or replace faulty components:
    1. IB Cable
    2. SFP/ Transceiver

Step 6 – OEM Escalation (If Required)

  1. If issue persists after troubleshooting:
    1. Raise a ticket with OEM/vendor
    2. Share observations and test results

Resolution

The issue is considered resolved when:
  1. IB link speed matches expected hardware specifications
  2. No degradation observed in IB performance
  3. Cluster communication latency is normal
  4. Alert is cleared

Escalation

  • L1 → L2: If not resolved within 20 minutes
  • L2 → OEM/Vendor: If hardware or configuration issue persists