How to Perform an Automated InfiniBand Link Health Check on a Linux Cluster

How to Perform an Automated InfiniBand Link Health Check on a Linux Cluster

1. Purpose

This script automates InfiniBand link health verification across compute nodes in a GPU or HPC cluster.

It connects to each node via SSH, runs ibstat, and summarizes InfiniBand port health by identifying:

  • Unreachable nodes
  • Down links
  • Degraded links (operating below expected bandwidth)
  • Lowest detected link rate

The script generates a structured, timestamped report for operational visibility.

2. When to Use

Use this script when:

  • Performing routine cluster health checks
  • Validating cluster network readiness before job scheduling
  • Troubleshooting RDMA or InfiniBand issues
  • After hardware replacement or maintenance
  • Investigating performance degradation
  • Validating link speeds after switch upgrades

3. What This Script Collects

For each node, the script collects:

  • Total InfiniBand ports detected
  • Number of Active ports
  • Number of Down ports
  • Number of Degraded ports (below expected rate)
  • Lowest detected link rate
  • Unreachable node count
  • Timestamped log file

3.1. Requirements

  • Linux-based login/management node
  • Passwordless SSH access to compute nodes
  • ibstat installed on compute nodes
  • Bash shell environment

Log files are stored under:

$HOME/gpu_logs/ib_health/

4. Script

🔧 Modify the configuration variables at the top to match your environment.

Quote

#!/bin/bash

# =========================

# CONFIGURATION VARIABLES

# =========================

START=1                   # Starting node index

END=10                    # Ending node index

BASE_HOST="compute"       # Hostname prefix (example: compute01, compute02)

EXPECTED_RATE=400         # Expected IB rate (Gb/s)

LOGDIR="$HOME/gpu_logs/ib_health"

CMD="ibstat"

# =========================

mkdir -p "$LOGDIR"

LOGFILE="$LOGDIR/ib_health_$(date +%F_%H%M).log"

TMPFILE=$(mktemp)

{

echo "========================================================"

echo " InfiniBand Health Report"

echo " Date: $(date '+%F  %T')"

echo "========================================================"

echo

} > "$LOGFILE"

MISSING_NODES=0

TOTAL_NODES=0

DOWNLINK_NODES=0

echo ""

echo "Checking InfiniBand Health..."

echo "This may take a few minutes depending on node count."

echo ""

for i in $(seq $START $END); do

NODE_NAME=$(printf "%s%02d" "$BASE_HOST" "$i")

((TOTAL_NODES++))

echo "===================== Checking $NODE_NAME ====================="

OUTPUT=$(ssh -o ConnectTimeout=5 "$NODE_NAME" "$CMD" 2>&1)

SSH_STATUS=$?

if [ $SSH_STATUS -eq 255 ]; then

printf "%-15s | %-12s\n" "$NODE_NAME" "Unreachable" >> "$TMPFILE"

((MISSING_NODES++))

continue

fi

TOTAL_PORTS=$(echo "$OUTPUT" | grep -c "CA 'mlx")

ACTIVE_PORTS=$(echo "$OUTPUT" | grep -c "State: Active")

DOWN_PORTS=$(echo "$OUTPUT" | grep -c "State: Down")

DEGRADED_PORTS=$(echo "$OUTPUT" | grep -A1 "State: Active" | \

grep -o "Rate: [0-9]*" | \

awk -v rate=$EXPECTED_RATE '$2 < rate' | wc -l)

LOWEST_RATE=$(echo "$OUTPUT" | grep -o "Rate: [0-9]*" | \

awk '{print $2}' | sort -n | head -1)

[ -z "$LOWEST_RATE" ] && LOWEST_RATE="N/A"

if [ "$ACTIVE_PORTS" -eq "$TOTAL_PORTS" ] && \

[ "$DOWN_PORTS" -eq 0 ] && \

[ "$DEGRADED_PORTS" -eq 0 ]; then

echo " ✅ $NODE_NAME – all links healthy"

echo ""

continue

fi

if [ "$DOWN_PORTS" -gt 0 ] || [ "$DEGRADED_PORTS" -gt 0 ]; then

echo " ❌ Issue detected on $NODE_NAME – manual check required"

echo ""

((DOWNLINK_NODES++))

fi

printf "%-15s | %-11s | %-6s | %-4s | %-8s | %-6s\n" \

"$NODE_NAME" "$TOTAL_PORTS" "$ACTIVE_PORTS" \

"$DOWN_PORTS" "$DEGRADED_PORTS" "$LOWEST_RATE" >> "$TMPFILE"

done

{

echo "Node Name       | Total Ports | Active | Down | Degraded | Lowest Rate"

echo "-----------------------------------------------------------------------"

cat "$TMPFILE"

echo "-----------------------------------------------------------------------"

echo

echo "Nodes Checked: $TOTAL_NODES"

echo "Nodes Unreachable: $MISSING_NODES"

echo "Nodes with Down/Degraded Links: $DOWNLINK_NODES"

echo "========================================================"

echo "Log generated on: $(date)"

echo "========================================================"

} >> "$LOGFILE"

rm -f "$TMPFILE"

echo

echo "========================================================"

echo "Summary:"

echo "  IB Health Check Completed for $TOTAL_NODES Nodes"

echo "  Nodes with issues: $DOWNLINK_NODES"

echo "  Log saved at: $LOGFILE"

echo "========================================================"

5. Procedure

Step 1 – Create Script

mkdir -p ~/gpu_logs/ib_health

nano ~/gpu_logs/ib_health_check.sh

 

Step 2 – Make Executable

chmod +x ~/gpu_logs/ib_health_check.sh

 

Step 3 – Execute

./gpu_logs/ib_health_check.sh


6. Output

Console Output Example

Checking InfiniBand Health...

This may take a few minutes depending on node count.

===================== Checking compute01 =====================

 ✅ compute01 – all links healthy

===================== Checking compute02 =====================

 ❌ Issue detected on compute02 – manual check required

========================================================

Summary:

  IB Health Check Completed for 10 Nodes

  Nodes with issues: 1

  Log saved at: /home/user/gpu_logs/ib_health/ib_health_2026-02-26_0910.log

========================================================

Log File Output Example

InfiniBand Health Report

Date: 2026-02-26 09:10:00

Node Name       | Total Ports | Active | Down | Degraded | Lowest Rate

-----------------------------------------------------------------------

compute01       | 2           | 2      | 0    | 0        | 400

compute02       | 2           | 1      | 1    | 0        | 400

-----------------------------------------------------------------------

Nodes Checked: 10

Nodes Unreachable: 0

Nodes with Down/Degraded Links: 1


7. Scope

Run on:

  • Login or management node
  • Any Linux-based GPU/HPC cluster
  • Environments with:
    • Passwordless SSH enabled
    • InfiniBand installed and configured
    • ibstat available on compute nodes

    • Related Articles

    • Mapping mlx5_x Names to Actual InfiniBand Ports

      Purpose In large GPU/AI clusters, each compute node often has multiple InfiniBand (IB) Host Channel Adapters (HCAs). On Linux, these HCAs appear as mlx5_x devices. To troubleshoot connectivity or performance issues, it is essential to map: mlx5_x ...

    • Network Down IB Alert - Troubleshooting Guide

      Purpose This document provides a standardized approach for handling and troubleshooting the Network Down – IB alert, including L1 and L2 actions, escalation process, and resolution criteria. Alert Name Network Down – IB Alert Description This alert ...

    • Network Speed IB Alert - Troubleshooting Guide

      Purpose This document provides a standardized approach for handling and troubleshooting the Network Speed – IB alert, including L1 and L2 actions, escalation process, and resolution guidelines. Alert Name Network Speed IB Alert Description This alert ...

    • What is Boot Process in Linux OS.

      Boot Process in Linux OS. Have you ever wondered what happens behind the scenes from the time you press the power button until the Linux login prompt appears? Press the power button on your system, and after few moments you see the Linux login ...

    • How to Collect InfiniBand Transceiver Temperature (Non-Root Method)

      1. Purpose This document explains how to collect InfiniBand (IB) transceiver (QSFP module) temperature from servers using the Linux sysfs hwmon interface without requiring root privileges. This method applies to systems using Mellanox / NVIDIA ...

    • Popular Articles

    • CP Plus Camera and NVR Configuration

      NVR Configuration The CP Plus Pro Series of NVRs have been meticulously designed for providing you with upgraded performance and higher recording quality in your IP video surveillance solution. The robust processor that has been inculcated in this ...

    • Kerberos Authentication – Overview

      What is Kerberos? Kerberos is a secure authentication method used in our Active Directory (AD) environment (mbuzztech.com). It allows users to: Access multiple systems without re-entering passwords (Single Sign-On – SSO) Log in once Where We Use It ...

    • How to Remove and Reinstall NVIDIA Drivers on Ubuntu

      This article provides step by step guide to completely remove existing NVIDIA drivers and reinstall specific version of the NVIDIA driver on the Ubuntu system Prerequisites Administrative (sudo) access to the Ubuntu system. Internet access to ...

    • Personal Computers and Servers - Classification and Point of Contact

      We can classify the computers that MBUZZ handles based on their form-factor as below: Tower Workstations, Desktops, Gaming PCs and SFF (Small form factor) PCs fall under this category. These are computers people would use on a desk and rarely move. ...

    • M.2 SSD Tier List

      The sequential read and write speeds, which are usually the most advertised number, are not a proper benchmark of real-world performance or the quality of an SSD. This article categorizes and tiers SSDs based on factors like the type of NAND flash, ...