How to check GPU detection across compute nodes using bash script

How to check GPU detection across compute nodes using bash script

1. Purpose

This script automates GPU detection validation across all compute nodes in a GPU cluster.

It verifies:

  • Nodes with missing GPUs
  • Nodes that are unreachable
  • Timestamp of execution
  • Summary report generation

This eliminates the need to manually SSH into each node and run nvidia-smi.


2. When to Use

Use this script when:

  • Performing routine GPU health checks
  • Validating cluster readiness before workload scheduling
  • Troubleshooting GPU-related issues
  • Verifying hardware after maintenance or reboot
  • Investigating driver failures


3. What This Script Exactly Collects

The script gathers the following information:

  • Total nodes checked
  • GPU count per node (configurable)
  • Nodes where:
    • GPUs are missing
    • NVIDIA-SMI fails
    • SSH connection fails
  • Date and time of execution
  • Timestamped log file

Requirements

  • Linux-based login node
  • Passwordless SSH access to compute nodes
  • nvidia-smi installed on compute nodes
  • Bash shell environment

Log files are stored under:

~/gpu_logs/

4. Script

🔧 Modify the variables at the top to match your environment.
Quote

#!/bin/bash

# =========================

# CONFIGURATION VARIABLES

# =========================

START=1                  # Starting node index

END=10                   # Ending node index

BASE_HOST="compute"      # Base hostname prefix (example: compute01, compute02)

EXPECTED_GPUS=8          # Expected GPUs per node

LOGDIR="$HOME/gpu_logs"

# =========================

mkdir -p "$LOGDIR"

LOGFILE="$LOGDIR/gpu_detection_$(date +%F_%H%M).log"

TMPFILE=$(mktemp)

{

  echo "=========================================="

  echo " GPU Detection Report"

  echo " Date: $(date +%F)"

  echo " Time: $(date +%T)"

  echo "=========================================="

  echo

} > "$LOGFILE"

MISSING_NODES=0

TOTAL_NODES=0

CMD="nvidia-smi -L"

for i in $(seq $START $END); do

    NODE_NAME=$(printf "%s%02d" "$BASE_HOST" "$i")

    ((TOTAL_NODES++))

    echo "===================== $NODE_NAME ====================="

    OUTPUT=$(ssh -o ConnectTimeout=5 "$NODE_NAME" "$CMD" 2>&1)

    SSH_STATUS=$?

    if [ $SSH_STATUS -eq 255 ]; then

        echo "❌ Node Unreachable: $NODE_NAME"

        printf "%-15s | %-20s\n" "$NODE_NAME" "Unreachable" >> "$TMPFILE"

        ((MISSING_NODES++))

        echo

        continue

    fi

    if [[ -z "$OUTPUT" || "$OUTPUT" == *"NVIDIA-SMI has failed"* ]]; then

        echo "⚠️  NVIDIA-SMI failed on $NODE_NAME — please check"

        printf "%-15s | %-20s\n" "$NODE_NAME" "NVIDIA-SMI Failed" >> "$TMPFILE"

        ((MISSING_NODES++))

        echo

        continue

    fi

    DETECTED_INDICES=$(echo "$OUTPUT" | grep -oP 'GPU \K[0-9]+')

    DETECTED_COUNT=$(echo "$DETECTED_INDICES" | wc -l)

    if [ "$DETECTED_COUNT" -lt "$EXPECTED_GPUS" ]; then

        echo "⚠️  GPU(s) missing on $NODE_NAME — please check"

        printf "%-15s | %-20s\n" "$NODE_NAME" "GPU(s) Missing" >> "$TMPFILE"

        ((MISSING_NODES++))

    else

        echo "✅ All $EXPECTED_GPUS GPUs detected on $NODE_NAME"

    fi

    echo

done

{

  echo "Nodes Checked: $TOTAL_NODES"

  echo "Nodes with Issues: $MISSING_NODES"

  echo

  echo "---------------------------------------------"

  echo "Node Name       | Status"

  echo "---------------------------------------------"

  cat "$TMPFILE"

  echo "---------------------------------------------"

  echo

  if [ "$MISSING_NODES" -eq 0 ]; then

      echo "✅ All nodes have all $EXPECTED_GPUS GPUs detected."

  else

      echo "⚠️  Some nodes require attention."

  fi

  echo

  echo "================================================="

  echo "Log generated on: $(date)"

  echo "================================================="

} >> "$LOGFILE"

rm -f "$TMPFILE"

echo

echo "================================================="

echo "Summary:"

echo "  Nodes with issues: $MISSING_NODES"

echo "Log saved at: $LOGFILE"

echo "================================================="


5. Procedure

Step 1 – Create Script

mkdir -p ~/gpu_logs
nano ~/gpu_logs/gpu_missing_c​​​​​heck.sh

Step 2 – Make Executable

chmod +x ~/gpu_logs/gpu_missing_check.sh

Step 3 – Execute

~/gpu_logs/gpu_missing_check.sh


6. Output

Console Output Example

===================== compute01 =====================

✅ All 8 GPUs detected on compute01

===================== compute06 =====================

⚠️  GPU(s) missing on compute06 — please check

Log File Location

Each run generates:

~/gpu_logs/gpu_detection_YYYY-MM-DD_HHMM.log

Sample Log Summary

Nodes Checked: 10

Nodes with Issues: 2

 

Node Name       | Status

---------------------------------------------

compute06       | GPU(s) Missing

compute08       | Unreachable

---------------------------------------------

7. Scope

Run on:

  • Login or management node
  • Any Linux-based GPU cluster
  • Environments with:
    • Passwordless SSH enabled
    • NVIDIA drivers installed
    • nvidia-smi available

    • Related Articles

    • GPU vBIOS Verification Across HGX Nodes

      1. Purpose This document provides a procedure to verify GPU vBIOS versions across all HGX nodes in the cluster using an automated script. The script connects to each node from the login node using passwordless SSH and retrieves the GPU vBIOS version ...

    • Troubleshooting steps if GPU is not detecting

      Steps to check if GPU is not detecting on the system Parameters for demonstrating the Troubleshooting System Model : ASRock TRX40 Creator AMD sTRX4 Socket ATX GPU : PNY A6000 x 4 Verify the GPUs on the system using below steps Execute the below ...

    • How to collect diagnostic logs using the NetApp Log Collection Script

      1. Purpose This document describes the procedure to collect diagnostic logs using the NetApp Log Collection Script in environments running: BeeGFS NetApp E-Series backend storage HA cluster using Pacemaker and Corosync This script is typically ...

    • Execution of NVIDIA Field Diagnostic (FD) Tool and Collection of Diagnostic Logs, and troubleshooting of GPU(s) not detected

      1. Objective To provide a standardized procedure for executing the NVIDIA Field Diagnostic (FD) tool, verifying GPU status, collecting the required diagnostic logs, and documenting the findings for further analysis. 2. Scope This procedure applies to ...

    • GPU Detection Checks in linux

      GPU Detection Checks Software checks Open terminal and type "nvdia-smi" , "lspci | grep -i vga "to list the detected GPU details Find out information's about GPU by using "sudo lshw -C display" command Update the Nvidia GPU driver using "sudo apt ...

    • Popular Articles

    • CP Plus Camera and NVR Configuration

      NVR Configuration The CP Plus Pro Series of NVRs have been meticulously designed for providing you with upgraded performance and higher recording quality in your IP video surveillance solution. The robust processor that has been inculcated in this ...

    • Kerberos Authentication – Overview

      What is Kerberos? Kerberos is a secure authentication method used in our Active Directory (AD) environment (mbuzztech.com). It allows users to: Access multiple systems without re-entering passwords (Single Sign-On – SSO) Log in once Where We Use It ...

    • How to Remove and Reinstall NVIDIA Drivers on Ubuntu

      This article provides step by step guide to completely remove existing NVIDIA drivers and reinstall specific version of the NVIDIA driver on the Ubuntu system Prerequisites Administrative (sudo) access to the Ubuntu system. Internet access to ...

    • Personal Computers and Servers - Classification and Point of Contact

      We can classify the computers that MBUZZ handles based on their form-factor as below: Tower Workstations, Desktops, Gaming PCs and SFF (Small form factor) PCs fall under this category. These are computers people would use on a desk and rarely move. ...

    • M.2 SSD Tier List

      The sequential read and write speeds, which are usually the most advertised number, are not a proper benchmark of real-world performance or the quality of an SSD. This article categorizes and tiers SSDs based on factors like the type of NAND flash, ...