1. Purpose

This document provides a procedure to verify GPU vBIOS versions across all HGX nodes in the cluster using an automated script.

The script connects to each node from the login node using passwordless SSH and retrieves the GPU vBIOS version using nvidia-smi.
It then compares the retrieved version with the expected target vBIOS and categorizes nodes accordingly.

This helps identify nodes that:

• Have already been updated

• Still have the older vBIOS

• Are unreachable

2. Scope

This procedure applies to:

• HGX cluster nodes

• Nodes accessible via passwordless SSH

• Systems running NVIDIA GPUs with nvidia-smi available

3. Environment Details

4. vBIOS Version Reference

5. Prerequisites

Before executing the script ensure the following:

1. Passwordless SSH access is configured

   ssh <node_ip>
   

2. NVIDIA drivers are installed

   nvidia-smi
   

3. The command below works on the nodes:

      nvidia-smi -q | grep -i "VBIOS Version"

6. Script for Cluster-wide vBIOS Verification

Create a script named: check_vbios_versions.sh

check_vbios_version.sh
#!/bin/bash

START=101
END=155
BASE_IP="10.152.241"

CURRENT="96.00.A5.00.01"
TARGET="96.00.D0.00.02"

printf "%-18s %-20s %-15s\n" "NODE IP" "VBIOS VERSION" "STATUS"
printf "%-18s %-20s %-15s\n" "-------" "-------------" "------"

for i in $(seq $START $END)
do
    NODE="$BASE_IP.$i"

    VBIOS=$(ssh -o ConnectTimeout=3 $NODE "nvidia-smi -q | grep -i 'VBIOS Version' | head -n1 | awk -F': ' '{print \$2}'" 2>/dev/null)

    if [ -z "$VBIOS" ]; then
        STATUS="UNREACHABLE"
        VBIOS="N/A"

    elif [ "$VBIOS" == "$TARGET" ]; then
        STATUS="UPDATED"

    elif [ "$VBIOS" == "$CURRENT" ]; then
        STATUS="OLD"

    else
        STATUS="UNKNOWN"
    fi

    printf "%-18s %-20s %-15s\n" "$NODE" "$VBIOS" "$STATUS"

done
7. Execution Steps

Step 1: Create the Script

vi check_vbios_versions.sh

Paste the script content and save the file.

Step 2: Grant Execute Permission

chmod +x check_vbios_versions.sh

Step 3: Execute the Script

./check_vbios_versions.sh

8. Sample Output

9. Status Definitions

10. Validation

To manually verify a node:

ssh <node_ip>

Then run:

nvidia-smi -q | grep -i vbios

Expected output example:

VBIOS Version: 96.00.D0.00.02