Fixing "Cable Data Invalid EEPROM" Error on NVIDIA QM9700 InfiniBand Switch

Issue

On NVIDIA QM9700 InfiniBand switches, some ports may appear down and show an error such as:

This issue is often caused by outdated CPLD firmware and can be resolved by updating the CPLD version on the switch.

Root Cause

The EEPROM error is typically due to compatibility or communication issues between the switch and the optical cables/transceivers. A CPLD firmware update resolves low-level hardware interface bugs related to EEPROM communication.

Solution: Update CPLD Firmware

NVIDIA Article: https://enterprise-support.nvidia.com/s/article/updateswitchcpld-for-Quantum1-Quantum2-Spectrum2-and-Spectrum3-based-Systems

Requirements

      ·       Linux server with SSH access to the switch

      ·       updateswitchcpld_v3.1.tgz CPLD update tool

      ·       Switch IP (management or IPoIB)

      ·       Admin username and password

Step-by-Step Procedure

1. Transfer and Extract the Update Tool

Copy updateswitchcpld_v3.1.tgz to your Linux host having connectivity to ib switch and extract it:

tar xzf updateswitchcpld_v3.1.tgz cd updateswitchcpld_v3.1 

2. Run the CPLD Update Tool

Use the following command to start the CPLD update process:

./updateswitchcpld --managed -t <SWITCH_IP> -u <ADMIN_USERNAME> -p <PASSWORD> --os mlnx-os

Example:

./updateswitchcpld --managed -t 10.152.15.55 -u admin -p <password> --os mlnx-os 

·       The switch will reboot during the CPLD update.

·       Save configuration before running this step.

3. OS Version Compatibility Note

There is a known compatibility issue with MLNX-OS version 3.11.4002 when running the CPLD update script. If you are using this version:

·       Upgrade temporarily to a compatible version like 3.12.1000 or 3.11.4000

·       Run the CPLD update

·       Revert back if needed

Switch OS Upgrade/Downgrade Commands

show images image boot next <image_name> configuration write reload 

4. Confirm CPLD Upgrade

After reboot, validate the CPLD version:updateswitchcpld --managed -t <SWITCH_IP> -u <ADMIN_USERNAME> -p <PASSWORD> --os mlnx-os --verbose --check_cpld 

Post-Update

·       Ensure ports previously showing “EEPROM” errors are now active.

·       Validate the cables using ibdiagnet or the switch web UI/CLI.

• Related Articles

• How to Fetch Transceiver Details of All Ports in an InfiniBand QM9700 Switch

  Overview On NVIDIA/Mellanox QM9700 InfiniBand switches, you may need to view the transceiver details (e.g., vendor, part number, and serial number) for connected optical modules or cables. This can be useful for troubleshooting, inventory tracking, ...

• Mapping mlx5_x Names to Actual InfiniBand Ports

  Purpose In large GPU/AI clusters, each compute node often has multiple InfiniBand (IB) Host Channel Adapters (HCAs). On Linux, these HCAs appear as mlx5_x devices. To troubleshoot connectivity or performance issues, it is essential to map: mlx5_x ...

• Viewing and Backing Up Running Configuration from QM9700 IB Switch Web GUI

  Purpose This guide outlines the steps to view and back up the running configuration of the QM9700 InfiniBand switch using its Web GUI interface. Prerequisites · Access to the QM9700 IB switch Web GUI (IP address or hostname) · User credentials with ...

• Using dmesg and Kernel Module Checks to Troubleshoot NVIDIA GPU Issues

  Overview This article outlines how to use dmesg logs and kernel module commands to diagnose issues where the operating system fails to detect NVIDIA GPUs—even though they appear in the system's BMC (Baseboard Management Controller). When to Use This ...

• How to Change Default IB Interface in NVIDIA UFM (e.g., from ib0 to ib1)

  By default, NVIDIA UFM binds to the ib0 InfiniBand interface during startup. If ib0 is down or unavailable, UFM will fail to start. In such cases, you may need to change the default interface to ib1 (or any other available IB interface). This article ...