Fixing "Cable Data Invalid EEPROM" Error on NVIDIA QM9700 InfiniBand Switch

Issue

On NVIDIA QM9700 InfiniBand switches, some ports may appear down and show an error such as:

This issue is often caused by outdated CPLD firmware and can be resolved by updating the CPLD version on the switch.

Root Cause

The EEPROM error is typically due to compatibility or communication issues between the switch and the optical cables/transceivers. A CPLD firmware update resolves low-level hardware interface bugs related to EEPROM communication.

Solution: Update CPLD Firmware

NVIDIA Article: https://enterprise-support.nvidia.com/s/article/updateswitchcpld-for-Quantum1-Quantum2-Spectrum2-and-Spectrum3-based-Systems

Requirements

      ·       Linux server with SSH access to the switch

      ·       updateswitchcpld_v3.1.tgz CPLD update tool

      ·       Switch IP (management or IPoIB)

      ·       Admin username and password

Step-by-Step Procedure

1. Transfer and Extract the Update Tool

Copy updateswitchcpld_v3.1.tgz to your Linux host having connectivity to ib switch and extract it:

tar xzf updateswitchcpld_v3.1.tgz cd updateswitchcpld_v3.1 

2. Run the CPLD Update Tool

Use the following command to start the CPLD update process:

./updateswitchcpld --managed -t <SWITCH_IP> -u <ADMIN_USERNAME> -p <PASSWORD> --os mlnx-os

Example:

./updateswitchcpld --managed -t 10.152.15.55 -u admin -p <password> --os mlnx-os 

·       The switch will reboot during the CPLD update.

·       Save configuration before running this step.

3. OS Version Compatibility Note

There is a known compatibility issue with MLNX-OS version 3.11.4002 when running the CPLD update script. If you are using this version:

·       Upgrade temporarily to a compatible version like 3.12.1000 or 3.11.4000

·       Run the CPLD update

·       Revert back if needed

Switch OS Upgrade/Downgrade Commands

show images image boot next <image_name> configuration write reload 

4. Confirm CPLD Upgrade

After reboot, validate the CPLD version:updateswitchcpld --managed -t <SWITCH_IP> -u <ADMIN_USERNAME> -p <PASSWORD> --os mlnx-os --verbose --check_cpld 

Post-Update

·       Ensure ports previously showing “EEPROM” errors are now active.

·       Validate the cables using ibdiagnet or the switch web UI/CLI.

• Related Articles

• Viewing and Backing Up Running Configuration from QM9700 IB Switch Web GUI

  Purpose This guide outlines the steps to view and back up the running configuration of the QM9700 InfiniBand switch using its Web GUI interface. Prerequisites · Access to the QM9700 IB switch Web GUI (IP address or hostname) · User credentials with ...

• How to Change Default IB Interface in NVIDIA UFM (e.g., from ib0 to ib1)

  By default, NVIDIA UFM binds to the ib0 InfiniBand interface during startup. If ib0 is down or unavailable, UFM will fail to start. In such cases, you may need to change the default interface to ib1 (or any other available IB interface). This article ...

• Using dmesg and Kernel Module Checks to Troubleshoot NVIDIA GPU Issues

  Overview This article outlines how to use dmesg logs and kernel module commands to diagnose issues where the operating system fails to detect NVIDIA GPUs—even though they appear in the system's BMC (Baseboard Management Controller). When to Use This ...

• Managing OpenSM within NVIDIA UFM and getting OpenSM Logs from UFM

  Overview In NVIDIA UFM (Unified Fabric Manager), OpenSM — the InfiniBand Subnet Manager — is managed internally by UFM rather than as a standalone systemd service. This allows UFM to control the IB fabric, collect telemetry, and maintain HA ...

• Updating Port Description on Mellanox Switches like QM9700

  Objective To update the description for multiple InfiniBand (IB) ports on a Mellanox QM9700 series switch using the web interface. Prerequisites Administrative access to the Mellanox QM9700 switch's web interface. A pre-prepared list of commands with ...