How to Change Default IB Interface in NVIDIA UFM (e.g., from ib0 to ib1)

How to Change Default IB Interface in NVIDIA UFM (e.g., from ib0 to ib1)

By default, NVIDIA UFM binds to the ib0 InfiniBand interface during startup. If ib0 is down or unavailable, UFM will fail to start. In such cases, you may need to change the default interface to ib1 (or any other available IB interface).

This article guides you through the steps to change the default IB interface by editing the gv.cfg configuration file.

Location of Configuration File

The IB interface UFM uses is defined in the file:

  1. /opt/ufm/conf/gv.cfg

Prerequisites

  • Root or sudo access to the server.
  • UFM must be stopped before making changes.

  1. Verify that ib1 is up and operational:


    ip link show ib1

Steps to Change IB Interface

  1. Stop UFM Service:
  2. sudo systemctl stop ufm

  3. Edit gv.cfg Configuration File:
    Open the file in a text editor:

    sudo nano /opt/ufm/conf/gv.cfg

  4. Locate the Following Line:

    ib_interface=ib0

  5. Change ib0 to ib1:
    Modify the line as follows:

    ib_interface=ib1

  6. Save and Exit the editor (For nano, press Ctrl + O, Enter, then Ctrl + X).

  7. Start UFM Service:

    sudo systemctl start ufm

Verification

To ensure UFM is using the new interface:
  1. Check UFM status:
  2. sudo systemctl status ufm

  3. Review UFM logs for interface bindings:

    sudo tail -f /opt/ufm/files/logs/ibdiagnet.log

  4. Confirm the IB interface is active and connected:

    ibstat ib1

Troubleshooting

  • If ib1 is also down, UFM will not start. Ensure the interface is physically connected and active.
  • Double-check that you edited the correct line in gv.cfg.

  • Run ibdiagnet manually to test interface functionality.


 Notes

  • Changing the IB interface does not persist across interface renaming or re-indexing (e.g., after firmware or OS updates). Always verify the IB device name.

  • It's recommended to maintain consistent naming using udev rules if your environment has multiple IB interfaces.



    • Related Articles

    • Managing OpenSM within NVIDIA UFM and getting OpenSM Logs from UFM

      Overview In NVIDIA UFM (Unified Fabric Manager), OpenSM — the InfiniBand Subnet Manager — is managed internally by UFM rather than as a standalone systemd service. This allows UFM to control the IB fabric, collect telemetry, and maintain HA ...

    • Fix: UFM Service Not Starting on UFM Appliance UFM-MUA9502H

      Note: This appliance UFM-MUA9502H is EOL as of 2022. Pre-requisites Ensure the following conditions are met before attempting to start the UFM service: Ensure the license is attached. Verify bond0 is configured with eth0 and eth1. Command: show ...

    • How to Fetch Transceiver Details of All Ports in an InfiniBand QM9700 Switch

      Overview On NVIDIA/Mellanox QM9700 InfiniBand switches, you may need to view the transceiver details (e.g., vendor, part number, and serial number) for connected optical modules or cables. This can be useful for troubleshooting, inventory tracking, ...

    • Mapping mlx5_x Names to Actual InfiniBand Ports

      Purpose In large GPU/AI clusters, each compute node often has multiple InfiniBand (IB) Host Channel Adapters (HCAs). On Linux, these HCAs appear as mlx5_x devices. To troubleshoot connectivity or performance issues, it is essential to map: mlx5_x ...

    • Updating Port Description on Mellanox Switches like QM9700

      Objective To update the description for multiple InfiniBand (IB) ports on a Mellanox QM9700 series switch using the web interface. Prerequisites Administrative access to the Mellanox QM9700 switch's web interface. A pre-prepared list of commands with ...