Purpose

In large GPU/AI clusters, each compute node often has multiple InfiniBand (IB) Host Channel Adapters (HCAs). On Linux, these HCAs appear as mlx5_x devices.
To troubleshoot connectivity or performance issues, it is essential to map:

• mlx5_x (Linux device name)

• Port GUID (from ibstat)

• Linux netdev interface (from ip a)

• Link speed (from ibstat or mlxlink)

• Connected switch and port (from ibnetdiscover or subnet manager topology dump)

Step 1 – Get Linux Device Mapping

Check available Mellanox devices:

ls /sys/class/infiniband

Example output:

mlx5_0  mlx5_1  mlx5_2  mlx5_3

Each corresponds to one HCA port.

Step 2 – Get Port GUIDs (ibstat)

Run the following for each HCA:

ibstat mlx5_0

Look for Port GUID:

Port GUID: 0x248a070300abcd01

This uniquely identifies the IB port in the fabric.

Step 3 – Map to Linux netdev (ip a)

Check ip a output for ibX interfaces:

ip -o link show | grep ib

Sample:

5: ib0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 4092 ...
6: ib1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 4092 ...

The netdev (e.g., ib0) corresponds to the mlx5_x device.
Cross-check via:

cat /sys/class/infiniband/mlx5_0/device/net/*

Step 4 – Check Link Speed

Use either:

ibstat mlx5_0

or

mlxlink -d mlx5_0 -p 1

Look for:

Link layer: InfiniBand
State: Active
Rate: 400 Gb/sec (EDR/HDR/NDR depending on gen)

Step 5 – Find Connected Switch Port

To identify which switch and port the HCA is connected to:

1. Run topology discovery on the fabric:

   ibnetdiscover | grep <Port GUID>
   

   or

   ibdiagnet -v | grep <Port GUID>
   

2. Output will show mapping like:

   CA: mlx5_0 (GUID 0x248a070300abcd01) port 1 <--> SwitchX-2 (GUID 0x248a070400123456) port 12
   

This gives the node HCA → switch port mapping.

Example Table

⚠️ Indicates ports running below expected speed.

Summary

• mlx5_x = Linux HCA device name.

• Use ibstat to fetch Port GUID and speed.

• Use ip a or /sys/class/infiniband to map to Linux netdev (ib0, ib1, etc.).

• Use ibnetdiscover / ibdiagnet to identify the connected switch and port.

• Build a mapping table for each node to simplify troubleshooting.