This guide explains how to enable RoCE on a SONiC switch, verify RDMA interfaces on connected compute nodes, and validate performance using RDMA bandwidth tests.
PFC ensures lossless Ethernet behavior required by RoCE.
sonic-cli
configure terminal
# Enable RoCe with default settings
sonic(config)# roce enable
force-defaults Clear any previous applied QOS buffer and force Initialize RoCEv2 default buffer configuration
pfc-priority RoCEv2 buffer configuration based on the PFC priorities
<cr>
sonic(config)# roce enable
This command will also restart the node after saving all configurations,if ROCE is configured first time or force-default. [Proceed y/N]: y
sonic(config)# Waiting for the reboot operation to complete
Check that PFC is active and watchdog is off (optional):
show qos interface Ethernet0
Expected output:
sonic# show qos interface Ethernet 0
scheduler policy: ROCE
dscp-tc-map: ROCE
dot1p-tc-map: ROCE
tc-queue-map: ROCE
tc-pg-map: ROCE
pfc-priority-queue-map: ROCE
pfc-priority-pg-map: ROCE
pfc-asymmetric: off
pfc-priority : 3,4
PFC Watchdog
Status : on
Action : drop
Detection Time : 200ms
Restoration Time : 400ms
Result: SONiC switch ports are now RoCE-ready.
Run these on each Ubuntu compute node.
rdma link show
Expected output:
link rocep65s0f0/1 state ACTIVE physical_state LINK_UP netdev enp65s0f0np0
link rocep65s0f1/1 state ACTIVE physical_state LINK_UP netdev enp65s0f1np1
ibstat
Expected output:
Link layer: Ethernet
Rate: 200
State: Active
Result: Indicates that RDMA (RoCE) links are active and running on Ethernet mode at 200 Gb/s.
| VLAN | Purpose | Node 1 IP | Node 2 IP | Interface |
|---|---|---|---|---|
| VLAN 100 | HSN-A | 10.1.1.11 | 10.1.1.12 | rocep65s0f1 / rocep1s0f0 |
| VLAN 200 | HSN-B | 10.1.2.11 | 10.1.2.12 | rocep65s0f0 / rocep1s0f1 |
Make sure RDMA utilities are installed:
sudo apt install rdma-core ibverbs-utils perftest -y
ib_write_bw -d rocep65s0f1 -F --report_gbits
ib_write_bw -d rocep1s0f0 -F --report_gbits 10.1.1.11
Expected output:
---------------------------------------------------------------------------------------
#bytes #iterations BW peak[Gb/sec] BW average[Gb/sec] MsgRate[Mpps]
65536 5000 185.05 185.03 0.352922
---------------------------------------------------------------------------------------
Result: Confirms 185+ Gb/s effective throughput on 200G RoCE link.
Check RDMA statistics:
rdma statistic show