March 29Mar 29 Hi,I added a ConnectX-3 network card with 2 SFP+ ports to my Unraid system.What I want to achieve with this are four things:Use 1 SFP+ Module with a DAC cable to connect to a switch, the second SFP+ port stays empty.Use the 10G connectivity in Unraid as the main network connectivity.Use the 10G connectivity for Docker Containers via bridge interfaces.Use the 10G connectivity via virtual interfaces inside VM instances using the SR-IOV functionality.To get the card with SR-IOV working I did the followingInstalled an Ubuntu 24.04 VM and passed the entire card into the VM in order to access it with the Mellanox tools version 4.22.1 [1].Enabled SR-IOV by issuing these two commandsmlxconfig -d /dev/mst/mt4099_pciconf0 set SRIOV_EN=1mlxconfig -d /dev/mst/mt4099_pciconf0 set NUM_OF_VFS=16Then I stopped the VM, unbound the Mellanox card from VIFO and restarted Unraid.To get the virtual interfaces populated, I had to add "options mlx4_core num_vfs=16 port_type_array=2,2 probe_vf=1" to the mlx4_core driver at System Drivers and restarted Unraid.Now the network card, along with its virtual interfaces, shows up in the System Devices section of Unraid. See picture 1.To get the green icon, I need to issue the command "/usr/local/sbin/vfio-pci" on Unraid.My problem now is, that all devices are listed in a single group. That makes it impossible within the GUI bind virtual interfaces only, but keep the physical interface at the host.In another post I saw, that the Mellanox card should show up with its virtual interfaces in separate groups. See [2].Sure, it is possible to manually edit the file "/boot/config/vfio-pci.cfg" and add just the virtual interfaces that should be bound (BIND=0000:02:01.1|15b3:1004 0000:02:00.1|15b3:1004)But with that, my Ubuntu VM is unable to load the Mellanox network card. This is what I get in dmesg.[ 1.187054] mlx4_core: Mellanox ConnectX core driver v4.0-0 [ 1.187322] mlx4_core: Initializing 0000:06:00.0 [ 1.187708] mlx4_core 0000:06:00.0: Detected virtual function - running in slave mode [ 31.246312] mlx4_core 0000:06:00.0: Communication channel is offline. [ 31.246551] mlx4_core 0000:06:00.0: PF is not responsive, skipping initialization [ 31.246766] mlx4_core 0000:06:00.0: Failed to initialize slave [ 31.248520] mlx4_core: probe of 0000:06:00.0 failed with error -5On the Ubuntu Host, with "mlxconfig -d 06:00.0 q" I'm gettingDevice #1: ---------- Device type: ConnectX3Pro Device: 06:00.0 Configurations: Next Boot SRIOV_EN True(1) NUM_OF_VFS 16 WOL_MAGIC_EN_P1 True(1) LOG_BAR_SIZE 3 BOOT_OPTION_ROM_EN_P1 False(0) BOOT_VLAN_EN_P1 False(0) BOOT_RETRY_CNT_P1 0 LEGACY_BOOT_PROTOCOL_P1 PXE(1) BOOT_VLAN_P1 1 BOOT_OPTION_ROM_EN_P2 False(0) BOOT_VLAN_EN_P2 False(0) BOOT_RETRY_CNT_P2 0 LEGACY_BOOT_PROTOCOL_P2 PXE(1) BOOT_VLAN_P2 1 IP_VER_P1 IPv4(0) IP_VER_P2 IPv4(0) CQ_TIMESTAMP True(1)Output of "flint -dev /dev/mst/mt4099_pciconf0 dc";; Generated automatically by iniprep tool on Tue Sep 05 14:05:13 IDT 2017 from ./cx3-1_MCX342A_ipmi_dis.prs [PS_INFO] Name = MCX342A-XCC_Ax Description = ConnectX-3 EN NIC for OCP;10GbE;dual-port SFP+;PCIe3.0 x8;IPMI disabled;R6 PRS_name = cx3-1_MCX342A_ipmi_dis.prs [ADAPTER] PSID = MT_1680110023 pcie_gen2_speed_supported = true pcie_gen3_speed_supported = true adapter_dev_id = 0x1003 silicon_rev = 0x01 gpio_mode1 = 0x08000021 gpio_mode0 = 0x04d06ade gpio_default_val = 0x0e78423f gpio_pull_up = 0xff2bae1f gpio_pull_enable = 0x00001100 receiver_detect_en = true no_oc_en_gpio = 1 nv_cfg_en = true [HCA] hca_header_device_id = 0x1003 hca_header_subsystem_id = 0x0113 hca_header_class_code = 0x020000 dpdp_en = false eth_xfi_en = true mdio_en_port1 = 0 pcie_tx_polarity = 0xf0 i2c2_slave_en = false power_save_enable = 1 port1_wol_en = 1 port1_wol_magic = 1 port1_rol_en = 1 port1_rol_magic = 1 port1_default_phy_type=2 port2_default_phy_type=2 [IB] mlpn_en_port0 = true mlpn_en_port1 = true phy_type_port1 = XFI phy_type_port2 = XFI ext_phy_board_port1 = HAWK3 ext_phy_board_port2 = HAWK3 gen_guids_from_mac = true do_sense = false new_gpio_scheme_en = true read_cable_params_port1_en = true read_cable_params_port2_en = true port1_802_3ap_cr4_ability = false port2_802_3ap_cr4_ability = false port1_802_3ap_cr4_enable = false port2_802_3ap_cr4_enable = false center_mix90phase = true ref_clk_to_use = 1 ;;Logic lane to Serdes mapping tx_logic_0_serdes = 0 tx_logic_1_serdes = 1 tx_logic_2_serdes = 2 tx_logic_3_serdes = 3 rx_logic_0_serdes = 0 rx_logic_1_serdes = 1 rx_logic_2_serdes = 2 rx_logic_3_serdes = 3 tx_logic_4_serdes = 4 tx_logic_5_serdes = 5 tx_logic_6_serdes = 6 tx_logic_7_serdes = 7 rx_logic_4_serdes = 4 rx_logic_5_serdes = 5 rx_logic_6_serdes = 6 rx_logic_7_serdes = 7 eth_tx_lane_polarity_port1 = 0x1 eth_tx_lane_polarity_port2 = 0x0 eth_rx_lane_polarity_port1 = 0x1 eth_rx_lane_polarity_port2 = 0x1 ; start of '#include "include_sfpp_serdes_prams_kfir.h"' ;;Serdes parameters ;; force 7 - XFI force_rx7_slicer_ind_en = 0xeb force_rx7_slicer1_enable = 0x0 force_rx7_slicer2_enable = 0x0 force_rx7_ffe_tap0 = 0x64 force_rx7_ffe_tap1 = 0x80 force_rx7_ffe_tap2 = 0xde force_rx7_ffe_tap3 = 0x80 force_rx7_ffe_tap4 = 0x46 force_tx7_ob_bias = 0xd force_tx7_ob_preemp_pre = 0x20 force_tx7_ob_preemp_post = 0x0 force_tx7_ob_preemp_main = 0x7f force_tx7_preemp = 0x0 force_tx7_pre_polarity = 0x1 force_tx7_post_polarity = 0x1 force_tx7_main_polarity = 0x0 ;; negotiation IB nego_rx1_slicer_ind_en = 0xeb nego_rx1_slicer1_enable=0 nego_rx1_slicer2_enable=0 nego_rx1_ffe_tap0=0x78 nego_rx1_ffe_tap1=0x80 nego_rx1_ffe_tap2=0xde nego_rx1_ffe_tap3=0x80 nego_rx1_ffe_tap4=0x3c ;; negotiation ETH nego_eth_rx1_slicer_ind_en = 0xff nego_eth_rx1_slicer1_enable= 0x8 nego_eth_rx1_slicer2_enable= 0x8 nego_eth_rx1_ffe_tap0=225 nego_eth_rx1_ffe_tap1=128 nego_eth_rx1_ffe_tap2=50 nego_eth_rx1_ffe_tap3=110 nego_eth_rx1_ffe_tap4=122 nego_eth_rx12_slicer_ind_en = 0xff nego_eth_rx12_slicer1_enable= 0x8 nego_eth_rx12_slicer2_enable= 0x8 nego_eth_rx12_ffe_tap0=241 nego_eth_rx12_ffe_tap1=128 nego_eth_rx12_ffe_tap2=61 nego_eth_rx12_ffe_tap3=99 nego_eth_rx12_ffe_tap4=128 parallel_detect_rx0_slicer_ind_en = 0xeb parallel_detect_rx0_slicer1_enable = 0x0 parallel_detect_rx0_slicer2_enable = 0x0 parallel_detect_rx0_ffe_tap0 = 0x64 parallel_detect_rx0_ffe_tap1 = 0x80 parallel_detect_rx0_ffe_tap2 = 0xde parallel_detect_rx0_ffe_tap3 = 0x80 parallel_detect_rx0_ffe_tap4 = 0x46 ;; ETH connected to third party device aba_non_mlpn_tx8_ob_preemp_pre = 5 aba_non_mlpn_tx8_ob_preemp_post = 0 aba_non_mlpn_tx8_ob_preemp_main = 65 aba_non_mlpn_tx8_ob_bias = 8 aba_non_mlpn_tx8_pre_polarity = 1 aba_non_mlpn_tx8_post_polarity = 1 aba_non_mlpn_tx8_main_polarity = 0 aba_non_mlpn_tx8_preemp = 0 ; end of '#include "include_sfpp_serdes_prams_kfir.h"' aba_tx9_ob_preemp_pre = 7 aba_tx9_ob_preemp_main = 120 aba_tx9_ob_preemp_post = 0 aba_tx9_ob_bias = 5 aba_tx9_pre_polarity = 1 aba_tx9_post_polarity = 1 aba_tx9_main_polarity = 0 aba_non_mlpn_tx8_ob_preemp_pre = 30 aba_non_mlpn_tx8_ob_preemp_main = 100 aba_non_mlpn_tx8_ob_preemp_post = 0 aba_non_mlpn_tx8_ob_bias = 8 aba_non_mlpn_tx8_pre_polarity = 1 aba_non_mlpn_tx8_post_polarity = 1 aba_non_mlpn_tx8_main_polarity = 0 aba_non_mlpn_tx9_ob_preemp_pre = 50 aba_non_mlpn_tx9_ob_preemp_main = 127 aba_non_mlpn_tx9_ob_preemp_post = 22 aba_non_mlpn_tx9_ob_bias = 8 aba_non_mlpn_tx9_pre_polarity = 1 aba_non_mlpn_tx9_post_polarity = 1 aba_non_mlpn_tx9_main_polarity = 0 nego_eth_rx2_slicer_ind_en=255 nego_eth_rx2_slicer1_enable=8 nego_eth_rx2_slicer2_enable=8 nego_eth_rx2_ffe_tap0=234 nego_eth_rx2_ffe_tap1=129 nego_eth_rx2_ffe_tap2=50 nego_eth_rx2_ffe_tap3=110 nego_eth_rx2_ffe_tap4=122 [PLL] lbist_en = 0 lbist_shift_freq = 3 flash_div = 0x3 lbist_array_bypass = 1 lbist_pat_cnt_lsb = 0x2 core_f = 42 core_r = 12 core_od = 2 [FW] flash_has_suspend_resume = 0 log_flashdev_size = 21 log_flash_sector_size = 6Does anybody have an idea what I'm doing wrong?Thanks,minePicture 1[1] https://network.nvidia.com/products/adapter-software/firmware-tools/[2] https://forums.unraid.net/topic/103323-how-to-using-sr-iov-in-unraid-with-1gb10gb40gb-network-interface-cards-nics/#findComment-955942
Join the conversation
You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.