mine

Hi, I added a ConnectX-3 network card with 2 SFP+ ports to my Unraid system. What I want to achieve with this are four things: Use 1 SFP+ Module with a DAC cable to connect to a switch, the second SFP+ port stays empty. Use the 10G connectivity in Unraid as the main network connectivity. Use the 10G connectivity for Docker Containers via bridge interfaces. Use the 10G connectivity via virtual interfaces inside VM instances using the SR-IOV functionality. To get the card with SR-IOV working I did the following Installed an Ubuntu 24.04 VM and passed the entire card into the VM in order to access it with the Mellanox tools version 4.22.1 [1]. Enabled SR-IOV by issuing these two commands mlxconfig -d /dev/mst/mt4099_pciconf0 set SRIOV_EN=1 mlxconfig -d /dev/mst/mt4099_pciconf0 set NUM_OF_VFS=16 Then I stopped the VM, unbound the Mellanox card from VIFO and restarted Unraid. To get the virtual interfaces populated, I had to add "options mlx4_core num_vfs=16 port_type_array=2,2 probe_vf=1" to the mlx4_core driver at System Drivers and restarted Unraid. Now the network card, along with its virtual interfaces, shows up in the System Devices section of Unraid. See picture 1. To get the green icon, I need to issue the command "/usr/local/sbin/vfio-pci" on Unraid. My problem now is, that all devices are listed in a single group. That makes it impossible within the GUI bind virtual interfaces only, but keep the physical interface at the host. In another post I saw, that the Mellanox card should show up with its virtual interfaces in separate groups. See [2]. Sure, it is possible to manually edit the file "/boot/config/vfio-pci.cfg" and add just the virtual interfaces that should be bound (BIND=0000:02:01.1|15b3:1004 0000:02:00.1|15b3:1004) But with that, my Ubuntu VM is unable to load the Mellanox network card. This is what I get in dmesg. [ 1.187054] mlx4_core: Mellanox ConnectX core driver v4.0-0 [ 1.187322] mlx4_core: Initializing 0000:06:00.0 [ 1.187708] mlx4_core 0000:06:00.0: Detected virtual function - running in slave mode [ 31.246312] mlx4_core 0000:06:00.0: Communication channel is offline. [ 31.246551] mlx4_core 0000:06:00.0: PF is not responsive, skipping initialization [ 31.246766] mlx4_core 0000:06:00.0: Failed to initialize slave [ 31.248520] mlx4_core: probe of 0000:06:00.0 failed with error -5 On the Ubuntu Host, with "mlxconfig -d 06:00.0 q" I'm getting Device #1: ---------- Device type: ConnectX3Pro Device: 06:00.0 Configurations: Next Boot SRIOV_EN True(1) NUM_OF_VFS 16 WOL_MAGIC_EN_P1 True(1) LOG_BAR_SIZE 3 BOOT_OPTION_ROM_EN_P1 False(0) BOOT_VLAN_EN_P1 False(0) BOOT_RETRY_CNT_P1 0 LEGACY_BOOT_PROTOCOL_P1 PXE(1) BOOT_VLAN_P1 1 BOOT_OPTION_ROM_EN_P2 False(0) BOOT_VLAN_EN_P2 False(0) BOOT_RETRY_CNT_P2 0 LEGACY_BOOT_PROTOCOL_P2 PXE(1) BOOT_VLAN_P2 1 IP_VER_P1 IPv4(0) IP_VER_P2 IPv4(0) CQ_TIMESTAMP True(1) Output of "flint -dev /dev/mst/mt4099_pciconf0 dc" ;; Generated automatically by iniprep tool on Tue Sep 05 14:05:13 IDT 2017 from ./cx3-1_MCX342A_ipmi_dis.prs [PS_INFO] Name = MCX342A-XCC_Ax Description = ConnectX-3 EN NIC for OCP;10GbE;dual-port SFP+;PCIe3.0 x8;IPMI disabled;R6 PRS_name = cx3-1_MCX342A_ipmi_dis.prs [ADAPTER] PSID = MT_1680110023 pcie_gen2_speed_supported = true pcie_gen3_speed_supported = true adapter_dev_id = 0x1003 silicon_rev = 0x01 gpio_mode1 = 0x08000021 gpio_mode0 = 0x04d06ade gpio_default_val = 0x0e78423f gpio_pull_up = 0xff2bae1f gpio_pull_enable = 0x00001100 receiver_detect_en = true no_oc_en_gpio = 1 nv_cfg_en = true [HCA] hca_header_device_id = 0x1003 hca_header_subsystem_id = 0x0113 hca_header_class_code = 0x020000 dpdp_en = false eth_xfi_en = true mdio_en_port1 = 0 pcie_tx_polarity = 0xf0 i2c2_slave_en = false power_save_enable = 1 port1_wol_en = 1 port1_wol_magic = 1 port1_rol_en = 1 port1_rol_magic = 1 port1_default_phy_type=2 port2_default_phy_type=2 [IB] mlpn_en_port0 = true mlpn_en_port1 = true phy_type_port1 = XFI phy_type_port2 = XFI ext_phy_board_port1 = HAWK3 ext_phy_board_port2 = HAWK3 gen_guids_from_mac = true do_sense = false new_gpio_scheme_en = true read_cable_params_port1_en = true read_cable_params_port2_en = true port1_802_3ap_cr4_ability = false port2_802_3ap_cr4_ability = false port1_802_3ap_cr4_enable = false port2_802_3ap_cr4_enable = false center_mix90phase = true ref_clk_to_use = 1 ;;Logic lane to Serdes mapping tx_logic_0_serdes = 0 tx_logic_1_serdes = 1 tx_logic_2_serdes = 2 tx_logic_3_serdes = 3 rx_logic_0_serdes = 0 rx_logic_1_serdes = 1 rx_logic_2_serdes = 2 rx_logic_3_serdes = 3 tx_logic_4_serdes = 4 tx_logic_5_serdes = 5 tx_logic_6_serdes = 6 tx_logic_7_serdes = 7 rx_logic_4_serdes = 4 rx_logic_5_serdes = 5 rx_logic_6_serdes = 6 rx_logic_7_serdes = 7 eth_tx_lane_polarity_port1 = 0x1 eth_tx_lane_polarity_port2 = 0x0 eth_rx_lane_polarity_port1 = 0x1 eth_rx_lane_polarity_port2 = 0x1 ; start of '#include "include_sfpp_serdes_prams_kfir.h"' ;;Serdes parameters ;; force 7 - XFI force_rx7_slicer_ind_en = 0xeb force_rx7_slicer1_enable = 0x0 force_rx7_slicer2_enable = 0x0 force_rx7_ffe_tap0 = 0x64 force_rx7_ffe_tap1 = 0x80 force_rx7_ffe_tap2 = 0xde force_rx7_ffe_tap3 = 0x80 force_rx7_ffe_tap4 = 0x46 force_tx7_ob_bias = 0xd force_tx7_ob_preemp_pre = 0x20 force_tx7_ob_preemp_post = 0x0 force_tx7_ob_preemp_main = 0x7f force_tx7_preemp = 0x0 force_tx7_pre_polarity = 0x1 force_tx7_post_polarity = 0x1 force_tx7_main_polarity = 0x0 ;; negotiation IB nego_rx1_slicer_ind_en = 0xeb nego_rx1_slicer1_enable=0 nego_rx1_slicer2_enable=0 nego_rx1_ffe_tap0=0x78 nego_rx1_ffe_tap1=0x80 nego_rx1_ffe_tap2=0xde nego_rx1_ffe_tap3=0x80 nego_rx1_ffe_tap4=0x3c ;; negotiation ETH nego_eth_rx1_slicer_ind_en = 0xff nego_eth_rx1_slicer1_enable= 0x8 nego_eth_rx1_slicer2_enable= 0x8 nego_eth_rx1_ffe_tap0=225 nego_eth_rx1_ffe_tap1=128 nego_eth_rx1_ffe_tap2=50 nego_eth_rx1_ffe_tap3=110 nego_eth_rx1_ffe_tap4=122 nego_eth_rx12_slicer_ind_en = 0xff nego_eth_rx12_slicer1_enable= 0x8 nego_eth_rx12_slicer2_enable= 0x8 nego_eth_rx12_ffe_tap0=241 nego_eth_rx12_ffe_tap1=128 nego_eth_rx12_ffe_tap2=61 nego_eth_rx12_ffe_tap3=99 nego_eth_rx12_ffe_tap4=128 parallel_detect_rx0_slicer_ind_en = 0xeb parallel_detect_rx0_slicer1_enable = 0x0 parallel_detect_rx0_slicer2_enable = 0x0 parallel_detect_rx0_ffe_tap0 = 0x64 parallel_detect_rx0_ffe_tap1 = 0x80 parallel_detect_rx0_ffe_tap2 = 0xde parallel_detect_rx0_ffe_tap3 = 0x80 parallel_detect_rx0_ffe_tap4 = 0x46 ;; ETH connected to third party device aba_non_mlpn_tx8_ob_preemp_pre = 5 aba_non_mlpn_tx8_ob_preemp_post = 0 aba_non_mlpn_tx8_ob_preemp_main = 65 aba_non_mlpn_tx8_ob_bias = 8 aba_non_mlpn_tx8_pre_polarity = 1 aba_non_mlpn_tx8_post_polarity = 1 aba_non_mlpn_tx8_main_polarity = 0 aba_non_mlpn_tx8_preemp = 0 ; end of '#include "include_sfpp_serdes_prams_kfir.h"' aba_tx9_ob_preemp_pre = 7 aba_tx9_ob_preemp_main = 120 aba_tx9_ob_preemp_post = 0 aba_tx9_ob_bias = 5 aba_tx9_pre_polarity = 1 aba_tx9_post_polarity = 1 aba_tx9_main_polarity = 0 aba_non_mlpn_tx8_ob_preemp_pre = 30 aba_non_mlpn_tx8_ob_preemp_main = 100 aba_non_mlpn_tx8_ob_preemp_post = 0 aba_non_mlpn_tx8_ob_bias = 8 aba_non_mlpn_tx8_pre_polarity = 1 aba_non_mlpn_tx8_post_polarity = 1 aba_non_mlpn_tx8_main_polarity = 0 aba_non_mlpn_tx9_ob_preemp_pre = 50 aba_non_mlpn_tx9_ob_preemp_main = 127 aba_non_mlpn_tx9_ob_preemp_post = 22 aba_non_mlpn_tx9_ob_bias = 8 aba_non_mlpn_tx9_pre_polarity = 1 aba_non_mlpn_tx9_post_polarity = 1 aba_non_mlpn_tx9_main_polarity = 0 nego_eth_rx2_slicer_ind_en=255 nego_eth_rx2_slicer1_enable=8 nego_eth_rx2_slicer2_enable=8 nego_eth_rx2_ffe_tap0=234 nego_eth_rx2_ffe_tap1=129 nego_eth_rx2_ffe_tap2=50 nego_eth_rx2_ffe_tap3=110 nego_eth_rx2_ffe_tap4=122 [PLL] lbist_en = 0 lbist_shift_freq = 3 flash_div = 0x3 lbist_array_bypass = 1 lbist_pat_cnt_lsb = 0x2 core_f = 42 core_r = 12 core_od = 2 [FW] flash_has_suspend_resume = 0 log_flashdev_size = 21 log_flash_sector_size = 6 Does anybody have an idea what I'm doing wrong? Thanks, mine Picture 1 [1] https://network.nvidia.com/products/adapter-software/firmware-tools/ [2] https://forums.unraid.net/topic/103323-how-to-using-sr-iov-in-unraid-with-1gb10gb40gb-network-interface-cards-nics/#findComment-955942

Just wanted to report, that today I ran into the same issue, that after a reboot, the cache devices were marked unmountable. I followed these steps, and got it back working. I did not have to issue the "btrfs rescue zero-log". Although I had a backup of the files on the cache, these explanations saved me a lot of work. This is why I want to say THANK YOU. I also want to thank @josephsiu, who was patient enough to follow up until the problem was solved.

Yes, that sounds to be exactly what I need. Thanks.

Hi, due to a drive defect inside an array, I ordered 2 new 14TB drives. Before the drives arrived, I already removed the defect harddisk from the array. The defect HDD is 3TB in size. One of the new 14TB drives I want to use to replace the old data drive, and the other one I want to use to replace the working parity drive (4TB) (to be able to consume the full 14TB of the new drive). What would be the correct procedure to replace the defect data disk from the degraded array and then to replace the parity drive? Would the following be the correct procedure? Stop the array Add the new data disk to the slot where the defect one was Then applying the changes by starting the array Wait for the sync to finish Stop the array Assign Parity disk from the small one to the new, larger one Submit changes by starting the array Rebuild of parity drive should be happening now, and I can use the server already Is the procedure correct? Will I be able after that change to automatically use the full space of the 14TB data drive? Thanks, MiNe

Thanks for confirming.

After a reboot of the system and checking the cables of the disk, I'm able to read out SMART and see these errors. I guess, this doesn't look good for the drive... The UDMA CRC errors are there since long time already, but "Current pending sector" and "Offline uncorrectable" are new.

Hi, While copying new data to my array, one of the drives became disabled. Looking at the logs, I saw multiple "I/O error". See the Diagnostics file attached. After stopping the array and starting it again Unraid mounted the drive but did not re-enable it. Also no I/O errors right after mounting starting the array. So I decided to follow the guide of rebuilding the device to itself, because with the help of the parity drive, all data were there. However, just after 0.3% of progress, I got the same I/O errors again and the re-build process stopped. Do you have any further advises how to proceed? Is the device defect? Do I need to perform any other steps before adding back the device? Thanks, MiNe zentrale-diagnostics-20220528-1036.zip