Fluxonium

Thanks I have ordered the Startech PEXESAT322I arrives tomorrow. I will report back after testing

Hi All, Can anybody recommend a PCIe 3.0 x1 SATA 1x adapter? I need to connect a DVD / Blue-Ray to a Virtual Machine and have run out of motherboard connections. I did ty the following from amazon: MZHOU PCIe SATA Card 2 Port But the system gets stuck at the BIOS / motherboard logo screen. Tested it in both PCIe 3.0 1x ports and the same result, I can get past the boot screen if I disconnect the DVD drive but with it plugged in the system will not post. I have not tested with a HDD but given I need the DVD drive I see no point. I have tested the DVD drive in a motherboard SATA connection and it works fine so I think the problem is the cheap amazon adapter. Best regards, Andy

Hi All, Thanks for your suggestions, I have ordered another Intel X540-T2 adapter (this time Dell, instead of HP) from eBay. This will allow me to rule out the TP-Link card and test the HP Intel X540-T2 adapter. Maybe I have a broken / half working NIC??? I have found a few post online regarding the Marvell AQtion 10Gbit which is in the TP-Link.

OK, so turned on Jumbo Packet to 9014 and set the unraid MUT to 9000 and get the following: Unraid Client root@Obsidian:~# iperf3 -c 192.168.10.1 Connecting to host 192.168.10.1, port 5201 [ 5] local 192.168.10.100 port 55066 connected to 192.168.10.1 port 5201 [ ID] Interval Transfer Bitrate Retr Cwnd [ 5] 0.00-1.00 sec 879 MBytes 7.37 Gbits/sec 0 489 KBytes [ 5] 1.00-2.00 sec 875 MBytes 7.34 Gbits/sec 0 437 KBytes [ 5] 2.00-3.00 sec 878 MBytes 7.36 Gbits/sec 0 437 KBytes [ 5] 3.00-4.00 sec 886 MBytes 7.43 Gbits/sec 0 454 KBytes [ 5] 4.00-5.00 sec 879 MBytes 7.37 Gbits/sec 0 454 KBytes [ 5] 5.00-6.00 sec 882 MBytes 7.40 Gbits/sec 0 489 KBytes [ 5] 6.00-7.00 sec 883 MBytes 7.41 Gbits/sec 0 437 KBytes [ 5] 7.00-8.00 sec 860 MBytes 7.22 Gbits/sec 0 454 KBytes [ 5] 8.00-9.00 sec 877 MBytes 7.36 Gbits/sec 0 507 KBytes [ 5] 9.00-10.00 sec 882 MBytes 7.39 Gbits/sec 0 437 KBytes - - - - - - - - - - - - - - - - - - - - - - - - - [ ID] Interval Transfer Bitrate Retr [ 5] 0.00-10.00 sec 8.57 GBytes 7.37 Gbits/sec 0 sender [ 5] 0.00-10.00 sec 8.57 GBytes 7.36 Gbits/sec receiver Unraid Server: ----------------------------------------------------------- Server listening on 5201 (test #1) ----------------------------------------------------------- Accepted connection from 192.168.10.1, port 62206 [ 5] local 192.168.10.100 port 5201 connected to 192.168.10.1 port 62207 [ ID] Interval Transfer Bitrate [ 5] 0.00-1.00 sec 945 MBytes 7.93 Gbits/sec [ 5] 1.00-2.00 sec 948 MBytes 7.95 Gbits/sec [ 5] 2.00-3.00 sec 952 MBytes 7.98 Gbits/sec [ 5] 3.00-4.00 sec 954 MBytes 8.00 Gbits/sec [ 5] 4.00-5.00 sec 954 MBytes 8.00 Gbits/sec [ 5] 5.00-6.00 sec 960 MBytes 8.05 Gbits/sec [ 5] 6.00-7.00 sec 956 MBytes 8.02 Gbits/sec [ 5] 7.00-8.00 sec 950 MBytes 7.97 Gbits/sec [ 5] 8.00-9.00 sec 958 MBytes 8.03 Gbits/sec [ 5] 9.00-10.00 sec 951 MBytes 7.98 Gbits/sec [ 5] 10.00-10.00 sec 3.38 MBytes 7.92 Gbits/sec - - - - - - - - - - - - - - - - - - - - - - - - - [ ID] Interval Transfer Bitrate [ 5] 0.00-10.00 sec 9.31 GBytes 7.99 Gbits/sec receiver ----------------------------------------------------------- Server listening on 5201 (test #2) ----------------------------------------------------------- not sure why this worked as i thought jumbo packets are normally not required. Maybe the issue is the TP-Link 10GB card in the windows 11 pc as it is a Marvell AQtion 10Gbit??? or the Intel X540-AT2 NIC has an issue? wrong firmware... (not sure how i would check).

Cable is 3m CAT6A also tested 3M CAT6 and CAT5E cable same result.

Unraid as the client: root@Obsidian:~# iperf3 -c 192.168.10.1 Connecting to host 192.168.10.1, port 5201 [ 5] local 192.168.10.100 port 47914 connected to 192.168.10.1 port 5201 [ ID] Interval Transfer Bitrate Retr Cwnd [ 5] 0.00-1.00 sec 238 MBytes 2.00 Gbits/sec 0 283 KBytes [ 5] 1.00-2.00 sec 234 MBytes 1.96 Gbits/sec 0 294 KBytes [ 5] 2.00-3.00 sec 240 MBytes 2.01 Gbits/sec 0 286 KBytes [ 5] 3.00-4.00 sec 247 MBytes 2.07 Gbits/sec 0 311 KBytes [ 5] 4.00-5.00 sec 251 MBytes 2.10 Gbits/sec 0 297 KBytes [ 5] 5.00-6.00 sec 268 MBytes 2.25 Gbits/sec 0 277 KBytes [ 5] 6.00-7.00 sec 261 MBytes 2.19 Gbits/sec 0 288 KBytes [ 5] 7.00-8.00 sec 259 MBytes 2.17 Gbits/sec 0 280 KBytes [ 5] 8.00-9.00 sec 245 MBytes 2.05 Gbits/sec 0 272 KBytes [ 5] 9.00-10.00 sec 257 MBytes 2.15 Gbits/sec 0 277 KBytes - - - - - - - - - - - - - - - - - - - - - - - - - [ ID] Interval Transfer Bitrate Retr [ 5] 0.00-10.00 sec 2.44 GBytes 2.10 Gbits/sec 0 sender [ 5] 0.00-10.00 sec 2.44 GBytes 2.09 Gbits/sec receiver Unraid as the server: ----------------------------------------------------------- Server listening on 5201 (test #1) ----------------------------------------------------------- Accepted connection from 192.168.10.1, port 60681 [ 5] local 192.168.10.100 port 5201 connected to 192.168.10.1 port 60682 [ ID] Interval Transfer Bitrate [ 5] 0.00-1.00 sec 1.01 GBytes 8.68 Gbits/sec [ 5] 1.00-2.00 sec 1.01 GBytes 8.67 Gbits/sec [ 5] 2.00-3.00 sec 1.01 GBytes 8.66 Gbits/sec [ 5] 3.00-4.00 sec 1.00 GBytes 8.61 Gbits/sec [ 5] 4.00-5.00 sec 1.00 GBytes 8.61 Gbits/sec [ 5] 5.00-6.00 sec 1.02 GBytes 8.76 Gbits/sec [ 5] 6.00-7.00 sec 1.01 GBytes 8.69 Gbits/sec [ 5] 7.00-8.00 sec 1.01 GBytes 8.69 Gbits/sec [ 5] 8.00-9.00 sec 1.01 GBytes 8.65 Gbits/sec [ 5] 9.00-10.00 sec 1.00 GBytes 8.63 Gbits/sec [ 5] 10.00-10.00 sec 4.78 MBytes 8.51 Gbits/sec - - - - - - - - - - - - - - - - - - - - - - - - - [ ID] Interval Transfer Bitrate [ 5] 0.00-10.00 sec 10.1 GBytes 8.67 Gbits/sec receiver ----------------------------------------------------------- Server listening on 5201 (test #2) -----------------------------------------------------------

Hi All, I have purchased an Intel X540-AT2 dual port 10Gbe card to replace the motherboard 2.5Gbe network card. Installed and working but i have some odd things happening: Copy a 10GB file from Windows to Unraid i get the full 10Gb speed, but copy any file from the array, or cache to Windows PC am capped at 230mb/s any ideas ? Windows 11: Intel i7 12700K 32GB 3600 DDR RAM 1TB SN850 TP-Link 10 Gigabit PCI Express Network Adapter Unraid: Intel i5-2500 48GB 3200 DDR4 RAM NVME cache pool 1 - 2x 1TB WDC_WDS100T2B0C in btrfs mirror SATA cache pool 2 - 3x 1TB WDC_WDS100T1R0A in zfs array Intel X540-AT2 dual port 10Gbe connecting in the main PCIex16 Gen4 slot. Port 1 - 1gb switch connection - DHCP from router Port 2 - static ip to WIndows 10GB card Thanks, Andy

You are correct, I thought by simply removing the IOMMU binding it would work. But looking at the XML VM config I also needed to remove the pass through settings by editing the xml. Thanks for your help

Now running a parity check on the main array . Should be fine it was spun down.

After the forced power-off: pool: zfs_cache state: ONLINE scan: scrub repaired 0B in 00:04:17 with 0 errors on Mon Apr 15 21:17:16 2024 config: NAME STATE READ WRITE CKSUM zfs_cache ONLINE 0 0 0 raidz1-0 ONLINE 0 0 0 /dev/sdc1 ONLINE 0 0 0 /dev/sdb1 ONLINE 0 0 0 /dev/sdd1 ONLINE 0 0 0 errors: No known data errors

Hi, I have just added 3x SATA 1TB WD SA500 SSD drives to the system and created a zfs, raidz, 1 group of 3 devices. I had a single nvme drive which hosted my VM's and docker containers, which I moved to the array then to the zfs cache (all without error). I had no docker containers so created a new docker image on the zfs cache which worked fine and installed plex also fine. When I try and start the old Windows 11 virtual machine I get zfs offset errors and the VM page is unrepsonsive and the server will not reboot or shutdown. so I have re-powered the system (holding down the power button - as shutdown and reboot in the console did not work, not sure what else I should have done). tired going to settings -> Vm Manager -> Enable No (but just hangs) Apr 15 20:56:52 Obsidian kernel: zio pool=zfs_cache vdev=/dev/sdc1 error=5 type=1 offset=121754525696 size=65536 flags=180880 Apr 15 20:56:52 Obsidian kernel: zio pool=zfs_cache vdev=/dev/sdd1 error=5 type=1 offset=121754521600 size=65536 flags=180880 Apr 15 20:56:52 Obsidian kernel: zio pool=zfs_cache vdev=/dev/sdb1 error=5 type=1 offset=121754521600 size=65536 flags=180880 Apr 15 20:56:52 Obsidian kernel: WARNING: Pool 'zfs_cache' has encountered an uncorrectable I/O failure and has been suspended. Apr 15 20:56:52 Obsidian kernel: Apr 15 20:56:55 Obsidian kernel: WARNING: Pool 'zfs_cache' has encountered an uncorrectable I/O failure and has been suspended. Apr 15 20:56:55 Obsidian kernel: root@Obsidian:~# zpool status -xv pool: zfs_cache state: SUSPENDED status: One or more devices are faulted in response to IO failures. action: Make sure the affected devices are connected, then run 'zpool clear'. see: https://openzfs.github.io/openzfs-docs/msg/ZFS-8000-HC scan: scrub repaired 0B in 00:04:16 with 0 errors on Mon Apr 15 20:40:48 2024 config: NAME STATE READ WRITE CKSUM zfs_cache ONLINE 0 0 0 raidz1-0 ONLINE 1 4 0 sdc1 ONLINE 2 2 0 sdb1 ONLINE 2 2 0 sdd1 ONLINE 2 4 0 errors: List of errors unavailable: pool I/O is currently suspended after the reboot, i did a scrub on the zfs cache which came back with no errors. i then decided to copy an new windows iso to the zfs cache and create a new VM, this worked with no reported zfs errors any ideas why the old VM image causes issues (wrong file system)??? Best regards, Andy

OK, after some more testing it looks like the issue is the QNAP QM2-2P-374 as the nvme drives work in the motherboard slots without error (so far :)). I am guessing this is a Linux / unraid driver issue as the card works fine in Windows.

Unfortunately, "nvme_core.default_ps_max_latency_us=0 pcie_aspm=off" did not work. Strange thing is I cannot break the drive in Windows both work perfectly (on the same hardware other than the boot disk), going to try the following but cannot see it working unless Windows is not doing something unraid is (better / more stable drivers). i have even tried sleeping / hibernate from Windows and both drives still work. Remove adapter and swap drive positions (clean drive and adapter contacts) Remove both drives from the adapter and insert into motherboard and re-test

"nvme_core.default_ps_max_latency_us=0 pcie_aspm=off" I will give it ago, seems unlikely to fix this issue as the other WDS100T2B0C has no issue. I have also tested the drives and controller in Windows using using HD sentinel - Extended SMART test then 2x write - read surface scans - all with no issues, also copied enough data to fill the drive to 90% in windows and no issues or warnings, using the same hardware just a different boot drive, so looks like an issue with Linux / unraid for some reason. i am also getting the following in the log from time to time: Mar 27 16:03:29 Obsidian kernel: pcieport 0000:00:1c.4: AER: Multiple Corrected error received: 0000:08:00.0 Mar 27 16:03:29 Obsidian kernel: nvme 0000:08:00.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, (Receiver ID) Mar 27 16:03:29 Obsidian kernel: nvme 0000:08:00.0: device [15b7:5009] error status/mask=00000001/0000e000 Mar 27 16:03:29 Obsidian kernel: nvme 0000:08:00.0: [ 0] RxErr Mar 27 16:24:19 Obsidian kernel: pcieport 0000:00:1c.4: AER: Multiple Corrected error received: 0000:09:00.0 Mar 27 16:24:19 Obsidian kernel: nvme 0000:09:00.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, (Receiver ID) Mar 27 16:24:19 Obsidian kernel: nvme 0000:09:00.0: device [15b7:5009] error status/mask=00000001/0000e000 Mar 27 16:24:19 Obsidian kernel: nvme 0000:09:00.0: [ 0] RxErr device 15b7:5009 is the QNAP nvme to PCIe adapter both nvme3n1 and nvme2n1 are connected to: IOMMU group 22: [15b7:5009] 08:00.0 Non-Volatile memory controller: Sandisk Corp SanDisk Ultra 3D / WD Blue SN550 NVMe SSD (rev 01) [N:2:1:1] disk WDC WDS100T2B0C-00PXH0__1 /dev/nvme2n1 1.00TB IOMMU group 23: [15b7:5009] 09:00.0 Non-Volatile memory controller: Sandisk Corp SanDisk Ultra 3D / WD Blue SN550 NVMe SSD (rev 01) [N:3:1:1] disk WDC WDS100T2B0C-00PXH0__1 /dev/nvme3n1 1.00TB BIOS has been updated to the latest version and all drive firmware has been checked in Windows for the latest version. Thanks for your help.

Hi All, I am new to unraid, swapped from a QNAP NAS as it failed and they wanted £1040 to try and fix the issue or buy I have the following: Asus PRIME B660-PLUS D4 i5-2400 48GB DDR4 - 2x 8GB 3200 DDR4, 2x 16GB 3200 DDR4 1x LSI 9211-8i IT Mode HBA SAS SATA 8x Seagate IronWolf 3TB drives (ST3000VN007) - connected to the LSI card 2x 1TB WDC WDS100T2B0C connected to a two port QNAP nvme to PCIe adapter (moved from the QNAP NAS) 1x 1TB WDC WDS100T2B0C in a motherboard slot 1x INTEL SSDPEKNW512G8 Main Array (8x ST3000VN007, connected to the LSI Card) Parity (ST3000VN007) Parity2 (ST3000VN007) Disk 1 .. 6 ST3000VN007 (xfs) Cache_ssd (xfs) - used for iso share INTEL_SSDPEKNW512G8_PHNH9420007Q512A - 512 GB (nvme1n1) Cache_nvme (xfs) - used for docker, vm and system folders WDC_WDS100T2B0C-00PXH0_2131CQ471804 - 1 TB (nvme0n1) Cache_protected (btrfs mirror) - used to cache data to the main array - Connected via the QNAP nvme to PCIe card WDC_WDS100T2B0C-00PXH0_20427P467007 - 1 TB (nvme2n1) WDC_WDS100T2B0C-00PXH0_2052FR446602 - 1 TB (nvme3n1) I have been testing unraid before I move any data across, so have copied 4 movie files (~50-80GB in size) and I am having some major issues with the Cache_protected pool. copying from Windows PC -> Cache_protected over 1GB ethernet works fine (~280GB, 4x files). Running Mover - copying from the Array -> Cache_protected works ever time I have tested (4-5 times now) Running Mover - copying from Cache_protected -> Array after between 150-200GB nvme3n1 (second disk in the Cache_protected) disconnects from unraid and the log has the following error message Mar 27 13:45:39 Obsidian move: file: /mnt/cache_protected/movies/test.mkv Mar 27 13:53:06 Obsidian kernel: nvme nvme3: controller is down; will reset: CSTS=0xffffffff, PCI_STATUS=0xffff Mar 27 13:53:06 Obsidian kernel: nvme nvme3: Does your device have a faulty power saving mode enabled? Mar 27 13:53:06 Obsidian kernel: nvme nvme3: Try "nvme_core.default_ps_max_latency_us=0 pcie_aspm=off" and report a bug Mar 27 13:53:08 Obsidian kernel: nvme3n1: I/O Cmd(0x2) @ LBA 111992656, 1024 blocks, I/O Error (sct 0x3 / sc 0x71) Mar 27 13:53:08 Obsidian kernel: I/O error, dev nvme3n1, sector 111992656 op 0x0:(READ) flags 0x84700 phys_seg 5 prio class 2 Mar 27 13:53:08 Obsidian kernel: nvme3n1: I/O Cmd(0x2) @ LBA 111993680, 1024 blocks, I/O Error (sct 0x3 / sc 0x71) Mar 27 13:53:08 Obsidian kernel: I/O error, dev nvme3n1, sector 111993680 op 0x0:(READ) flags 0x84700 phys_seg 5 prio class 2 Mar 27 13:53:08 Obsidian kernel: nvme3n1: I/O Cmd(0x2) @ LBA 111994704, 1024 blocks, I/O Error (sct 0x3 / sc 0x71) Mar 27 13:53:08 Obsidian kernel: I/O error, dev nvme3n1, sector 111994704 op 0x0:(READ) flags 0x84700 phys_seg 5 prio class 2 Mar 27 13:53:08 Obsidian kernel: nvme3n1: I/O Cmd(0x2) @ LBA 111995728, 1024 blocks, I/O Error (sct 0x3 / sc 0x71) Mar 27 13:53:08 Obsidian kernel: I/O error, dev nvme3n1, sector 111995728 op 0x0:(READ) flags 0x84700 phys_seg 5 prio class 2 Mar 27 13:53:08 Obsidian kernel: nvme3n1: I/O Cmd(0x2) @ LBA 111996752, 1024 blocks, I/O Error (sct 0x3 / sc 0x71) Mar 27 13:53:08 Obsidian kernel: I/O error, dev nvme3n1, sector 111996752 op 0x0:(READ) flags 0x84700 phys_seg 5 prio class 2 Mar 27 13:53:08 Obsidian kernel: nvme3n1: I/O Cmd(0x2) @ LBA 111997776, 1024 blocks, I/O Error (sct 0x3 / sc 0x71) Mar 27 13:53:08 Obsidian kernel: I/O error, dev nvme3n1, sector 111997776 op 0x0:(READ) flags 0x84700 phys_seg 5 prio class 2 Mar 27 13:53:08 Obsidian kernel: nvme3n1: I/O Cmd(0x2) @ LBA 111998800, 1024 blocks, I/O Error (sct 0x3 / sc 0x71) Mar 27 13:53:08 Obsidian kernel: I/O error, dev nvme3n1, sector 111998800 op 0x0:(READ) flags 0x84700 phys_seg 24 prio class 2 Mar 27 13:53:08 Obsidian kernel: nvme3n1: I/O Cmd(0x2) @ LBA 111999824, 1024 blocks, I/O Error (sct 0x3 / sc 0x71) Mar 27 13:53:08 Obsidian kernel: I/O error, dev nvme3n1, sector 111999824 op 0x0:(READ) flags 0x80700 phys_seg 32 prio class 2 Mar 27 13:53:08 Obsidian kernel: nvme3n1: I/O Cmd(0x2) @ LBA 112000848, 1024 blocks, I/O Error (sct 0x3 / sc 0x71) Mar 27 13:53:08 Obsidian kernel: I/O error, dev nvme3n1, sector 112000848 op 0x0:(READ) flags 0x84700 phys_seg 32 prio class 2 Mar 27 13:53:08 Obsidian kernel: nvme3n1: I/O Cmd(0x2) @ LBA 112001872, 1024 blocks, I/O Error (sct 0x3 / sc 0x71) Mar 27 13:53:08 Obsidian kernel: I/O error, dev nvme3n1, sector 112001872 op 0x0:(READ) flags 0x84700 phys_seg 20 prio class 2 Mar 27 13:53:08 Obsidian kernel: nvme 0000:09:00.0: Unable to change power state from D3cold to D0, device inaccessible Mar 27 13:53:08 Obsidian kernel: nvme nvme3: Removing after probe failure status: -19 Mar 27 13:53:08 Obsidian kernel: nvme3n1: detected capacity change from 1953525168 to 0 Mar 27 13:53:08 Obsidian kernel: BTRFS error (device nvme2n1p1): bdev /dev/nvme3n1p1 errs: wr 21837240, rd 7, flush 5, corrupt 1, gen 0 Mar 27 13:53:08 Obsidian kernel: BTRFS error (device nvme2n1p1): bdev /dev/nvme3n1p1 errs: wr 21837240, rd 7, flush 5, corrupt 1, gen 0 It is strange that the nvme3n1 only disconnects / has issues when moving from Cache_protected to the Array??? any other operation works fine. I have removed the Parity2 from the array to speed up testing and it seems to copy more data before nvme3n1 disconnects, any ideas or suggestions I am going to try swapping nvme3n1 and nvme2n1 and see if the error follows the port or the drive. NOTE: I have tested the RAM using memtest overnight and all passes. NOTE: The LSI card and all eight drives have been tested in Windows using HD sentinel - Extended SMART test then 2x write - read surface scans - all working fine without error NOTE: I have no docker containers or VM's running.