JorgeB

All SSDs, my cache is NVMe but earlier I tested on a regular SSD and the difference was similar, though it can vary with brand/model.

Opps, misread as 8125, not sure about this one, did you try beta25?

They are already available in the latest beta (v6.9-beta25), and should be on any future ones, assuming they compile.

SMART attributes look OK but not being able to complete a SMART test is a red flag.

Just wan slow or LAN transfers also? Did you run an iperf test?

New install won't permit rebuilding a failed drive (without going through the invalid slot procedure), you should use the old install and just do a standard disk replacement.

Yep, you can rebuild it after.

But there might be a serious issue with the drive, in that case you can also try ddrescue first.

See here for some recovery options.

Still many errors, if you replaced both cables (power + SATA) disk is likely dying, if you didn't do it now.

Not that, but SMART might be disable when not on the array, what's the output of: smartctl -x /dev/sdX

No. Correct

I understand the desire to do that, but allow me to make a case for not going back: -yes, raid5/6 is mostly experimental and not in use by many, but with the old way of doing it Unraid also reports the wrong free space for different size devices using raid1, e.g., a 120 + 240GB pool and it will report 180GB usable, and this keeps coming up in the forums when the users run out space despite what the GUI shows. -if you leave it as is these issues will never get fixed. -if you make the change and start reporting the same way df does: both used and free space stats would be correctly reported for raid1 pools with different size devices both used and free space stats would be correctly reported for raid5/6 pools df is widely used for scripts and such with btrfs so any future bugs should be quickly found and fixed AFAIK there's currently only one situation where df doesn't report the correct stats, a raid1 pool with an odd number of devices, I already reported this to the btrfs mailing list and it was already confirmed by another user, so I expect it will be fixed soon, I mean in the near future

Upgraded one of my main servers to b25 and re-formatted the cache device with the new alignment and consider this issue fixed for me, I have a single btrfs cache device with 3 Windows VMs always running, as well as the docker image and appdata: v6.8 was writing about 3TB a day v6.8 with space cache v2 brought it down to a more reasonable 700GB a day v6.9-beta25 with the new alignment brought it down even further to 191.87GB in the last 24 hours While 192GB a day is still considerable and I know that if I went with xfs it would less, as the previously linked study found I believe we must accept btrfs will always have higher write amplification due to being a COW filesystem, and while I don't need a pool for this I rely on btrfs snapshots and send/receive for my backups, so I can live with these daily writes, just couldn't with 3TB a day, that was just crazy.

Problems started with an IOMMU related error: Jul 24 03:46:50 Unraid kernel: nvme 0000:01:00.0: AMD-Vi: Event logged [IO_PAGE_FAULT domain=0x0001 address=0xbe75f000 flags=0x0000] That resulted in the device timing out: Jul 24 03:47:22 Unraid kernel: nvme nvme0: I/O 702 QID 4 timeout, aborting Jul 24 03:47:30 Unraid kernel: nvme nvme0: I/O 640 QID 4 timeout, aborting Jul 24 03:47:30 Unraid kernel: nvme nvme0: I/O 911 QID 6 timeout, aborting Jul 24 03:47:32 Unraid kernel: nvme nvme0: I/O 181 QID 8 timeout, aborting Jul 24 03:47:51 Unraid kernel: nvme nvme0: I/O 5 QID 0 timeout, reset controller Jul 24 03:47:53 Unraid kernel: nvme nvme0: I/O 702 QID 4 timeout, reset controller Jul 24 03:49:00 Unraid kernel: nvme nvme0: Device not ready; aborting reset, CSTS=0x1 Jul 24 03:49:00 Unraid kernel: nvme nvme0: Abort status: 0x371 Jul 24 03:49:05 Unraid kernel: nvme nvme0: Device not ready; aborting reset, CSTS=0x1 Jul 24 03:49:05 Unraid kernel: nvme nvme0: Removing after probe failure status: -19 Jul 24 03:49:11 Unraid kernel: nvme nvme0: Device not ready; aborting reset, CSTS=0x1 If you don't need it you can try disabling IOMMU, BIOS update might also help.

Please post SMART report.

There are constant ATA errors on disk3, there's not even a full SMART report, check/replace cables and post new diags.

This would never work since it's missing the partition, but please post the diagnostics: Tools -> Diagnostics

Initial transfer should be done without using cache, you can enable turbo write for increased performance, but 40MB/s should be possible even without it.

Replace cables on the SSD, Samsung SSDs can be very picky with cable quality.

You can swap that disk with one using the onboard controller before rebuilding to see if it doesn't happen again, but either way you should replace that controller, they are not recommended for some time due to various known issues.

You can check with "lsscsi -v", you'll get something like this: lsscsi -v [0:0:0:0] disk Kingston DataTraveler 2.0 PMAP /dev/sda dir: /sys/bus/scsi/devices/0:0:0:0 [/sys/devices/pci0000:00/0000:00:14.0/usb1/1-2/1-2:1.0/host0/target0:0:0/0:0:0:0] [1:0:0:0] disk ATA WDC WD100EMAZ-00 0A83 /dev/sdb dir: /sys/bus/scsi/devices/1:0:0:0 [/sys/devices/pci0000:00/0000:00:17.0/ata1/host1/target1:0:0/1:0:0:0] [2:0:0:0] disk ATA WDC WD60EZRZ-00G 0A80 /dev/sdc dir: /sys/bus/scsi/devices/2:0:0:0 [/sys/devices/pci0000:00/0000:00:17.0/ata2/host2/target2:0:0/2:0:0:0] [3:0:0:0] disk ATA WDC WD60EZRX-00M 0A80 /dev/sdd dir: /sys/bus/scsi/devices/3:0:0:0 [/sys/devices/pci0000:00/0000:00:17.0/ata3/host3/target3:0:0/3:0:0:0] [4:0:0:0] disk ATA WDC WD60EFRX-68M 0A82 /dev/sde dir: /sys/bus/scsi/devices/4:0:0:0 [/sys/devices/pci0000:00/0000:00:17.0/ata4/host4/target4:0:0/4:0:0:0] [9:0:0:0] disk ATA WDC WDS500G1R0B 00WR /dev/sdj dir: /sys/bus/scsi/devices/9:0:0:0 [/sys/devices/pci0000:00/0000:00:01.0/0000:01:00.0/ata9/host9/target9:0:0/9:0:0:0] [10:0:0:0] disk ATA WDC WDS500G1R0B 00WR /dev/sdk dir: /sys/bus/scsi/devices/10:0:0:0 [/sys/devices/pci0000:00/0000:00:01.0/0000:01:00.0/ata10/host10/target10:0:0/10:0:0:0] [11:0:0:0] disk ATA WDC WDS500G1R0B 00WR /dev/sdl dir: /sys/bus/scsi/devices/11:0:0:0 [/sys/devices/pci0000:00/0000:00:01.0/0000:01:00.0/ata11/host11/target11:0:0/11:0:0:0] [12:0:0:0] disk ATA WDC WDS500G1R0B 00WR /dev/sdm dir: /sys/bus/scsi/devices/12:0:0:0 [/sys/devices/pci0000:00/0000:00:01.0/0000:01:00.0/ata12/host12/target12:0:0/12:0:0:0] [14:0:0:0] disk ATA ST1000LM014-1EJ1 HPP1 /dev/sdn dir: /sys/bus/scsi/devices/14:0:0:0 [/sys/devices/pci0000:00/0000:00:1c.0/0000:04:00.0/ata14/host14/target14:0:0/14:0:0:0] [16:0:0:0] disk ATA TOSHIBA MQ01ABD1 5J /dev/sdo dir: /sys/bus/scsi/devices/16:0:0:0 [/sys/devices/pci0000:00/0000:00:1c.0/0000:04:00.0/ata16/host16/target16:0:0/16:0:0:0] [17:0:0:0] disk ATA TOSHIBA MQ01ABD1 2C /dev/sdp dir: /sys/bus/scsi/devices/17:0:0:0 [/sys/devices/pci0000:00/0000:00:1c.0/0000:04:00.0/ata17/host17/target17:0:0/17:0:0:0] [18:0:0:0] disk ATA TOSHIBA MQ01ABD1 4M /dev/sdq dir: /sys/bus/scsi/devices/18:0:0:0 [/sys/devices/pci0000:00/0000:00:1c.0/0000:04:00.0/ata18/host18/target18:0:0/18:0:0:0] [N:0:8215:1] dsk/nvm WDS500G3X0C-00SJG0__1 /dev/nvme0n1 dir: /sys/class/nvme/nvme0/nvme0n1 [/sys/devices/pci0000:00/0000:00:01.1/0000:02:00.0/nvme/nvme0/nvme0n1] [N:1:0:1] dsk/nvm TOSHIBA-RD400__1 /dev/nvme1n1 dir: /sys/class/nvme/nvme1/nvme1n1 [/sys/devices/pci0000:00/0000:00:01.2/0000:03:00.0/nvme/nvme1/nvme1n1] Look at the PCI devices, in this case there are drives on devices 00:17.0, 01:00.0 and 04:00.0, then type lspci to identify them: lspci 00:00.0 Host bridge: Intel Corporation Xeon E3-1200 v5/E3-1500 v5/6th Gen Core Processor Host Bridge/DRAM Registers (rev 07) 00:01.0 PCI bridge: Intel Corporation Xeon E3-1200 v5/E3-1500 v5/6th Gen Core Processor PCIe Controller (x16) (rev 07) 00:01.1 PCI bridge: Intel Corporation Xeon E3-1200 v5/E3-1500 v5/6th Gen Core Processor PCIe Controller (x8) (rev 07) 00:01.2 PCI bridge: Intel Corporation Xeon E3-1200 v5/E3-1500 v5/6th Gen Core Processor PCIe Controller (x4) (rev 07) 00:13.0 Non-VGA unclassified device: Intel Corporation 100 Series/C230 Series Chipset Family Integrated Sensor Hub (rev 31) 00:14.0 USB controller: Intel Corporation 100 Series/C230 Series Chipset Family USB 3.0 xHCI Controller (rev 31) 00:14.2 Signal processing controller: Intel Corporation 100 Series/C230 Series Chipset Family Thermal Subsystem (rev 31) 00:16.0 Communication controller: Intel Corporation 100 Series/C230 Series Chipset Family MEI Controller #1 (rev 31) 00:16.1 Communication controller: Intel Corporation 100 Series/C230 Series Chipset Family MEI Controller #2 (rev 31) 00:17.0 SATA controller: Intel Corporation Q170/Q150/B150/H170/H110/Z170/CM236 Chipset SATA Controller [AHCI Mode] (rev 31) 00:1c.0 PCI bridge: Intel Corporation 100 Series/C230 Series Chipset Family PCI Express Root Port #1 (rev f1) 00:1c.4 PCI bridge: Intel Corporation 100 Series/C230 Series Chipset Family PCI Express Root Port #5 (rev f1) 00:1d.0 PCI bridge: Intel Corporation 100 Series/C230 Series Chipset Family PCI Express Root Port #9 (rev f1) 00:1d.1 PCI bridge: Intel Corporation 100 Series/C230 Series Chipset Family PCI Express Root Port #10 (rev f1) 00:1d.2 PCI bridge: Intel Corporation 100 Series/C230 Series Chipset Family PCI Express Root Port #11 (rev f1) 00:1f.0 ISA bridge: Intel Corporation C236 Chipset LPC/eSPI Controller (rev 31) 00:1f.2 Memory controller: Intel Corporation 100 Series/C230 Series Chipset Family Power Management Controller (rev 31) 00:1f.4 SMBus: Intel Corporation 100 Series/C230 Series Chipset Family SMBus (rev 31) 01:00.0 SATA controller: JMicron Technology Corp. Device 0585 02:00.0 Non-Volatile memory controller: Sandisk Corp WD Black 2018/PC SN720 NVMe SSD 03:00.0 Non-Volatile memory controller: OCZ Technology Group, Inc. RD400/400A SSD (rev 01) 04:00.0 SATA controller: JMicron Technology Corp. Device 0585 05:00.0 Ethernet controller: Mellanox Technologies MT27500 Family [ConnectX-3] 06:00.0 Ethernet controller: Intel Corporation I210 Gigabit Network Connection (rev 03) 07:00.0 Ethernet controller: Intel Corporation I210 Gigabit Network Connection (rev 03) 08:00.0 PCI bridge: ASPEED Technology, Inc. AST1150 PCI-to-PCI Bridge (rev 03) 09:00.0 VGA compatible controller: ASPEED Technology, Inc. ASPEED Graphics Family (rev 30) 00:17.0 is the onboard Intel SATA controller, 01:00.0 and 04:00.0 are JMB controllers.

Did you upgrade to v6.8 recently? Parity check can be slower than before especially with larger arrays.

It's the docker image, Settings -> Docker to set the path, you can delete it and recreate on cache or UD.

There are a few writes going on to disk8, not much but it can still make a difference, current speed is 85MB/s, make sure it stops writing to the array and see if it improves, other than that nothing jumps out.