JorgeB

Very far from the possible speed, we'd need the diags to see what happened, maybe you paused the check?

It was successful, no read errors on other disks during the rebuild.

It's necessary in the sense that btrfs doesn't auto trim.

Also check this for better pool monitoring.

First time Adata SSD dropped offline: Dec 3 12:06:09 TOWER kernel: ata4.00: qc timeout (cmd 0xec) Dec 3 12:06:09 TOWER kernel: ata4.00: failed to IDENTIFY (I/O error, err_mask=0x4) Dec 3 12:06:09 TOWER kernel: ata4.00: revalidation failed (errno=-5) Dec 3 12:06:09 TOWER kernel: ata4.00: disabled Second time it dropped again: Dec 4 09:03:48 TOWER kernel: ata4.00: link online but device misclassified Dec 4 09:04:18 TOWER kernel: ata4.00: qc timeout (cmd 0xec) Dec 4 09:04:18 TOWER kernel: ata4.00: failed to IDENTIFY (I/O error, err_mask=0x4) Dec 4 09:04:18 TOWER kernel: ata4.00: revalidation failed (errno=-5) Dec 4 09:04:18 TOWER kernel: ata4.00: disabled Adata devices don't have a very good reputation, but it can also be a connection issue, suggest you try again with the SSD in a different slot/cable/port, if it happens again replace with a new device, preferably from a different brand

Try re-creating the flash drive manually: If it's not a new Unraid install backup flash drive first. Open a command prompt window as administrator, then type, in this order: -diskpart -list disk -select disk x (x=your flash drive) -clean -create partition primary -format fs=fat32 label=UNRAID quick -active -assign -exit -close cmd window -download Unraid release zip and extract to flash (or restore from backup) -execute make_bootable as admin

You need to run xfs_repair without -n, or nothing will be done.

The previous check was also correct, do another one, if errors are still zero then most likely parity was just invalid during first check.

You should be able to hit 10GbE or close with a single iperf thread, I get around 9Gbits, and if not a single transfer will never be that quick, it's usually not faster than a single iperf thread, unless you plan on doing multiple simultaneous transfers. D:\temp\iperf>iperf3 -c 10.0.0.7 Connecting to host 10.0.0.7, port 5201 [ 4] local 10.0.0.50 port 59456 connected to 10.0.0.7 port 5201 [ ID] Interval Transfer Bandwidth [ 4] 0.00-1.00 sec 1.03 GBytes 8.84 Gbits/sec [ 4] 1.00-2.00 sec 1.05 GBytes 9.03 Gbits/sec [ 4] 2.00-3.00 sec 1.05 GBytes 9.05 Gbits/sec [ 4] 3.00-4.00 sec 1.04 GBytes 8.94 Gbits/sec [ 4] 4.00-5.00 sec 1.04 GBytes 8.93 Gbits/sec [ 4] 5.00-6.00 sec 1.04 GBytes 8.97 Gbits/sec [ 4] 6.00-7.00 sec 1.01 GBytes 8.66 Gbits/sec [ 4] 7.00-8.00 sec 1.05 GBytes 9.06 Gbits/sec [ 4] 8.00-9.00 sec 1.05 GBytes 9.04 Gbits/sec [ 4] 9.00-10.00 sec 1.04 GBytes 8.91 Gbits/sec - - - - - - - - - - - - - - - - - - - - - - - - - [ ID] Interval Transfer Bandwidth [ 4] 0.00-10.00 sec 10.4 GBytes 8.94 Gbits/sec sender [ 4] 0.00-10.00 sec 10.4 GBytes 8.94 Gbits/sec receiver Reversed (receive from server): D:\temp\iperf>iperf3 -c 10.0.0.7 -R Connecting to host 10.0.0.7, port 5201 Reverse mode, remote host 10.0.0.7 is sending [ 4] local 10.0.0.50 port 59467 connected to 10.0.0.7 port 5201 [ ID] Interval Transfer Bandwidth [ 4] 0.00-1.00 sec 1.09 GBytes 9.36 Gbits/sec [ 4] 1.00-2.00 sec 1.09 GBytes 9.38 Gbits/sec [ 4] 2.00-3.00 sec 1.09 GBytes 9.36 Gbits/sec [ 4] 3.00-4.00 sec 1.12 GBytes 9.59 Gbits/sec [ 4] 4.00-5.00 sec 1.08 GBytes 9.29 Gbits/sec [ 4] 5.00-6.00 sec 1.11 GBytes 9.58 Gbits/sec [ 4] 6.00-7.00 sec 1.10 GBytes 9.48 Gbits/sec [ 4] 7.00-8.00 sec 1.10 GBytes 9.42 Gbits/sec [ 4] 8.00-9.00 sec 1.10 GBytes 9.43 Gbits/sec [ 4] 9.00-10.00 sec 1.10 GBytes 9.47 Gbits/sec - - - - - - - - - - - - - - - - - - - - - - - - - [ ID] Interval Transfer Bandwidth Retr [ 4] 0.00-10.00 sec 11.0 GBytes 9.44 Gbits/sec 1 sender [ 4] 0.00-10.00 sec 11.0 GBytes 9.44 Gbits/sec receiver

Unless something is writing to an array device using sdX1 instead of the md device, for example if you use dd to write to /dev/sdc1 instead of /dev/md1 (disk1), that would make parity out of sync.

Unfortunately no.

Still constant ATA errors, there will always be issues until those are gone.

You want disk drives to be under 40C, 45C tops.

Correct. Yes, though hardware can fail at any time, also weird that that sync errors started on the same sectors on both runs, but if I understood correctly the total number was very different correct? Dec 1 03:00:01 illmatic kernel: md: using 1536k window, over a total of 9766436812 blocks. Dec 1 03:00:19 illmatic kernel: md: recovery thread: P corrected, sector=4194400 Dec 1 03:00:19 illmatic kernel: md: recovery thread: P corrected, sector=4194408 Dec 1 03:00:19 illmatic kernel: md: recovery thread: P corrected, sector=4194416 Dec 1 03:00:19 illmatic kernel: md: recovery thread: P corrected, sector=4194424 Dec 1 03:00:19 illmatic kernel: md: recovery thread: P corrected, sector=4194432 Dec 1 03:00:19 illmatic kernel: md: recovery thread: P corrected, sector=4194440 Dec 3 09:30:01 illmatic kernel: md: using 1536k window, over a total of 9766436812 blocks. Dec 3 09:30:19 illmatic kernel: md: recovery thread: P corrected, sector=4194400 Dec 3 09:30:19 illmatic kernel: md: recovery thread: P corrected, sector=4194408 Dec 3 09:30:19 illmatic kernel: md: recovery thread: P corrected, sector=4194416 Dec 3 09:30:19 illmatic kernel: md: recovery thread: P corrected, sector=4194424 Dec 3 09:30:19 illmatic kernel: md: recovery thread: P corrected, sector=4194432 Dec 3 09:30:19 illmatic kernel: md: recovery thread: P corrected, sector=4194440 All I can say is that I don't see how these sync errors can be a bug or software, only a hardware issue makes sense.

It's not, and unlikely to be a disk problem, IIRC those servers can use ECC or non ECC, if using non ECC run a memtest, if using ECC I would guess the board/controller as the most likely culprits.

In that case it can be the device or the board/adapter.

OK, IMHO these are the best 2 options: 1) let parity2 sync finish, it will take more time than normal but as long as there are no read errors on another disk at the same time as the ones on disk3 parity1 can correct the read errors on disk3, and parity2 will be valid once synced, then replace disk3 when possible. 2) use parity2 disk to replace disk3 now, to do that unassign parity2, start array, stop array, assign it as disk3, start array to rebuild. P.S. disk2 also shows some read errors, but they were some time ago and it should be fine for now, but it's a good idea to monitor the raw_read_error_rate attribute, if it keeps increasing it will likely fail soon.

Yes, at least with LSI firmware.

Still the constant ATA errors on ATA5/6, might be the ports, swap those disks with other two and see if the errors remain with the same ports. Dec 4 00:12:48 Tower kernel: ata5: SATA link up 1.5 Gbps (SStatus 113 SControl 310) Dec 4 00:12:48 Tower kernel: ata5.00: configured for UDMA/33 Dec 4 00:12:48 Tower kernel: ata5: EH complete Dec 4 00:12:48 Tower kernel: ata5.00: exception Emask 0x10 SAct 0x0 SErr 0x10200 action 0xe frozen Dec 4 00:12:48 Tower kernel: ata5.00: irq_stat 0x00400000, PHY RDY changed Dec 4 00:12:48 Tower kernel: ata5: SError: { Persist PHYRdyChg } Dec 4 00:12:48 Tower kernel: ata5.00: failed command: READ DMA EXT Dec 4 00:12:48 Tower kernel: ata5.00: cmd 25/00:80:80:62:00/00:01:00:00:00/e0 tag 24 dma 196608 in Dec 4 00:12:48 Tower kernel: res 50/00:00:7f:62:00/00:00:00:00:00/e0 Emask 0x10 (ATA bus error) Dec 4 00:12:48 Tower kernel: ata5.00: status: { DRDY } Dec 4 00:12:48 Tower kernel: ata5: hard resetting link Dec 4 00:12:48 Tower kernel: ata6.00: exception Emask 0x10 SAct 0x0 SErr 0x10200 action 0xe frozen Dec 4 00:12:48 Tower kernel: ata6.00: irq_stat 0x00400000, PHY RDY changed Dec 4 00:12:48 Tower kernel: ata6: SError: { Persist PHYRdyChg } Dec 4 00:12:48 Tower kernel: ata6.00: failed command: READ DMA EXT Dec 4 00:12:48 Tower kernel: ata6.00: cmd 25/00:40:00:64:00/00:05:00:00:00/e0 tag 24 dma 688128 in Dec 4 00:12:48 Tower kernel: res 50/00:00:4f:0e:00/00:00:80:01:00/e0 Emask 0x10 (ATA bus error) Dec 4 00:12:48 Tower kernel: ata6.00: status: { DRDY } Dec 4 00:12:48 Tower kernel: ata6: hard resetting link

It might work, that's not an LSI firmware, so not sure what kind it is, you'd want the LSI IT firmware to be used.

Array drives are always limited by that single device speed for reads and the slowest device in the array for writes (using turbo write), you can add SSDs to the array but only reads will be fast, assuming parity is a hard drive, also array devices can't be trimmed for now, so write performance might degrade with use.

Yes, p20.00.07.00 IT, yes, SFF-8087 to SATA forward breakout cables.

You can use an homemade DAS, e.g.:

Please post the diagnostics

Disk3 and to a lesser extent disk2 are suffering from apparent connection issues, start by replacing the cables.