Woke up to a surprise in that one of the disks on my array had errors. I tried to take disk2 out of the array, restart and re-add but immediately got an error so removed it. Then, as its first day of the month, the parity check started and I noticed the logs filled with read errors for disk3.

I've stopped the parity check and have ordered replacement drives. While /dev/sdc (formerly disk2) shows a smart error, I can't see anything wrong with disk3?

And for rebuilding the array, given one disk is completely out of action, and the other is full of read errors, any recommended steps for how to start the rebuild process? Should I attempt a rebuild by replacing disk2 and then if that succeeds, replace disk3?

EDIT: I can mount disk2 (now of out of array) from unassigned devices. I guess the issue is how to now rebuild the array with replacement disks? Can I try to re-add disk2 to the array without reformatting it?

Diagnostics attached. Any help very appreciated.

amiga-diagnostics-20250701-0658.zip

Edited July 1, 20251 yr by paulmorabi

Look more like a power/connection issue, check/replace cables for disk3 and post new diags after array start

ok, unplugged and re-connected everything. Diagnostics attached. No errors so far but the disks are idle and I didn't start a check.

amiga-diagnostics-20250701-1431.zip

Edited July 1, 20251 yr by paulmorabi

Looks good so far, try rebuilding again.

I've rebuilt - unplugged everything and reseated/connected. I also got two new drives. Started a pre-clear and it seems to work fine for a couple of hours and then overwhelmed with read errors from all drives. I am thinking it is possibly the LSI card itself. I got it from China originally and its 1.5 years old.

Could be, post current diags to see if there's more info.

3 hours ago, JorgeB said:

Could be, post current diags to see if there's more info.

here you go. Thanks.

I had the LSI card connected to a PCI riser cable. As part of the rebuild, I removed that and connected it directly to the main board. Current pre-clear is still running and no errors so far. About 2/3 way through which is as far as its gotten before. If I get errors again, then I'll try replacing the card. The strange thing is the errors after after a period of time, not immediately and possibly related to high array activity (during parity check/rebuild).

amiga-diagnostics-20250705-0544.zip

Don't see the disk errors on the log, even on syslog-previous, maybe you rebooted or were not logged because of the log spam about this:

Jul 4 10:19:00 Amiga kernel: pcieport 0000:00:01.0: AER: Corrected error message received from 0000:01:00.0

Jul 4 10:19:00 Amiga kernel: mpt3sas 0000:01:00.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, (Receiver ID)

Jul 4 10:19:00 Amiga kernel: mpt3sas 0000:01:00.0: device [1000:0097] error status/mask=00000001/00002000

Jul 4 10:19:00 Amiga kernel: mpt3sas 0000:01:00.0: [ 0] RxErr (First)

This is about the HBA, using a different PCIe slot may help get rid of them, or https://forums.unraid.net/topic/118286-nvme-drives-throwing-errors-filling-logs-instantly-how-to-resolve/?do=findComment&comment=1165009

49 minutes ago, JorgeB said:

Don't see the disk errors on the log, even on syslog-previous, maybe you rebooted or were not logged because of the log spam about this:

Jul 4 10:19:00 Amiga kernel: pcieport 0000:00:01.0: AER: Corrected error message received from 0000:01:00.0

Jul 4 10:19:00 Amiga kernel: mpt3sas 0000:01:00.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, (Receiver ID)

Jul 4 10:19:00 Amiga kernel: mpt3sas 0000:01:00.0: device [1000:0097] error status/mask=00000001/00002000

Jul 4 10:19:00 Amiga kernel: mpt3sas 0000:01:00.0: [ 0] RxErr (First)

This is about the HBA, using a different PCIe slot may help get rid of them, or https://forums.unraid.net/topic/118286-nvme-drives-throwing-errors-filling-logs-instantly-how-to-resolve/?do=findComment&comment=1165009

I only have one pci slot but I have removed the riser cable so maybe that could fix it? I can't see the same error in the logs. Thus far.

The read errors were in the millions and I think crashed Unraid on one of the rebuild attempts. The snippet you posted above is from the rebuild attempt before I removed the riser cable and connected the HBA card directly to the pci slot. That too started to show a lot of read errors but I stopped it, removed the riser cable and started again. Thus far, ~4 hours to go, its ok and no errors on rebuilding.

Roger, if there are new errors, post new diags.