Hi, I've recently upgraded my Unraid server to a new Intel Board+CPU off of my old xeon processors with ecc ram.

(This is running 6.12.6)

Ever since I did that, I've been getting an increasing amount of errors on each drive. enough so that at this point over the past week I think each hard drive has failed at least once and needed to be rebuilt.

 

I just recently had a failure during a rebuild and it officially put me over the "too many failures total" at this point I'm stopping everything, mounting everything one at a time and pulling the data off to an external hard drive. (It'll take a while cause its 25TB but I gotta do what I gotta do)

 

I'm really not sure what I need to do to try to solve this issue, this is a really weird issue and I can't pinpoint whats particularly going on here.

 

On average its not really many errors causing the drives to drop out of the raid either. The drive i was rebuilding this time had 1 error, and it caused it to drop out. I'm getting extremely concerned with the data.

 

Any advice?

Edited February 18, 20242 yr by Killklli
Adding version

Really wish you had asked before doing anything. 

 

Bad connections are much more common than bad disks.

 

Can you put all the disks back in your server and

 

Attach Diagnostics to your NEXT post in this thread. 

The lucky thing is I don't *really* have data loss right now. I can mount each drive one at a time and ship it off. As is originally as it was the parity drive and one other drive failing (over and over) I thought it was just two bad drives as they were bought around the same time, then during the rebuild it decided to start crippling itself. So thus far data wise I've only *really* lost the parity drive and a new drive I got to replace what I assumed was a failing drive. I still have the data, it'll just be a pain to restore slowly, but I'll take that.

 

As a note, and I'll be attaching it here in a sec, I did reboot in between the dump, so the syslog does not match, so I'll attach that as I keep that shipping off.

tower-diagnostics-20240218-1831.zip

 

syslog-127.0.0.1.zip

Edited February 19, 20242 yr by Killklli
attaching syslog

SMART for all disks looks fine. You haven't completed extended self-test on any but no reason to think they wouldn't pass.

 

RAID controllers are not recommended for many reasons, and this may be one of them:

Feb 14 18:01:31 Tower kernel: sd 1:1:17:0: [sdc] tag#643 access beyond end of device

You are getting these on multiple disks.

 

Syslog had lots of macvlan traces, don't know if you fixed that yet. See release notes:

 

https://docs.unraid.net/unraid-os/release-notes/6.12.6/#call-traces-and-crashes-related-to-macvlan

 

Also some out-of-date plugins.

 

Filesystem corruption on disk5. Not clear if that is still the case, or if emulated disk5 is mountable, since these diagnostics were taken without the array started.

 

Can you start the array and post new diagnostics?

 

 

I'll see what I can do about getting the array started. The filesystem corruption happened during a rebuild, so thats somewhat correct at this point. I'll work on getting a diagnostic.

But as a question, could this be related?

```Feb 17 12:15:16 Tower kernel: pcieport 0000:00:1d.7: AER: Multiple Corrected error received: 0000:06:00.0
Feb 17 12:15:16 Tower kernel: atlantic 0000:06:00.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, (Receiver ID)
Feb 17 12:15:16 Tower kernel: atlantic 0000:06:00.0:   device [1d6a:00b1] error status/mask=00000081/0000a000
Feb 17 12:15:16 Tower kernel: atlantic 0000:06:00.0:    [ 0] RxErr                 
Feb 17 12:15:16 Tower kernel: atlantic 0000:06:00.0:    [ 7] BadDLLP              ```

 

I see that fairly consistently since the hardware change.

You can try this first for the PCIe errors:

https://forums.unraid.net/topic/118286-nvme-drives-throwing-errors-filling-logs-instantly-how-to-resolve/?do=findComment&comment=1165009

 

So I just spent some time troubleshooting an error "GPT header corruption has been detected" thats occurring on the board with the raid card installed. I think its a misdirection on the actual issue for that, but does anyone know if that usually indicates a raid card issue?

 

(I'm still working on getting the follow up diagnostics, its just going to take some time as i have to re-export some data to get the array back online)

 

Just to be clear, I'm not using the RAID card for anything but passthrough. I'm using it so I can connect to a DAS backplane.

Edited February 20, 20242 yr by Killklli
Explanation on raid card

8 hours ago, Killklli said:

but does anyone know if that usually indicates a raid card issue?

It can be an issue with RAID controllers, since they not always passthrough the complete partition, and those Adaptec controllers specifically have been known to change/wipe disks MBR sometimes.

Out of precaution I've ordered a new card. This time I've gone with `SAS 9200-8E` thank you for all the support thus far. I'll be posting some updates in here as I can get data moved off. (Only goes so fast with that much data). I'm focusing on the drives that I know I need to get off, then from there I think theres a setting I can do to force unraid to just remove the missing drives in particular so I can re-add those particular drives outside of it.

I might still out of safety move everything off and just rebuild the whole cluster.

Note that the drives may not mount on the new controller without a rebuild, if the partition changes, so keep the old controller intact for now.

Will do, being extremely careful on the transfer on this.

So with no real data loss (but a little bit of corruption I have some UDMA CRC errors but not a massive amount, <5 on actual drives and then the parity drive has like 13)

I'm rebuilding both parity drives after replacing the raid card and that setting change. Thus far I have no further errors. If in a few days or so I don't have any errors I'll mark that as the solve.

I just had an explosion of errors and have no idea why, a lot more than befortower-diagnostics-20240225-2055.zipe

Problems communicating with multiple drives. The new controller doesn't appear in lspci anymore.

 

Feb 25 17:54:01 Tower kernel: mpt2sas_cm0: SAS host is non-operational !!!!

 

Make sure the controller is well seated and sufficiently cooled, you can also try a different PCIe slot.