A parity check (the quarterly one) started running and about 20% in I got a notification os multiple drives errors and failing.
I have no idea how this happened or what could have caused it. Everything has been running fine now and I dont see how every drive could fail like this.

I stopped the array and it looks like this:

This would be absolutely catastrophic if all the drives failed.

What can I do to try and fix this or diagnose the problem?

Rebooting gives me this

No idea what to do now or how to proceed. I don't believe 2 drives failed at the same time.

Added the diagnostics

zoidberg-diagnostics-20260128-0205.zip

Edited January 28Jan 28 by plunderisley

Diags after rebooting (at least twice) so we can't see what happened. Start the array and post new diags to see if the emulated disks are mounting.

2 hours ago, JorgeB said:

Diags after rebooting (at least twice) so we can't see what happened. Start the array and post new diags to see if the emulated disks are mounting.

zoidberg-diagnostics-20260128-1058.zipzoidberg-diagnostics-20260128-1050.zip

Also, all this happened during a parity scan. I'm leaning more towards a failure of the controller or some motherboard issue.

Both emulated disks are mounting; assuming contents look correct, you can rebuild top, or since both disks should be OK, you could also do a new config, check parity is already valid, and then run a correcting check, either way, if there are any errors post the diags before rebooting.

37 minutes ago, JorgeB said:

Both emulated disks are mounting; assuming contents look correct, you can rebuild top, or since both disks should be OK, you could also do a new config, check parity is already valid, and then run a correcting check, either way, if there are any errors post the diags before rebooting.

I'm a bit of a noob with unraid still. Assuming both disks are OK and unraid parity check just went crazy (who knows if it was a controller mobo issue, or some docker container also trying to do something the same time and messed something up as I am running it on AMD), what would be the exact steps to do the new config and get this back up online?

Tools - New Config - Keep All - Apply

Then on Main check "parity is already valid" and start the array, then run a correcting check, a few sync errors are expected

17 minutes ago, JorgeB said:

Tools - New Config - Keep All - Apply

Then on Main check "parity is already valid" and start the array, then run a correcting check, a few sync errors are expected

Okay. I got the array up. I'll stop all the docker containers to make sure nothing is running in the background.
Should I do a write correction parity-check (assuming that is what you meant) or to be safe first do a check parity before doing a write corrections one?

1 minute ago, plunderisley said:

Okay. I got the array up. I'll stop all the docker containers to make sure nothing is running in the background.
Should I do a write correction parity-check (assuming that is what you meant) or to be safe first do a check parity before doing a write corrections one?

Running a parity check cant hurt but the second you start to see errors, stop it and run a correcting parity check. The longer your array sits without valid parity, the more likely you have another failure but this time without parity to recover from.

3 minutes ago, MowMdown said:

Running a parity check cant hurt but the second you start to see errors, stop it and run a correcting parity check. The longer your array sits without valid parity, the more likely you have another failure but this time without parity to recover from.

Okay. My thought here is if this is a hardware issue (say motherboard is dying or something else), and the parity check goes crazy again, I can then replace the motherboard or psu as I'd assume that would be the cause (unless Im missing out on something)

Edited January 28Jan 28 by plunderisley

How are the drives powered? Are they all on the same single power cable from the PSU or are you using multiple cables to power the drives?

Sometimes you gotta split the drives between two separate cables even though a single cable might have enough "ends" to connect 4 or 5 drives.

Edited January 28Jan 28 by MowMdown

Just now, MowMdown said:

How are the drives powered? Are they all on the same single power cable from the PSU or are you using multiple cables to power the drives?

Multiple cables.

Like JorgeB said, without the diagnostics from before a reboot, it's hard to determine a cause.

If you haven't already, enabling the remote syslog server and pointing it to the local IP of the unraid box will allow it to save a log to the share or your choosing.

Just now, MowMdown said:

Like JorgeB said, without the diagnostics from before a reboot, it's hard to determine a cause.

If you haven't already, enabling the remote syslog server and pointing it to the local IP of the unraid box will allow it to save a log to the share or your choosing.

Oh I do have that now once the array is up.

syslog.txt

Edited January 28Jan 28 by plunderisley

Multiple drives disconnected. Could be power or controller.

You also have segfaults. Have you done memtest recently?

Also

Just now, trurl said:

Multiple drives disconnected. Could be power or controller.

You also have segfaults. Have you done memtest recently?

Also

memtest I've done 2 months or so ago when I added 2 additional sticks and it passed fine (did multiple passes over it to make sure it was good). I'm using the XMP profile that showed in the BIOS for the RAM sticks.

DIMM 0: Unknown CMK16GX4M2A2666C16, 8 GiB DDR4 @ 3200 MT/s

DIMM 1: A-DATA Technology AX4U320016G16A-BB10, 16 GiB DDR4 @ 3200 MT/s

DIMM 0: Unknown CMK16GX4M2A2666C16, 8 GiB DDR4 @ 3200 MT/s

DIMM 1: A-DATA Technology AX4U320016G16A-BB10, 16 GiB DDR4 @ 3200 MT/s

I remember I did change the c-state in the BIOS back in May or June (?) when I was first setting up the server as I had problems with it. Since then it has been running fine. Unless somewhere it lost the setting or it was never configured correctly.

6 hours ago, JorgeB said:

Diags after rebooting

If it happens again get diagnostics before rebooting. That will give us more information to associate with the syslog.

I only see one controller in your diagnostics.

02:00.1 SATA controller [0106]: Advanced Micro Devices, Inc. [AMD] 400 Series Chipset SATA Controller [1022:43c8] (rev 01)
	Subsystem: ASMedia Technology Inc. Device [1b21:1062]
	Kernel driver in use: ahci

I assume this is on the motherboard.

1 minute ago, trurl said:

If it happens again get diagnostics before rebooting. That will give us more information to associate with the syslog.

I only see one controller in your diagnostics.

02:00.1 SATA controller [0106]: Advanced Micro Devices, Inc. [AMD] 400 Series Chipset SATA Controller [1022:43c8] (rev 01)
	Subsystem: ASMedia Technology Inc. Device [1b21:1062]
	Kernel driver in use: ahci

I assume this is on the motherboard.

Okay. If it happens again, I'll do a diagnostics before I reboot. (it was a panic shutdown as I've never seen that before)

Right now its 1.5hrs into a parity-check (5.7%) and 7 sync errors found (in the beginning of the scan).

The SATA controller is on the motherboard.

The parity check completed.

Parity Check Tuning Manual Non-Correcting Parity-Check finished (1375 errors) Elapsed Time 1 day, 8 hr, 56 min, 9 sec, Runtime 1 day, 8 hr, 56 min, 9 sec, Increments 1, Average Speed 168.7 MB/s

I'll run a correcting parity check now and fix those errors.

It seems the issue was probably caused by either a controller issue or some issue with AMD Ryzen and unRAID (there was some issues iirc with those).

50 minutes ago, plunderisley said:

I'll run a correcting parity check now and fix those errors.

Post Diagnostics before and after

Just now, trurl said:

Post Diagnostics before and after

Shoot, already started the write correction. I did save a diagnostic just now (7 sync errors corrected, 845GB/4.2% completed) and I'll do one once it'll be done sometime tomorrow night/next morning.

If you haven't rebooted since that first parity check that is OK. Just wanted to see the before and after in the syslog.

Attached are:

diagnostics that I grabbed when I started the write correction (after the parity scan without write correction and no reboot).zoidberg-diagnostics-20260130-0006.zip

the diagnostics after the write correction finished.zoidberg-diagnostics-20260131-1312.zip

I shutdown the server, opened it up to check and re-seat the SATA cables, and booted it up. I got a unclean shutdown detected (no idea why, as I pressed the shutdown button on the main web ui and it did say it shutdown). I attached those diagnostics too.
zoidberg-diagnostics-20260131-1404.zip

I'm not sure if I need to do another whole parity scan, as I already did 2 and the previous 1375 errors were fixed.

That looks OK for those 2 parity checks.

Then the unclean shutdown, and you cancelled its parity check. Usually if there are any parity errors from unclean shutdown they will show up early in the scan.

Do you have syslog server logs from the shutdown?

There is a "sticky" thread about unclean shutdowns pinned near the top of this same General Support subforum.

7 minutes ago, trurl said:

That looks OK for those 2 parity checks.

Then the unclean shutdown, and you cancelled its parity check. Usually if there are any parity errors from unclean shutdown they will show up early in the scan.

Do you have syslog server logs from the shutdown?

There is a "sticky" thread about unclean shutdowns pinned near the top of this same General Support subforum.

Yup.

syslog.log

I read the first post on that unclean shutdown and changed the disk settings from 90 to 420sec and VM to 300sec.
It does make sense, given I have a ton of docker containers running (though when I did run the scan, I only had 2 containers up and rest were stopped as not to interfere with the parity check)

Edited January 31Jan 31 by plunderisley