Two Drives Disabled During Parity Check for the 2nd Time

[RoachBot]

This is the second time I've had 2 drives disabled at once. Both times occurred during my monthly scheduled Parity Check, and both involved a Parity drive. This post is about the first time.

 

1. Is my data already corrupt since there were write errors or were those ultimately corrected?
2. Should I use 2 drives I have on hand and keep the old ones in case there is a 3rd disk failure during rebuild?
3. Any ideas why I keep getting 2 drives disabled at once?

obelisk-diagnostics-20230602-0443.zip

Edited June 17, 2023 by RoachBot

[JorgeB]

These look like a power/connection problem, usually cables, sometimes PSU.

[RoachBot]

Thanks, for the reply. I took a pause on fixing my server due to stress, but I'm back at it. It seems the PSU may have been the issue, but I'm running into issues with restoring the array.

Shorter Summary

I rebuilt disk4 separately (see longer summary) and had no errors or indication of unmountable. Next, I synced parity1 and had no errors. I start the array normally (not maintenance) and disk4 was unmountable. I use `xfs_repair` and now every file/directory is in "lost+found". I have the original disk4 and it mounts fine using Unassigned Devices.

 

What is the best way to proceed if my main concern is losing data or corrupted files? I could try copying original disk4 contents to new disk4, but I'm worried it had write errors from original post.

Longer Summary

1. Replaced power cables and checked connections
2. Replaced both drives (parity1 and disk4) and attempted Data Rebuild/Parity Sync
3. Cancelled due to 20k+ read errors and the same i/o power reset errors
4. Replaced PSU
5. Before restoring drives, button and text seem different from step 2 and seem to only indicate Parity Sync (without mention of Data Rebuild)
6. Decide to split up restore to ensure disk4 is rebuilt
7. Rebuild disk4 in maintenance mode. Success, no errors, no indication of unmountable.
8. Sync parity1 in maintenance mode. Success, no errors.
9. Start array normally but disk4 is unmountable.
10. Run `xfs_repair` after doing some research. Every file/directory on disk4 is in "lost+found".
11. Mount original disk4 with Unassigned Devices without issue.

[JorgeB]

Run an extended SMART test on old disk4, if it passes I would format new disk4 and copy the that from the old one.