Well, dang - a lot going on today out of the blue

13 disk array, got a failed disk error today so did the usual and shut it down, popped in a new disk and fired her back up again. It kept trying to start the array, which took a long time. Eventually was able to stop the array, replace the missing disk, and start the rebuild process.

Suddenly showed a ton of errors on two other disks, so I paused the rebuild, and then shut down. (I'm pretty sure that's not what I was supposed to do...)

Took forever (couple hours) before I could stop the array -

Saw the missing disk (the replacement)

Saw another showing "unmountable - wrong or no file system"

What steps would y'all take to try and recover from this situation?

Anything in the syslog - other than "I'm screwed"?

tower-syslog-20250606-0122.zip

Edited June 6, 20251 yr by rockytt
new info

Please post the complete diagnsotics zip.

thanks - took a bit to get one - curious what secrets she'll reveal...

tower-diagnostics-20250606-1507.zip

Jun 6 07:41:38 Tower kernel: sd 5:0:2:0: Power-on or device reset occurred

Log is full of these, they typically mean a power/connection issue, check/replace cables for disk13 and post new diags after array start.

thank you for taking a look at this Jorge -

tower-diagnostics-20250607-0619.zip

Another update - I pulled both the (supposedly) bad disk 12 that started this whole thing, and also disk 13 which is what appears to be causing the big issues now.
Connected 'em both to a Windows machine using Paragon's Linex File systems for Windows and can actually see and read the files on 12.

Disk 13 however is given me a huge :( - (no supported volumes have been found)

Is there anyway to put 12 back into the array and have it recognized as the original disk and then rebuild 13 with a fresh drive?

(Been doing some reading here to see if I can manually modify the config files on the usb drive to show drive 12 as an active data drive - then I could put in a new drive in 13 and rebuild parity?)

(Yeah - I'm flailing a bit and grasping at straws here)

Edited June 8, 20251 yr by rockytt

Now there are issues with disk13, also check its cables, any power splitters in use?

A bunch - but this build has been up for years.

Additionally,as you can see from my latest post, 13 appears ok while 13 can't even connect - this using an external caddy to my windows machine

1 minute ago, rockytt said:

A bunch - but this build has been up for years.

Those can be a be problem, even if they weren't before, especially if splitting one SATA plug to more than two disks.

Replace cables for disk13 and post new diags after array start.

Thank you Jorge - but 13 is dead - confirmed after pulling disk and putting in external enclosure. No file system found.

12 appears ok in external enclosure -

Then do a standard disk replacement, if there are other issues or errors during the rebuild post new diags.

Wish I could -

Going back to the beginning, 12 showed as bad so I attempted a regular disk replacement which I cancelled after a couple of minutes when 13 started throwing errors.

I can't just put the original 12 back in - which is why I'd asked if there was a way to manually edit one or more of the config files so the system will recognize my original 12 again instead of treating it like a new disk and threatening to erase it.

Jorge - I appreciate you jumping in and trying to help - I know how difficult that can be at a distance -

I'm going to pursue some other options at this time to recover the data - but thank you for you willingness to answer my questions and get me going again. All the best!

You can try to force enable old disk12 to rebuild disk13, but it will only work if parity is still valid with it, can post the instructions for that if you want.

Unfortunately no - parity is enabled with 13, not 12...

That's not what I meant, I meant that parity still needs to be valid with the old disk, like if there weren't any writes made to the new disk12 after it got replaced, though it will never be 100% valid because of just mounting and unmounting the disks, but if there weren't any actual writes, it usually still works.

bad sata controller card - replaced and rebuilding array as we speak -