Hi Guys, as in the title, this is my first multi-drive failure with UNRAID. Contrary to what I would have expected from waking up to multiple dead drives I'm actually somewhat excited to learn the process of recovering from a situation like this. I have an array of 26+2 and 3 drives have spat the dummy - one of the main reasons I picked UNRAID was for this exact situation, if more drives than parity can restore detonate then you only lose the data on the drives that turned themselves into paper weights. At this point in time I have 2 failed Parity disks and 1 data disk that decided to grenade itself after parity went bye-bye. Timeline: Midnight - Routine Parity check starts ~2am - Parity disk 1 starts throwing errors and gets marked bad ~3am - Parity disk 2 starts throwing errors and gets marked bad ~3:30am - Disk 23 starts throwing 'Sector 1 unreadable' ~3:50am - I notice issues with server and start troubleshooting ~4am Drive goes from 'Unmountable' to 'Missing' after restarting server 4:30 - Here we are at time of writing Troubleshooting: So, for troubleshooting of all disks I've done the basics: Restart the server Power Cycle the disk shelf For Parity 1 and 2 - Both are SAS (PD1 and PD2) SMART DST - Pass - I suspect they are still functional and contain valid parity data as everything that gets written to the array gets written to Cache first and transferred to disk at 6am logs show IO errors on at least PD1, I'm revising my opinion here, both have probably taken an arrow to the knee. For Disk 23 - SAS (D23) Reconnect the drive in the same bay slot - Still not detected Swap the drive to another bay in the chassis (on a different backplane) - No detect Conclusion: Here is my conclusion, Please do tell me if you think there is something else I can/should do before moving on: D23 going Dodo would have been recoverable in addition to PD1 so long as PD2 hadn't also decided to fall flat on its face. Unfortunately with all 3 drives marked bad by UNRAID this means that the array is 'unrecoverable' in a traditional sense. I have a hot spare disk that I could use to replace D23 literally installed and waiting, and can get a couple of disks for the Parity drives later today to build a new array but I want to keep the data on all other disks. Boiled down, I have to replace the failed drives and create a new array config with new drives maintaining disk order to keep data on the drives that aren't dead but... Questions: 1: I have 23 other disks from this array all with information on them and I would like to keep the information on these disks - I understand that this should be possible when creating a new config with the drives but I would like someone with experience to point me to the right resource or 'hold my hand' while doing this for the first time to make sure I don't do something dumb. Things like letting me know of any risks/things to be aware of when doing this would also be very much appreciated! 2: Why did I say 23 and not 25 'other disks'? I added 2 disks within the last 30 days and, as of ~6 hours prior to the failure, neither of them had anything stored on them yet. My question is: One of these drives happens to be the right size to become a parity disk, would it be possible to remove this disk and assign it as parity instead of buying a replacement or would doing this be a case of 'don't know what could happen, just be safe and don't do it' recommendation? 3: Is it possible that the 2 Parity drives are A-OK (they pass SMART etc.) and could be forced to resume opteration in an attempt to restore the data on failed D23 to the hot spare before getting the parity drives replaced just in case? 4: If option 3 were tried what possible issues could occur if the parity drives aren't 'dead' but just not reading correctly? Would D23's content be mangled and unusable or worse - like, rest of the array data takes a grenade? - AKA, is this something reasonably low risk and worth trying or something high risk and I should just write off D23 and count my blessings that the other disks aren't FUBAR too? Things to note: Not all data on this array is backed up but all critical files are - any data stored on this array that isn't backed up, primarily - lets say for argument sake - ISOs, is replaceable and only classed as 'nice to have.' If push comes to shove I'm more than capable of manually verifying what ones are missing and re-acquiring anything lost. I'm not in a huge rush to fix this problem so if there are options that take time to try thats fine. Its now 5:55am and I'm going back to bed and will check back in a little while, I can post any information that you would like when I get back up in a couple of hours Earlier in the day I had updated UNRAID 6.11.0 --> 6.11.5 and the unit was awaiting a restart which I was planning on doing after parity check had completed today but had to restart the server during troubleshooting which completed the update - not sure if this is an issue for logs or something.

If parity is not 100% in sync only the bad part on the rebuilt disk would also be bad. Yes, it might be better to re-enable only parity2, since if both are re-enabled only parity1 will be used to emulated the missing disk, assuming no read errors on a different disk. The procedure would be the following (you need a new disk to replace disk23): -Tools -> New Config -> Retain current configuration: All -> Apply -Unassign parity1 and then check all assignments, parity2 must remain assigned as parity2, they are not interchangeable, assign any missing disk(s) if needed, including the new disk23, replacement disk should be same size or larger than the old one -IMPORTANT - Check both "parity is already valid" and "maintenance mode" and start the array (note that the GUI will still show that data on parity disk(s) will be overwritten, this is normal as it doesn't account for the checkbox, but it won't be as long as it's checked) -Stop array -Unassign disk23 -Start array (in normal mode now), ideally the emulated disk will now mount and contents look correct, if it doesn't post new diags

First time multi-drive failure

Followers

December 29, 20223 yr

Author

36 minutes ago, trurl said:

It is emulated, but unmountable. Check filesystem on emulated disk23

Dear lord, you can checkfs an emaulated disk - OK, done and remounted the array in online mode, disk is mounted and only 83GB in L+F out of 1.6TB on the disk - That is monstrously good.

Am I correct in assuming the process to complete will be as Follows:

Offline array --> Assign replacement disk23 -> rebuild data

Once data rebuild is complete --> offline array --> Assign Parity Disk 1 --> rebuild Parity

Done after that right?

Edited December 29, 20223 yr by TheIronAngel

Quote

December 29, 20223 yr

Author

Much appreciated for all your help guys, I have the rebuild of D23 running to the pre-cleared hot spare and another preclear is running on the new 10TB drives before I use them. I wasn't expecting to get anything back from that disk but y'all are miracle workers and looks like everything is back!

Happy new year to you all!

Quote

December 29, 20223 yr

Author

@trurl @JorgeB Looks like the parity disk is bad after all, any insight into the current status from a diag?

It got 625GB into the rebuild before getting marked bad again.

Since I only attempted a rebuild using PD2 do you think it would be possible for me to complete the same steps again this time with PD1 as the parity disk or is it time to call it unrecoverable?

Since I was able to emulate the disk I know exactly what was stored on the drive so recovering the information shouldn't be particularly difficult.

diskstation2-diagnostics-20221230-0136.zip

Edited December 29, 20223 yr by TheIronAngel

Quote

December 29, 20223 yr

Community Expert

It's not logged as a disk problem, but looks like a white label drive? Don't really trust those.

Quote

December 29, 20223 yr

Author

It is indeed a white label unit. I can see why you wouldn't trust them, I've lost trust in them too now to be honest. Although it has a 1yr warranty I think I'll avoid them going forward. Not worth the hassle but I'll still get them RMA'd since they're only a little over a month old. Unraid has disabled the disk after 11,166 read errors.

Quote

December 29, 20223 yr

Author

~~Would it be even worth attempting a rebuild using PD1 or just scrap them?~~

I've realised how dumb this idea is since the array has been online and the parity will now be *very* out of sync - unless you guys have something you can recommend trying I'll dump the old Parity disks out tomorrow, install the new ones and and rebuild with them. As mentioned before now that I have a file list of what was on D23 I can easily replace the missing content much quicker than checking every single folder for missing data so it's not all that much of a time sink.

I think I'll have to raw-dog the new drives striaght into production as I simply don't have the time to wait for a full Read/Write/Read clear of the drive - I will make it known that I would really prefer to burn in the new drives with a complete preclear but these are brand new Ironwolf units so I have a baseline trust that they work out of the box and as it stands right now getting some form of parity protection on the array is more important that running a preclear on brand new disks.

Edited December 29, 20223 yr by TheIronAngel
Revision without ping

Quote

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.

Followers

Go to topic listing

First time multi-drive failure

Featured Replies

Join the conversation

Account

Navigation

Search

Configure browser push notifications

Chrome (Android)

Chrome (Desktop)

Safari (iOS 16.4+)

Safari (macOS)

Edge (Android)

Edge (Desktop)

Firefox (Android)

Firefox (Desktop)