Parity rebuild problem - any ideas?

[StatMatt]

I recently decided to upgrade one of my array drives from an older 6TB to a new 18TB drive.  I followed the steps in the UNRAID documentation (https://docs.unraid.net/unraid-os/manual/storage-management/) and watched my server do a parity rebuild over the course of about 2 days.  Now the parity rebuild says it's done.  The old drive was about half full so it had about 3TB of files, but the new drive is now showing only 2.75MB (i.e. 0.00000275TB), so basically nothing.  On the main tab if I browse the contents of the drive there are no files or folders showing AT ALL (it says: 0 objects: 0 directories, 0 files (0 B total)).  It appears the parity rebuild didn't actually work.  I was watching the main tab from time-to-time throughout the rebuild and it was showing tons of reads from the other seven array drives and writes to the new drive, so I'm really confused about where my data is.  What do I do now?

 

Not sure if it's relevant, but the old 6TB drive was formatted xfs, but I decided to try zfs on the new drive.  I didn't think the underlying FS would matter for parity rebuild but maybe it does?

 

I had most of my docker containers except Emby turned off during the rebuild, and this morning when I saw the parity rebuild seems to have failed.  I also hadn't disabled mover, but I didn't think that was necessary since the documentation didn't mention anything about that.

 

The old 6TB drive is sitting on the workbench waiting to be cleared for re-use for another purpose.  I don't have an extra 3.5" bay in that server so it would be difficult to put it back but I could probably gin something up if needed.  I also have another backup UNRAID server that I could put it in pretty easily (also a couple of Win10 boxes) if that would help, but I'm reluctant to do anything until I get some guidance. 

 

Diagnostics attached.

bonzo-diagnostics-20230829-0904.zip

[JorgeB]

Do you mean disk2? The disk was newly formatted zfs:

 

Aug 27 11:43:09 Bonzo emhttpd: creating volume: disk2 (zfs)

 

Formatting disks is never part of a rebuild.

[itimpi]

21 minutes ago, StatMatt said:

I didn't think the underlying FS would matter for parity rebuild but maybe it does?

It does - you can only rebuild to the same file system as the parity rebuild process is unaware of file systems as it work at the raw sector level.

 

If you tried to format it you would have got a popup warning you that this would erase all data, update parity accordingly and that a format is never part of a rebuild. It looks like you continued with the format anyway?

 

If you still have the old 6TB drive untouched you should be able to mount it using Unassigned Devices to get the data back.

[StatMatt]

I somewhat remember a popup, but I thought that was a normal part of the process. This was my first time replacing a drive in the array so I didn't realize the message was abnormal.  I thought it was just warning me that any existing data on the new drive would be wiped.  I don't recall seeing any warnings about "format is never part of a rebuild".  Why would it then spend nearly 2 days doing all of the reads from the other pool drives and writes to the new drive telling me that it was updating parity?  Shouldn't it have been doing mostly writes TO the parity drive?

 

Suggest updating the documentation and warnings to be more clear.

[itimpi]

8 minutes ago, StatMatt said:

Why would it then spend nearly 2 days doing all of the reads from the other pool drives and writes to the new drive telling me that it was updating parity? 

It should have been doing writes to the drive you were replacing.   Since the rebuild does not understand file systems it was unaware that you formatted the emulated disk.

[JorgeB]

40 minutes ago, StatMatt said:

I don't recall seeing any warnings about "format is never part of a rebuild"

[StatMatt]

Obviously this is frustrating to me and not my intended outcome, to now have to mess around for several more days figuring out how to get my data back and off the old drive. I think the documentation and warnings could have been more clear about the fact that you can not change filesystem during a rebuild.  It wasn't clear to me until you said it just now that the format was of the EMULATED disk (I don't even understand what that means, tbh) - it seemed to me like it was talking about formatting the new disk (which made sense to me - whenever I've added a new disk to the array one of the first things that needs to happen is for it to be formatted).  If I can make this mistake, others can too which is why I'm suggesting updating the documentation and warnings.

 

Data recovery question: Does it matter *how* I connect my old drive using unassigned devices?  I have a USB3 drive bay that I could use - apart from some likely transfer speed disadvantages is there any reason I couldn't use USB3 instead of SATA?  There aren't any more 3.5" bays in the server case so I don't have an easy way to put that disk back in.

[itimpi]

6 minutes ago, StatMatt said:

Data recovery question: Does it matter *how* I connect my old drive using unassigned devices?  I have a USB3 drive bay that I could use - apart from some likely transfer speed disadvantages is there any reason I couldn't use USB3 instead of SATA?  There aren't any more 3.5" bays in the server case so I don't have an easy way to put that disk back in.

 

It depends of on whether the USB case you put the drive into changes the presentation to the OS in any way.    I would suggest just try it as it normally works.  

[StatMatt]

In reply to JorgeB's post of the warning message: that warning says it'll lead to loss of all data on the drive being formatted.  That is where I feel it might be confusing, because I know from past experience that if I'm adding a new drive into the array that any of the data it might have had on it would be lost.  It wasn't at all clear to me that the drive being formatted was (apparently?) the EMULATED drive.  I thought the warning was about the new drive, which was blank and I wasn't expecting or even wanting any files on it to somehow become part of the array.  So this warning didn't seem applicable to me in my scenario.

[StatMatt]

By the way, thanks for posting that warning.  That was a couple of days ago and I didn't remember specifically what it said. Re-reading it now really helps me realize that I wasn't just being careless - I actually did read that message and honestly thought it wasn't relevant to my situation.

[itimpi]

6 minutes ago, StatMatt said:

I actually did read that message and honestly thought it wasn't relevant to my situation

Any suggestions for rewording it so that you would have realised it WAS relevant?

 

Looking at the documentation on rebuilding it never mentions formatting as part of rebuild, but it also does not explicitly warn you against that so adding that feels like a worthwhile change.

[StatMatt]

Two thoughts: regarding the warning message maybe adding an example ("e.g. replacing a drive in the array"), and also maybe something like "(if your array currently has an emulated disk due to disk failure or routine replacement, formatting may result in loss of all data on the emulated disk")?  Going a bit further, can you somehow have an additional check in the formatting process so if the array has an emulated disk that it gives either another stronger warning or maybe even prevents disk formatting (unless in enabled by checkbox somewhere deep in the settings that casual users wouldn't normally use)?  Seems to me like it would be an edge-case where someone would need to format a disk in an array with an emulated disk, so locking out that functionality (with a work-around for those who really know what they're doing) may be best.  I also agree that the documentation should clearly state that the filesystem cannot change during array disk replacement.  Thanks for the opportunity to provide input.

[StatMatt]

Sorry, one more follow-up question.  I've successfully used the Unassigned Devices plugin to mount the old 6TB drive via a USB-to-SATA adapter.  Now that I've got it mounted, I was thinking I would set up a user script utilizing rsync to move the files back onto the array.  It's usually my practice to write to a share (mnt/user/*sharename*) and I have always tried to avoid directly writing to a drive in the array (mnt/*disk#*).  For this scenario, would you recommend that I just copy everything to the replacement drive directly (i.e. /mnt/*disk#*), or should I copy to the share (for a few folders that would write to the cache pool, which has ample capacity for the contents of those folders) and let Mover sort everything out?  Or am I completely off-base thinking about using rsync for this task?  Any other suggestions?  Thanks!

[JorgeB]

2 hours ago, StatMatt said:

For this scenario, would you recommend that I just copy everything to the replacement drive directly (i.e. /mnt/*disk#*)

I would.