Hello all, I've just messed up a drive swap and I'm really, really close to losing data. First things first, the vital statistics:

• Unraid v6.11.5, running on...
• Dell PowerEdge R720xd
• 2x parity drives, 10x data drives, 2x cache drives - all formatted as BTRFS - controlled by...
• PERC H310 Mini, Firmware version 20.13.3-0001. Exposing the drives in JBOD mode, of course.
• Diagnostics are attached
• The drive I tried to swap, Disk 6, is dedicated to the share that holds my Time Machine backups, and similarly that share is pinned to Disk 6.

 

So what did I do to get into this mess? I needed to remove what I'm going to call drive 6A, so I stopped the array and removed it. I thought it would be out for a while, so I replaced it with an identically-sized drive I'm going to call 6B. I assigned drive 6B to the array, then started the array. The emulated contents of Disk 6 were correct, and they were eventually rebuilt onto drive 6B. All was well for a time.

 

After a while drive 6A became available again. Because it's a better drive than 6B, I decided to swap it back in. I stopped the array, removed 6B, and put 6A back in. Then things started to go a bit wrong. When I went back to restart the array Disk 6 showed as having no filesystem (though I'd never wiped it - it should have had all the same data on it as when it was removed). Although the option to format it was available, I did not do so - the warning message when you tick the format checkbox was quite clear that it wasn't what should be done. Although I found it a little bit odd that this hadn't happened with the earlier disk swap, I assumed from what I know of Unraid that the rebuild would of course rebuild the file system, so I started the array and allowed the rebuild to go on.

 

Here's where I made my second mistake: I didn't check the emulated contents of Disk 6. While the rebuild was ongoing Time Machine kept failing to do its backups. I thought it was just being fussy about its storage (as it can be sometimes) and it would sort itself out once the rebuild was complete. In hindsight, this should also have tipped me off that all was not well on Disk 6 and I should have checked its emulated contents.

 

Then, this evening, the rebuild finished. I ran Time Machine and it still failed. I went and had a look in Unraid, and Disk 6 was still showing as unmountable with no or wrong file system. I looked on both /mnt/user/Time\ Machine and /mnt/disk6. In both places the data was completely gone; the only thing remaining was the share directory.

 

So, what do I do now? The server's sitting there with the array stopped. I presume there's no hope of recovering the data from within Unraid, but I do still have drive 6B. I haven't checked yet, but it should still have the data from Disk 6. Is the best course of action to just format 6A then rsync the data from 6B back onto the array?

 

Finally, it's much less pressing, but I'd also like to know what I did wrong, and how I should have done it so I don't make this mistake again. Thank you in advance for all of your help.

 

Edited December 28, 20232 yr by ScottAS2
Remove diagnostics

The simplest would be to 

1. Pause any time machine backups 

2. Format 6A and ensure the share is available again on array

3. Rsync the data from 6B to array

 

Another approach could be to put back 6A and rebuild parity to match current disk contents, but this risks the whole array during rebuild, so i suggest you don't attempt this

Is 6B mountable in Unassigned Devices?

7 hours ago, apandey said:

Another approach could be to put back 6A and rebuild parity to match current disk contents, but this risks the whole array during rebuild, so i suggest you don't attempt this

Noted. It's tempting to look for ways to avoid having to transfer several terabytes back to Unraid but, unlike Disk 6, I don't have spare copies of everything else.

 

6 hours ago, trurl said:

Is 6B mountable in Unassigned Devices?

6B isn't connected to the server, but I've got it attached for now to a Raspberry Pi and it seems to be happy and have the data on it.

 

6A is in the server and does mount with Unassigned Devices and the (slightly outdated) data is still there. Again, it's tempting to look for a way to use it, but am I correct in thinking that any attempt to do so puts us into the scenario apandey warns against above?

 

7 hours ago, apandey said:

The simplest would be to 

1. Pause any time machine backups 

2. Format 6A and ensure the share is available again on array

3. Rsync the data from 6B to array

Okay. This looks like what I'm going to have to do.

Okay, I'm completely bamboozled now. As I mentioned above I had drive 6A mounted with Unassigned Devices. I pondered copying the data off it onto other drives in the array or connecting 6B to the server to do a local copy onto 6A (which I guess is where trurl was going). I decided against these because I'd have had to zero out a drive to remove it from the array, and/or copy Disk 6's contents more than once and settled on formatting 6A and copying 6B onto it from the Raspberry Pi (it turns out USB3 + gigabit Ethernet is actually faster then just USB2).

 

So, I reallocated drive 6A to the array. Yes, all contents will be overwritten when the array is started. Start the array, please. Then I went to find the format button. It wasn't there any more. Disk 6 showed as emulated with no complaint about missing or no file system, and a rebuild was in progress. Finding this odd, I took a look at the contents of Drive 6, and... everything seems to be there, as it was just before drive 6B was removed. Obviously, I'm going to wait for the rebuild to complete before declaring this fixed, but it's looking good, even if I don't understand why. My best guess at the moment is that Unraid found drive 6A somehow suspicious (perhaps due to an unclean dismount while it was being used outside of Unraid?) and thus declined to take it into the array, but mounting it with Unassigned Devices cleared this and allowed it to re-enter the array. I don't know why it wouldn't show the emulated contents as it would if Disk 6 was physically absent, though.

None of that would have anything to do with it.

 

Probably a problem with another disk (maybe just connection) was interfering with accurate emulation of the disk. All disks must be reliably read to reliably emulate and rebuild a disk.

 

Do any of your disks have SMART warnings on the Dashboard page?

3 hours ago, trurl said:

Do any of your disks have SMART warnings on the Dashboard page?

Nope. All healthy. Nor did I get any notifications about warnings other than the expected ones when a drive was removed, when rebuilds finished, etc..

Edited March 4, 20233 yr by ScottAS2

So the rebuild has completed, all the data is there, and Time Machine has just finished its first post-rebuild backup. I've still no idea where I went wrong, but everything works again and I didn't lose any data. Ultimately I'm happy.

Edited March 5, 20233 yr by ScottAS2