[SOLVED] Unmountable disk when trying to replace a cache pool disk

[Jaybau]

Unmountable disk present: Cache

I attempted to follow these instructions to Replace/upgrade a cache pool disk.

 

1. stop the array
2. on the main page click on the cache device you want to replace/upgrade and select the new one from the drop down list (any data on the new device will be deleted)
3. start the array
4. a btrfs device replace will begin, wait for cache activity to stop, the stop array button will be inhibited during the operation, this can take some time depending on how much data is on the pool and how fast your devices are.
5. when the cache activity stops or the stop array button is available the replacement is done.

 

I got to step #3, thinking Unraid was doing #4.  The process didn't take very long.

When the activity "stopped" or was "done", I didn't have any of my cache drives mountable.  I tried the new drive, and it wouldn't mount.  I retried the original drive, and it would no longer mount.

 

I did not format any of the drives, so I hope my original drive still has the data.  But I did try and switch back to my original drive, and hopefully that didn't delete any data.

 

tower-diagnostics-20220521-1303.zip

Edited May 23, 2022 by Jaybau

[JorgeB]

Log doesn't show the replacement attempt, server was rebooted, only both devices without a filesystem, which one was the original device? I assume it was btrfs formatted?

 

 

[Jaybau]

I might have the log before the reboot, but it's not anonymized.  Which file(s) would be helpful?

 

Original Cache Drive (single drive BTRFS cache pool):  

SAMSUNG_MZ7LN256HAJQ-000H1_S3TCNX0M705032 (sdf)

 

New Cache Drive:

M4-CT256M4SSD2_00000000130309265B2B (sdg)

 

"both devices without a filesystem"

There was no warning or confirmation that a drive was going to be erased by simply swapping drives in the dropdown list.

 

"you can still have a single-device "pool" if the number of defined cache slots >= 2"

I didn't increase the defined cache slots to 2; I only had 1 slot defined.  I was going to try this missed step, but not if the original drive is now erased.

 

What I did is swap sdf -> sdg in the dropdown.  When I noticed sdg is unmountable, I swapped back to sdf, and noticed that drive too was unmountable.

 

Other things that I've tried: 

Rebooting.

I also tried mounting the original cache drive sdf to the array to see if I can read it from there.  This required a New Config (but only after the problem).

 

Is the sdf drive truly erased and unmountable?  

Does the swapping in the dropdown (with only 1 slot) initiate a drive erasure?

I've read other posts saying this process is "broken" on lower Unraid versions (I'm on 6.10 stable).  Might it still be broken?  Might it cause a drive erasure?

 

Thank you.

[JorgeB]

14 hours ago, Jaybau said:

I didn't increase the defined cache slots to 2; I only had 1 slot defined.

Yes, without this step the replacement won't work, and the old device will still be wiped.

 

If the old device was wiped by Unraid but nothing more was done you should be able to recover the fs by running:

 

btrfs-select-super -s 1 /dev/sdX1

 

Replace X with the correct letter, note the 1 in the end, if the command completes OK assign that device and start the array, if it mounts you can now try the replacement.

 

 

 

 

[Jaybau]

This command worked to get my original cache drive back:

btrfs-select-super -s 1 /dev/sdX1

 

I finally got the drive swap to work, but not without issues.  

 

Here's what I did to swap a cache disk of a single disk pool (I may have missed something):

1. Stop the array
2. Create an extra slot for the cache pool.
3. Add the new drive to the new slot. (both the original and new drive in the cache pool).  In the cache pool section, make sure none of the drives have a note about erasing a drive.  I think I saw this warning based on which drive was in which slot (but can't remember which drive was in which slot to get the warning).  The warning is not obvious unless you look for it.  Can't remember when or what happened to get the message.
4. Start the array.
5. Wait for the activity to stop.
6. This gave me a two drive cache pool.  Still need to remove the original drive from the cache pool (next step).
7. Stop the array.
8. Remove the original drive from the cache pool, and reordered so my new drive is slot #1. (again, make sure there's no erasure warning in the cache pool section).
9. Remove slot #2.  If you don't do this, I think the filesystem/Unraid will think the pool is broken.
10. Start the array.

I think basically what is happening is this process is forcing BTRFS to expand itself (1 drive -> 2 drives), then shrink itself (2 drives -> 1 drive).

 

Hopefully in the future there's a guided script to make this idiot/worry proof (including backup).

 

Thank you for the assistance.

[JorgeB]

8 hours ago, Jaybau said:

Add the new drive to the new slot.

That's not part of the replacement procedure, you just assign the new device to the same slot the old one was, i.e. replace, what you did if for when you want to add a new device.