Moving Cache devices


Recommended Posts

My situation:

 

I had 1TB NVME and 1TB Sata SSDs as a cache pool. I bought an aiddtional 1TB NVME so that i could swap out the sata SSD to make a separate unprotected cache pool.

 

So, i stopped the array, and removed the sata ssd from the cache pool, and swapped the NVME in its place. Started array and it complained that the pool could nto be started up due to too many changes. Ok, so i dropped the pool down to just 1 device (the original NVME and it started up fine. I then set the old sata SSD into its own cache (called cache_unprotected). and started the array.

 

I saw that the cache_unprotected pool had data in it, must have been when it was being used as parity, So i went through using MC and deleted the appdata folder and a couple others. What actually happenned though was that these folders were deleted from BOTH cache pools!

 

Cue panic and a server reboot. Nope, the appdata folders have been deleted from both pools.

 

So i'm now restoring my appdata folder from a backup, and i can see its also restoring that folder to BOTH cache pools (screenshot)

image.thumb.png.4fc6fa5717948ebfa43e82cbc9227867.png

 

So, what is the right way to do what I was doing? and how can i "unlink" these devices? Am I going to have to move all data and everything to the array, then delete all cache pools and move it all back again (as if I only had the one cache device and was replacing it? I have gone through that once before already)

Link to comment
20 minutes ago, ChatNoir said:

Please attach your diagnostics to your next post.

 

EDIT: i have removed the second cache pool in an effort to resolve, but made no difference. the sata SSD is now an unassigned device (not mounted) and its still being written to/read from as if it were still in the original cache pool

 

Done

unraid-diagnostics-20220303-1501.zip

Edited by eatoff
spelling
Link to comment
4 hours ago, eatoff said:

Ok, so i dropped the pool down to just 1 device

You can't do this, and it shouldn't be allowed, it is because of a bug , by doing this the removed device won't be deleted from the pool, so when you assign it to a different pool both will show the same content, to correctly remove the device do this:

 

-stop array

-if docker/VM services are using the pool temporarily disable them

-unassign all pool devices

-start array

-stop array

-assign all pool devices to the same pool (there can't be a data on this device will be deleted in front of any pool member)

-start array (you can re-enable docker/VM services now)

-stop array

-unassign the pool member you want to removed

-start array

-once the balance and device delete finishes you can stop the array to re-assign the removed device to a new pool

Link to comment
59 minutes ago, JorgeB said:

You can't do this, and it shouldn't be allowed, it is because of a bug , by doing this the removed device won't be deleted from the pool, so when you assign it to a different pool both will show the same content, to correctly remove the device do this:

 

-stop array

-if docker/VM services are using the pool temporarily disable them

-unassign all pool devices

-start array

-stop array

-assign all pool devices to the same pool (there can't be a data on this device will be deleted in front of any pool member)

-start array (you can re-enable docker/VM services now)

-stop array

-unassign the pool member you want to removed

-start array

-once the balance and device delete finishes you can stop the array to re-assign the removed device to a new pool

Awesome, thanks for the effort writing this up. Will I need to move all the data off the cache before unassigning all pool devices?

 

Edit: or do I just put the drive I want to keep the data from in slot 1 of the cache pool and then parity will be built from it, that's preserving that data?

Edited by eatoff
Link to comment
24 minutes ago, eatoff said:

Will I need to move all the data off the cache before unassigning all pool devices?

No.

 

24 minutes ago, eatoff said:

Edit: or do I just put the drive I want to keep the data from in slot 1 of the cache pool and then parity will be built from it, that's preserving that data?

What do you mean parity? There's no parity with pools, pool is in raid1 (mirror) so the remaining device will have all the data.

Link to comment
44 minutes ago, JorgeB said:

What do you mean parity? There's no parity with pools, pool is in raid1 (mirror) so the remaining device will have all the data.

I mean when having 2 SSD in a pool for redundancy, so yes, RAID 1 pool.

 

So the drive with the data I want to keep I put in slot 1 of the pool, and then whatever one I put in the second slot will be overwritten to make RAID 1?

Edited by eatoff
Link to comment
25 minutes ago, eatoff said:

So the drive with the data I want to keep I put in slot 1 of the pool, and then whatever one I put in the second slot will be overwritten to make RAID 1?

Yes, you can add another member later, but first you need to remove the second member from the pool by doing the steps outlined above.

Link to comment
16 hours ago, JorgeB said:

You can't do this, and it shouldn't be allowed, it is because of a bug , by doing this the removed device won't be deleted from the pool, so when you assign it to a different pool both will show the same content, to correctly remove the device do this:

 

-stop array

-if docker/VM services are using the pool temporarily disable them

-unassign all pool devices

-start array

-stop array

-assign all pool devices to the same pool (there can't be a data on this device will be deleted in front of any pool member)

-start array (you can re-enable docker/VM services now)

-stop array

-unassign the pool member you want to removed

-start array

-once the balance and device delete finishes you can stop the array to re-assign the removed device to a new pool

 

This didnt work unfortunately.

 

I followed the steps, added all three devices to the same pool and started up the array. Let it copy everything between the devices across the pool. stopped the array. Removed the sata SSD from the pool and started the array. Unraid complained the cache was unmountable and that i should format the cache (which i declined to do since that would wipe all data that i wanted to keep). So then reverting back, i made the pool only have the 1 device (the original NVME i wanted to keep) and restarted the array. Now when writing to the array i can see the old sata ssd also being written to in the unassigned devices. I have attached the diagnostics, but i think I'm going to have to just delete the pool entirely and start it fresh.

unraid-diagnostics-20220304-0853.zip

Link to comment
5 minutes ago, eatoff said:

I'm going to have to just delete the pool entirely and start it fresh.

 

So, the result of deleting the pool and starting fresh is the exact same. Writing to the array sees data being written to both the NVME SSD and the SATA SSD in the unassigned devices section.

 

image.thumb.png.e2b15d776d7d8bdf9fbb743af7734780.png

Link to comment
1 hour ago, JonathanM said:

BTRFS is finding the second member and using it anyway, regardless of how it's assigned. If you don't need the data on either of the devices, run blkdiscard on the /dev device names. That will clear off all formatting and data on them.

Which 2 devices are you referring to? the 2 devices in the unassigned devices list, or the 2 devices that were in the original pool? Or just nuke all 3 SSDs and start from scratch (after moving appdata etc off the cache pool)?

 

EDIT: I'm just moving everything off the cache now, and will run the blkdiscard on the 3 ssds and start from scatch.

Edited by eatoff
Link to comment
7 hours ago, eatoff said:

This didnt work unfortunately.

 

I followed the steps, added all three devices to the same pool and started up the array

 

Doesn't matter now if you nuked it but the pool was 2 devices, not 3, that's the members I said to use, the original ones:

 

                  Data      Metadata System               
Id Path           RAID1     RAID1    RAID1     Unallocated
-- -------------- --------- -------- --------- -----------
 1 /dev/nvme1n1p1 770.00GiB  5.00GiB  32.00MiB   156.48GiB
 2 /dev/sdf1      770.00GiB  5.00GiB  32.00MiB   156.48GiB
-- -------------- --------- -------- --------- -----------
   Total          770.00GiB  5.00GiB  32.00MiB   312.96GiB
   Used           400.38GiB  1.56GiB 144.00KiB

 

Then you'd remove the SSD and only after that add the new device.

 

21 hours ago, JorgeB said:

Yes, you can add another member later, but first you need to remove the second member from the pool by doing the steps outlined above.

 

Link to comment

Taking a look at the diags, and although you didn't follow my instructions, it would have still worked but you ran into same bug again, like mentioned above you can't remove a pool device and then change the number of pool slots, this shouldn't be allowed, and it won't be in rc3, since by doing that the unassigned device is not deleted from the pool.

Link to comment

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.

Guest
Reply to this topic...

×   Pasted as rich text.   Restore formatting

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.