BTRFS cache pool reorganization

[boxer74]

Question on reworking my cache pool. Points as follows:

• Currently I have 4x 400gb SSDs in a BTRFS RAID10 pool.
• Would like to replace with 2x 1TB NVME drives in BTRFS RAID1 pool.
• To avoid using the old approach of having the mover temporarily move shares off the old pool and then back onto the new pool, I'd like to avoid downtime (wife+kids+friends need their plex) as follows and need to know if this will work:
  • Convert current pool to RAID1 using balance command in GUI.
  • Stop array.
  • Add 2x 1TB NVME drives to pool, let rebalance.
  • Stop array.
  • Unassign first 400gb SSD from pool.
  • Start array.
  • Let rebalance finish.
  • Stop array.
  • Unassign second 400gb SSD from pool.
  • Start array.
  • Let rebalance finish.
  • Stop array.
  • Unassign third 400gb SSD from pool.
  • Start array.
  • Let rebalance finish.
  • Stop array.
  • Unassign fourth 400gb SSD from pool.
  • Start array.
  • Let rebalance finish.
  • Stop array.

 

In hindsight after writing the above steps, I'm not sure this is any less disruptive to my plex viewers, though the start/stop processes are quick. Moving 400GB off and back onto the pool could take hours.

 

Any thoughts on which method is best?

 

[JorgeB]

Fastest and least intrusive way would be to do an online device replacement using the command line, replace cache1 with an NVMe, then replace the other one, balance the pool to raid1 and then remove the 2 extra devices, after all is done stop the array, unassign all the SSDs, assign the two NVMe devices and start the array.

[boxer74]

I had already started re-balancing the 4x SSD pool to RAID1 before you responded. Can your approach still be done? Rebalancing doesn't seem to take too long. Is there a guide anywhere on the commands needed to do this?

[JorgeB]

2 minutes ago, aberg83 said:

Can your approach still be done?

Yes, I'm about to leave work, but give me half an hour or so and I'll post the commands you can use.

[boxer74]

Thanks, I really appreciate it!

[JorgeB]

Well, turns out this don't work on the latest releases since unRAID will consider the new devices new and delete part of the btrfs info, you'll need to do it using the GUI.

 

Note, as long as the pool is healthy this should be a low risk operation, but it's always a good idea to backup any irreplaceable data before beginning.

 

If profile is already raid1 you can start:

 

-stop the array

-on the GUI replace cache1 with 1st NVMe device

-start the array, balance will start, wait for it to finish, you won't be able to stop the array before it's finished, you'll see a "btrfs operation running", as long as you're using latest stable or rc, and you should be using one or the other.

-when done stop array

-replace cache2 with 2nd NVMe device

-start array

-when balance done stop array

-unassign one of the extra SSDs

-start the array, wait for balance to finish

-stop array

-unassign remaining SSD

-start the array, when balance finishes you're done.

 

I'm going out for a couple of hours but will check in later if you have any issues.

 

 

 

 

 

 

[boxer74]

Thanks! I've actually got a parity check running now so I'll wait until tonight when it's finished before attempting this. Appreciate your help!

[JorgeB]

After some testing the online replacement/removal can still be done, it just needs an extra step in the end, but the procedure above works just as well, the main advantage of the this one is not needing all those array stop/starts, I'm going to post it anyway, also for my future reference, you can use either one.

 

Again like the other one, this should be a low risk operation, but it's always a good idea to backup any irreplaceable data before beginning.

 

To do the online replacement both NVMe devices need to be already partitioned, if you haven't yet install the Unassigned devices plugin and format both devices, any filesystem will do, then you can start:

 

btrfs replace start -f /dev/sdX1 /dev/nvmeXXXp1 /mnt/cache

Replace X with cache1 identifier, note the 1 in the end, replace XXX with first NVMe identifier, note p1 in the end.

 

You can check replacement progress with:

btrfs replace status /mnt/cache

When done repeat the command for cache2 and NVMe 2.

 

Again check status and when done you can remove the extra devices, you can remove both at the same time, if there's lots of data and it fails remove one at a time:

btrfs dev remove /dev/sdX1 /dev/sdY1 /mnt/cache

Replace X with cache3 and Y with cache 4, again note the 1 in the end.

 

When done you'll get the cursor back, stop the array, unassign all old cache devices, and this is very important, start the array with no cache devices assigned so that unRAID "forgets" the old cache pool, stop the array, only after doing can you assign the new cache devices, start the array and you're done.

 

 

[boxer74]

Thanks. I'll keep those commands in mind for future. I followed your previous steps and it worked just fine. Running now with an NVME cache pool of two devices. Now just need to find a better way of keeping these guys cool.