I have briefly consulted the FAQ yet I don't seem to be able to find out how to rebuild cache pool.

 

UnRaid 6.2.1

Cache pool = 120GB + 480GB SSDs

 

Cache pool contains app data, VM images.  Not used for array caching

 

I want to swap out the 480GB with another spare 120GB I have around.  Standard replacement steps won't work as I am using a replacement disk smaller than the 480GB to be swapped out.

 

I believe the steps would involve

 

[*]Backup the whole cache pool to somewhere on the array (should I simply do "cp /mnt/cache /mnt/user/cache-backup"?)

[*]Disable all apps that run from the cache pool

[*]Stop the array, and dis-allow array auto-start upon power up

[*]Unassign the 480GB disk

[*]Power down UnRaid, replace 480GB and new 120GB

[*]Power up the UnRaid, assign the new 120GB to the cache pool

[*]Start the array, re-store cache backup

[*]Re-start the apps

Can someone comment on these steps?

 

Thanks in advance!

The replacement won't work for a smaller device, but you can remove the larger one following this procedure and add the other one after, obviously all the data must fit the smaller SSD.

It should also be possible to shrink the 480GB SDD to a smaller size and then replace it, I'll be trying this soon as I'm going to need this procedure myself and then add the steps to the FAQ.

It should also be possible to shrink the 480GB SDD to a smaller size and then replace it, I'll be trying this soon as I'm going to need this procedure myself and then add the steps to the FAQ.

 

Sounds good.

 

Unraid reports that the cache pool has 135GB used so I cannot remove the 480GB as the used space won't fit onto the other 120GB.  The reported cache pool size is 600GB total so I believe it's not using any RAID format (I have no access to my unraid atm).

Turns out it's very simple, I'll post here and later add to the FAQ:

 

first type:

 

btrfs fi show /mnt/cache

 

You'll get something like this:

 

btrfs fi show /mnt/cache
Label: none  uuid: 85941ebc-286d-4dac-9bbc-448e85d626cc
        Total devices 2 FS bytes used 2.92GiB
        devid    1 size 232.89GiB used 4.03GiB path /dev/sde1
        devid    2 size 132.89GiB used 4.00GiB path /dev/sdf1

 

Find the devid of the device you want to shrink, in this example it's sde = devid 1, now type:

 

btrfs filesystem resize X:-YG /mnt/cache

 

replace X with devid and Y with amount to decrease, e.g., if I wanted to decrease sde by 100GB:

 

btrfs filesystem resize 1:-100G /mnt/cache

 

You just need to shrink the large SSD to below 120GB to then do a direct replacement, obviously data needs to fit on new smaller size.

 

 

Thanks Johnnie, much appreciated!

 

According to this procedure, replacing a cache disk requires a spare SATA port for the new disk to sync.  Unfortunately I don't seem to be a spare available.

 

I could remove a disk from the array temporarily (therefore rendering the array unprotected), and make a SATA port available for this replacement procedure.  Then I will re-attach the disk back to the array again after all is good. If I do that, what would be the correct and less painful procedure?

Only if your were using raid1, if you have 600GB space you're using the single (non redundant) profile, post the output of:

 

btrfs fi df /mnt/cache

 

and

 

btrfs fi show /mnt/cache

 

Although I can't access my unraid now, I'm pretty sure it's single profile as you said, since the total size is 600GB.

 

So, do I need both the 480GB and the new 120GB online to do the replacement?

 

I can also find a SATA PCIe card to expand the SATA ports.

 

Please advice!

If it's the singe profile yes, cache won't mount with one of them missing, if you can move some data so all remaining will fit the 120GB device you can then remove the 480GB SSD and then add the other one.

If it's the singe profile yes, cache won't mount with one of them missing, if you can move some data so all remaining will fit the 120GB device you can then remove the 480GB SSD and then add the other one.

Ok, let me try to summarize what I should do.

[*]I will re-size the 480GB device to 120GB according to the steps above

[*]Reduce the cache usage (e.g., move VM images to the array) so that only the 1st 120GB disk is used

[*]Shutdown the box, and swap out the 480GB with the 2nd 120GB

[*]Turn it back on, fingers acrossed

 

For 2 above, how can I guarantee?

That won't work, if you have more than 120GB used you need to:

 

-move some that from the pool so it uses less than 120GB

-use the remove cache device procedure to remove the 480GB from the pool

-add the new 120GB

That won't work, if you have more than 120GB used you need to:

 

-move some that from the pool so it uses less than 120GB

-use the remove cache device procedure to remove the 480GB from the pool

-add the new 120GB

 

Ok, right now the cache pool has the following stats

 

root@UnRaid:~# btrfs fi df /mnt/cache
Data, single: total=144.78GiB, used=125.84GiB
System, single: total=4.00MiB, used=16.00KiB
Metadata, single: total=1.01GiB, used=32.42MiB
GlobalReserve, single: total=16.00MiB, used=0.00B

root@UnRaid:~# btrfs fi show /mnt/cache
Label: none  uuid: 967e7ee2-326d-40de-b35b-1cd805317520
        Total devices 2 FS bytes used 125.87GiB
        devid    1 size 111.79GiB used 111.79GiB path /dev/sdb1
        devid    2 size 447.13GiB used 34.00GiB path /dev/sdg1

I will now backup the whole /mnt/cache, remove an VM image to reduce the cache usage, then follow step 3 and onward of this procedure to remove the 480GB disk.

Hmm hit a snag...

 

This is after I trimmed down the cache usage:

 

root@UnRaid:/mnt/cache/domains# btrfs fi show /mnt/cache/

Label: none  uuid: 967e7ee2-326d-40de-b35b-1cd805317520

        Total devices 2 FS bytes used 86.31GiB

        devid    1 size 111.79GiB used 108.79GiB path /dev/sdb1

        devid    2 size 447.13GiB used 34.00GiB path /dev/sdg1

 

Somehow, /dev/sdb1 shows 108.79GiB used, when in fact the cache pool is only using 86.31GiB

 

Then,

 

root@UnRaid:/mnt/cache/domains# btrfs device remove /dev/sdg1 /mnt/cache

 

After a few minutes:

ERROR: error removing device '/dev/sdg1': No space left on device

 

Now /dev/sdb1 is shown full:

 

root@UnRaid:/mnt/cache# btrfs fi show /mnt/cache/
Label: none  uuid: 967e7ee2-326d-40de-b35b-1cd805317520
        Total devices 2 FS bytes used 86.30GiB
        devid    1 size 111.79GiB used 111.79GiB path /dev/sdb1
        devid    2 size 447.13GiB used 3.00GiB path /dev/sdg1

 

 

Any thoughts?

Hmm hit a snag...

 

 

Ok looks like the balance step is required

 

root@UnRaid:/mnt/user/cache-backup/domains# btrfs balance start -f /mnt/cache
WARNING:

        Full balance without filters requested. This operation is very
        intense and takes potentially very long. It is recommended to
        use the balance filters to narrow down the balanced data.
        Use 'btrfs balance start --full-balance' option to skip this
        warning. The operation will start in 10 seconds.
        Use Ctrl-C to stop it.
10 9 8 7 6 5 4 3 2 1
Starting balance without any filters.

 

Then after 20mins or so

 

root@UnRaid:/mnt/cache# btrfs fi show /mnt/cache/
Label: none  uuid: 967e7ee2-326d-40de-b35b-1cd805317520
        Total devices 2 FS bytes used 86.30GiB
        devid    1 size 111.79GiB used 0.00B path /dev/sdb1
        devid    2 size 447.13GiB used 89.03GiB path /dev/sdg1

 

And the removal seems to be doing the right thing now

 

root@UnRaid:/mnt/cache# btrfs device remove /dev/sdg1 /mnt/cache

 

root@UnRaid:/mnt/user/cache-backup/domains# btrfs fi show /mnt/cache
Label: none  uuid: 967e7ee2-326d-40de-b35b-1cd805317520
        Total devices 2 FS bytes used 86.30GiB
        devid    1 size 111.79GiB used 51.03GiB path /dev/sdb1
        devid    2 size 0.00B used 37.00GiB path /dev/sdg1

root@UnRaid:/mnt/user/cache-backup/domains# btrfs fi show /mnt/cache
Label: none  uuid: 967e7ee2-326d-40de-b35b-1cd805317520
        Total devices 2 FS bytes used 87.29GiB
        devid    1 size 111.79GiB used 66.03GiB path /dev/sdb1
        devid    2 size 0.00B used 23.00GiB path /dev/sdg1

 

And finally...

 

root@UnRaid:/mnt/user/cache-backup/domains# btrfs fi show /mnt/cache
Label: none  uuid: 967e7ee2-326d-40de-b35b-1cd805317520
        Total devices 1 FS bytes used 86.30GiB
        devid    1 size 111.79GiB used 88.03GiB path /dev/sdb1

root@UnRaid:/mnt/user/cache-backup/domains#

 

Now onto exchanging the disks...

Ok looks like the balance step is required

 

Yes, balance can be required after some deletions to recover unused or little used chunks.

The disk replacement is successful.

 

One step that I didn't expect to do, however, was to convert the pool to single profile, as when I assigned the second 120GB SSD, UnRaid processed the cache pool as Raid1 profile, to which UnRaid issued a "balance" operation:

 

root@UnRaid:/boot/config/plugins# btrfs fi df /mnt/cache
Data, RAID1: total=89.00GiB, used=77.93GiB
Data, single: total=10.00GiB, used=8.99GiB
System, single: total=32.00MiB, used=16.00KiB
Metadata, single: total=1.00GiB, used=24.47MiB
GlobalReserve, single: total=16.00MiB, used=0.00B

 

The command to convert to Single profile:

 

root@UnRaid:/boot/config/plugins# btrfs balance start -f -dconvert=single -mconvert=single /mnt/cache/
Done, had to relocate 100 out of 100 chunks

 

... was only possible after unRaid finished it's own balancing.

 

root@UnRaid:/boot/config/plugins# btrfs fi show /mnt/cache
Label: none  uuid: 967e7ee2-326d-40de-b35b-1cd805317520
        Total devices 2 FS bytes used 87.12GiB
        devid    1 size 111.79GiB used 50.03GiB path /dev/sdb1
        devid    2 size 111.79GiB used 49.00GiB path /dev/sdg1

root@UnRaid:/boot/config/plugins# btrfs fi df /mnt/cache
Data, single: total=98.00GiB, used=87.10GiB
System, single: total=32.00MiB, used=16.00KiB
Metadata, single: total=1.00GiB, used=25.39MiB
GlobalReserve, single: total=16.00MiB, used=0.00B

 

Mission accomplished!

 

Yes, as it is now any new device added to the pool will trigger a raid1 re-balance, Tom said more functionality will be added in the future.

PS: Since you are now using 2 same size devices you can use the raid0 profile, same redundancy and capacity but more performance.

PS: Since you are now using 2 same size devices you can use the raid0 profile, same redundancy and capacity but more performance.

 

Got an error after around 10% progress left:

 

root@UnRaid:/mnt/cache# btrfs balance start -dconvert=raid0 -mconvert=raid1 /mnt/cache/
ERROR: error during balancing '/mnt/cache/': No space left on device
There may be more info in syslog - try dmesg | tail
root@UnRaid:/mnt/cache# btrfs fi show /mnt/cache
Label: none  uuid: 967e7ee2-326d-40de-b35b-1cd805317520
        Total devices 2 FS bytes used 127.16GiB
        devid    1 size 111.79GiB used 110.79GiB path /dev/sdb1
        devid    2 size 111.79GiB used 109.76GiB path /dev/sdg1

root@UnRaid:/mnt/cache# btrfs fi df /mnt/cache
Data, RAID0: total=217.52GiB, used=127.14GiB
Data, single: total=1.00GiB, used=0.00B
System, single: total=32.00MiB, used=16.00KiB
Metadata, RAID1: total=1.00GiB, used=25.78MiB
GlobalReserve, single: total=16.00MiB, used=0.00B

 

I moved back a 50GB VM image back to /mnt/cache... the conversion failed probably because of that?

 

I have a hunch that the conversion from Single to RAID0 requires the current usage to be less than the size of a single device.  Let me try that...

I moved back a 50GB VM image back to /mnt/cache... the conversion failed probably because of that?

 

I have a hunch that the conversion from Single to RAID0 requires the current usage to be less than the size of a single device.  Let me try that...

 

After reducing the cache pool usage to around 90GB which is smaller than a 120GB device, the conversion was successful.

 

Before conversion:

root@UnRaid:/mnt/cache# btrfs fi show /mnt/cache
Label: none  uuid: 967e7ee2-326d-40de-b35b-1cd805317520
        Total devices 2 FS bytes used 87.13GiB
        devid    1 size 111.79GiB used 55.03GiB path /dev/sdb1
        devid    2 size 111.79GiB used 56.06GiB path /dev/sdg1

root@UnRaid:/mnt/cache# btrfs fi df /mnt/cache
Data, single: total=109.03GiB, used=87.10GiB
System, RAID1: total=32.00MiB, used=16.00KiB
Metadata, RAID1: total=1.00GiB, used=25.84MiB
GlobalReserve, single: total=16.00MiB, used=0.00B

 

Conversion command:

root@UnRaid:/mnt/cache# btrfs balance start -dconvert=raid0 -mconvert=raid1 /mnt/cache/
Done, had to relocate 112 out of 112 chunks

 

After conversion:

root@UnRaid:/mnt/cache# btrfs fi show /mnt/cache
Label: none  uuid: 967e7ee2-326d-40de-b35b-1cd805317520
        Total devices 2 FS bytes used 87.13GiB
        devid    1 size 111.79GiB used 110.73GiB path /dev/sdb1
        devid    2 size 111.79GiB used 109.73GiB path /dev/sdg1

root@UnRaid:/mnt/cache# btrfs fi df /mnt/cache
Data, RAID0: total=218.84GiB, used=87.11GiB
Data, single: total=1.00GiB, used=0.00B
System, RAID1: total=32.00MiB, used=16.00KiB
Metadata, RAID1: total=279.62MiB, used=25.64MiB
GlobalReserve, single: total=16.00MiB, used=0.00B

 

Strangely, after conversion, btrfs shows that each device used up ~110GiB.  And yet, btrfs df shows Data ... used=87.11GiB.  Perhaps a bug in the commands?

 

Anyways, everything seems to be working now, and Cache Data is RAID0.

That is used space and allocated space, they are different, you should run one more balance with the same settings to get rid of the remaining data single profile chunk:

 

Data, single: total=1.00GiB, used=0.00B

 

It should go away after another balance.

... you should run one more balance with the same settings to get rid of the remaining data single profile chunk:

 

Data, single: total=1.00GiB, used=0.00B

 

It should go away after another balance.

 

Clean now

 

root@UnRaid:/mnt/cache# btrfs balance start /mnt/cache/                         WARNING:

        Full balance without filters requested. This operation is very
        intense and takes potentially very long. It is recommended to
        use the balance filters to narrow down the balanced data.
        Use 'btrfs balance start --full-balance' option to skip this
        warning. The operation will start in 10 seconds.
        Use Ctrl-C to stop it.
10 9 8 7 6 5 4 3 2 1
Starting balance without any filters.
Done, had to relocate 113 out of 113 chunks
root@UnRaid:/mnt/cache# btrfs fi show /mnt/cache
Label: none  uuid: 967e7ee2-326d-40de-b35b-1cd805317520
        Total devices 2 FS bytes used 128.99GiB
        devid    1 size 111.79GiB used 67.06GiB path /dev/sdb1
        devid    2 size 111.79GiB used 67.06GiB path /dev/sdg1

root@UnRaid:/mnt/cache# btrfs fi df /mnt/cache
Data, RAID0: total=132.06GiB, used=128.96GiB
System, RAID1: total=32.00MiB, used=16.00KiB
Metadata, RAID1: total=1.00GiB, used=31.36MiB
GlobalReserve, single: total=16.00MiB, used=0.00B
root@UnRaid:/mnt/cache#