Cache Pool full after adding another drive

[Autchirion]

Hey Guys,

 

Yesterday I added a second SSD to my cache pool. It used to be only my Samsung_SSD_850_PRO_512GB, but now I added my old Samsung_SSD_840_EVO_120GB. My Pool is brtfs (encrypted), I got a warning, that if I add the 120GB SSD to the pool I will loose all the data on it (which is expected and perfectly fine).

After bringing the Array back online I expected it to format the data etc. unfortunately it seems like something else happened.

 

Do you guys have any ideas how to resolve this, right now all my self hosted services are unusable because they are running from cache.

 

Information (might be growing as I try to recover the whole situation).

 

After bringing up the Array I got the following warning (at 17:56):

Event: Unraid Cache disk message
Subject: Warning [SERVERNAME] -  pool BTRFS too many profiles (You can ignore this warning when a pool balance operation is in progress)
Description: Samsung_SSD_850_PRO_512GB_S250xxxxxxxxx (sdg)
Importance: warning

 

I assume this is because I added the new drive, but at 18:11 I got the warning:

Event: Unraid Cache disk disk utilization
Subject: Alert [SERVERNAME] - Cache disk is low on space (92%)
Description: Samsung_SSD_850_PRO_512GB_S250xxxxx (sdg)
Importance: alert

followed by more warnings (95% until 100%)

 

First thing I'm seeing is, that the Size is 632GB, used is 255GB and free is 131KB, so something went wrong here.

 

Balance Status (after running Perform full balance):

 

output of "btrfs fi df /mnt/cache" after running "Perform full balance" on the cache 

Data, RAID1: total=110.74GiB, used=110.74GiB
Data, single: total=18.00GiB, used=15.96GiB
System, RAID1: total=32.00MiB, used=48.00KiB
Metadata, RAID1: total=1.00GiB, used=142.50MiB
GlobalReserve, single: total=30.86MiB, used=0.00B
WARNING: Multiple block group profiles detected, see 'man btrfs(5)'.
WARNING:   Data: single, raid1

 

[JorgeB]

By default Unraid converts the pool to raid1 when you add a device, and raid1 will only use the capacity from the smallest device, if you want to expand the pool see here.

 

[Autchirion]

And I can only use 2x smalles drive size, so this absolutely doesn't make sense to add my second drive because it will actually decrease the size.

 

Ok, I figured out what to do, thank you!

Edited April 17, 2021 by Autchirion

[itimpi]

1 minute ago, Autchirion said:

And I can only use 2x smalles drive size, so this absolutely doesn't make sense to add my second drive because it will actually decrease the size.

The default of RAID1 gives redundancy which is why you are limited to the smallest drive for available space.  If you do not care about redundancy then you can use all the space as described in the link given above.

[JorgeB]

4 minutes ago, Autchirion said:

so this absolutely doesn't make sense to add my second drive because it will actually decrease the size.

Correct, you can balance to single mode to fully use both devices (without redundancy).

 

 

[Autchirion]

I just tried to set it to single mode, however I immediately got an error:

Unable to write to cache	Drive mounted read-only or completely full. Begin Investigation Here:

 

Now, I understand my cache drive is now completely full, how can I recover from this, can I just copy data via command line (using mv) from cache to a different drive?

 

Or can I just remove my small drive and everything will be fine, of course this step scares the hell out of me, because I don't to fall back to an old backup. Unfortunately I can't stop my array at the moment because it says mover is running, but it doesn't do anything (at least that what it looks like).

Edited April 17, 2021 by Autchirion

[JorgeB]

Please post diagnostics.

[Autchirion]

I was just about to edit my earlier comment, to report, that I fixed it.

So, I backed up the data to an external drive and then I just removed the small cache drive, that way I was able to remove the small cache drive and then bring the array back online.

 

Now I'm back at how it used to be before all that messing around, my Idea for the next try is to just make sure all my VMs aren't on the Cache and then restart the process, this way I assume I'll be able to switch from raid1 to raid0.