Started getting a cache drive full error and docker/VMs stopping/pausing...however the cache disk shows plenty of free space. Server has been extremely stable in the current configuration since about Feb. Though, I did add  musicbranz/headphones dockers maybe 4-6 weeks ago.

I did a reboot this morning (sorry, I'm from the Windows Server world...when things get squirrely it's time for a reboot) and this changed nothing.

I also expanded the docker vdisk from 20GB to 25GB which also didn't help.

 

Cache shouldn't ever get full before mover runs...I don't download/move much data around on average.

 

Diagnostics are attached.

 

server-diagnostics-20171118-1010.zip

The filesystem is fully allocated, see here:

 

https://forums.lime-technology.com/topic/54533-624-cache-drive-reporting-quotno-space-left-on-diskquot/#comment-532931

 

I'm confused....

 

Before I did anything else:

root@Server:~# btrfs fi show /mnt/cache
Label: none  uuid: 8df4175c-ffe2-44d7-91e2-fbb331319bed
        Total devices 1 FS bytes used 131.82GiB
        devid    1 size 232.89GiB used 232.89GiB path /dev/sdc1

 

After looking at /mnt/cache I forgot I have downloads sitting on cache drive...I deleted that (~11GB)

I then ran the below command as suggested in the linked post

root@Server:/mnt/cache/system# btrfs balance start -dusage=5 /mnt/cache
Done, had to relocate 1 out of 236 chunks

 

I then get:

root@Server:/mnt/cache/system# btrfs fi show /mnt/cache
Label: none  uuid: 8df4175c-ffe2-44d7-91e2-fbb331319bed
        Total devices 1 FS bytes used 120.47GiB
        devid    1 size 232.89GiB used 232.88GiB path /dev/sdc1

 

I only have 4 shares on /mnt/cache:

root@Server:/mnt/cache# du -sh /mnt/cache/appdata/
38G     /mnt/cache/appdata/
root@Server:/mnt/cache# du -sh /mnt/cache/domains/
45G     /mnt/cache/domains/
root@Server:/mnt/cache# du -sh /mnt/cache/downloads/
205M    /mnt/cache/downloads/
root@Server:/mnt/cache# du -sh /mnt/cache/system/
26G     /mnt/cache/system/

Which should add up to ~110GB used...

 

 

 

 

That balance didn't free up many chunks but it's best to start small or you can get a ENOSPC, and it did relocate one chunk.

 

Now do a more complete balance, e.g.:

 

btrfs balance start -dusage=80 /mnt/cache

Well....

root@Server:/mnt/user/james# btrfs balance start -dusage=80 /mnt/cache
ERROR: error during balancing '/mnt/cache': No space left on device
There may be more info in syslog - try dmesg | tail

root@Server:/mnt/user/james# dmesg | tail
[27506.319628] BTRFS info (device sdc1): relocating block group 58003030016 flags 1
[27511.777268] BTRFS info (device sdc1): found 25126 extents
[27606.230821] BTRFS info (device sdc1): found 25126 extents
[27606.418496] BTRFS info (device sdc1): relocating block group 56929288192 flags 1
[27627.136389] BTRFS info (device sdc1): found 30137 extents
[27682.014305] BTRFS info (device sdc1): found 30137 extents
[27682.216675] BTRFS info (device sdc1): relocating block group 55855546368 flags 1
[27707.130530] BTRFS info (device sdc1): found 30129 extents
[27773.906438] BTRFS info (device sdc1): found 30127 extents
[27774.372412] BTRFS info (device sdc1): 3 enospc errors during balance

Not sure what to do next...do I need to clear more space? That would mean moving off docker data in appdata or domains (Win10 / Xubuntu) vdisks.

That's why I said you need to start small, I though that since it reallocated one chunk with 5 you could go straight to a much higher number, try d-usage=10, then 20 and you should be able to go to 80, if it still fails, try smaller increasing values until it works or you'll need to clear more space, when it's done schedule a weekly balance so it doens't happen again.

Thank you johnnie.black! I would have never figured this one out on my own...and I've learned a little more than I wanted about btrfs than I ever wanted to 

 

15 hours ago, johnnie.black said:

That's why I said you need to start small, I though that since it reallocated one chunk with 5 you could go straight to a much higher number, try d-usage=10, then 20 and you should be able to go to 80, if it still fails, try smaller increasing values until it works or you'll need to clear more space, when it's done schedule a weekly balance so it doens't happen again.

I had misunderstood what you meant by start small. So, even though it failed at 80, it did balance a significant amount.  I went back and was  able to quickly increment up to about 70....then worked my way up to a 100% balance with no errors. Now I'm showing:

root@Server:~# btrfs fi show /mnt/cache
Label: none  uuid: 8df4175c-ffe2-44d7-91e2-fbb331319bed
        Total devices 1 FS bytes used 121.61GiB
        devid    1 size 232.89GiB used 125.02GiB path /dev/sdc1

Thanks again for all the help!

There are many nice things with btrfs, but the free-space administration seems developed by the bastard developer from hell. 

 

It's almost impossible to move to btrfs without having to learn the hard way.

9 minutes ago, Spatial Disorder said:

Now I'm showing:

 

Looks good, I recommend you schedule a weekly balance, you can use the user scripts plugin, something like this is enough to keep it under control:

 

btrfs balance start -dusage=75 /mnt/cache

 

4 minutes ago, pwm said:

There are many nice things with btrfs, but the free-space administration seems developed by the bastard developer from hell. 

 

It's almost impossible to move to btrfs without having to learn the hard way.

 

Yep, btrfs is kind of high maintenance, hopefully this specific issue should be fixed once we go to kernel 4.14.