[Solved] Cache Disk Full - Shows Plenty of Free Space


Recommended Posts

Started getting a cache drive full error and docker/VMs stopping/pausing...however the cache disk shows plenty of free space. Server has been extremely stable in the current configuration since about Feb. Though, I did add  musicbranz/headphones dockers maybe 4-6 weeks ago.

I did a reboot this morning (sorry, I'm from the Windows Server world...when things get squirrely it's time for a reboot) and this changed nothing.

I also expanded the docker vdisk from 20GB to 25GB which also didn't help.

 

Cache shouldn't ever get full before mover runs...I don't download/move much data around on average.

 

Diagnostics are attached.

 

server-diagnostics-20171118-1010.zip

cache_storage.JPG

Link to comment

I'm confused....

 

Before I did anything else:

root@Server:~# btrfs fi show /mnt/cache
Label: none  uuid: 8df4175c-ffe2-44d7-91e2-fbb331319bed
        Total devices 1 FS bytes used 131.82GiB
        devid    1 size 232.89GiB used 232.89GiB path /dev/sdc1

 

After looking at /mnt/cache I forgot I have downloads sitting on cache drive...I deleted that (~11GB)

I then ran the below command as suggested in the linked post

root@Server:/mnt/cache/system# btrfs balance start -dusage=5 /mnt/cache
Done, had to relocate 1 out of 236 chunks

 

I then get:

root@Server:/mnt/cache/system# btrfs fi show /mnt/cache
Label: none  uuid: 8df4175c-ffe2-44d7-91e2-fbb331319bed
        Total devices 1 FS bytes used 120.47GiB
        devid    1 size 232.89GiB used 232.88GiB path /dev/sdc1

 

I only have 4 shares on /mnt/cache:

root@Server:/mnt/cache# du -sh /mnt/cache/appdata/
38G     /mnt/cache/appdata/
root@Server:/mnt/cache# du -sh /mnt/cache/domains/
45G     /mnt/cache/domains/
root@Server:/mnt/cache# du -sh /mnt/cache/downloads/
205M    /mnt/cache/downloads/
root@Server:/mnt/cache# du -sh /mnt/cache/system/
26G     /mnt/cache/system/

Which should add up to ~110GB used...

 

 

 

 

Link to comment

Well....

root@Server:/mnt/user/james# btrfs balance start -dusage=80 /mnt/cache
ERROR: error during balancing '/mnt/cache': No space left on device
There may be more info in syslog - try dmesg | tail

root@Server:/mnt/user/james# dmesg | tail
[27506.319628] BTRFS info (device sdc1): relocating block group 58003030016 flags 1
[27511.777268] BTRFS info (device sdc1): found 25126 extents
[27606.230821] BTRFS info (device sdc1): found 25126 extents
[27606.418496] BTRFS info (device sdc1): relocating block group 56929288192 flags 1
[27627.136389] BTRFS info (device sdc1): found 30137 extents
[27682.014305] BTRFS info (device sdc1): found 30137 extents
[27682.216675] BTRFS info (device sdc1): relocating block group 55855546368 flags 1
[27707.130530] BTRFS info (device sdc1): found 30129 extents
[27773.906438] BTRFS info (device sdc1): found 30127 extents
[27774.372412] BTRFS info (device sdc1): 3 enospc errors during balance

Not sure what to do next...do I need to clear more space? That would mean moving off docker data in appdata or domains (Win10 / Xubuntu) vdisks.

Link to comment

That's why I said you need to start small, I though that since it reallocated one chunk with 5 you could go straight to a much higher number, try d-usage=10, then 20 and you should be able to go to 80, if it still fails, try smaller increasing values until it works or you'll need to clear more space, when it's done schedule a weekly balance so it doens't happen again.

  • Like 1
  • Upvote 1
Link to comment

Thank you johnnie.black! I would have never figured this one out on my own...and I've learned a little more than I wanted about btrfs than I ever wanted to :S

 

15 hours ago, johnnie.black said:

That's why I said you need to start small, I though that since it reallocated one chunk with 5 you could go straight to a much higher number, try d-usage=10, then 20 and you should be able to go to 80, if it still fails, try smaller increasing values until it works or you'll need to clear more space, when it's done schedule a weekly balance so it doens't happen again.

I had misunderstood what you meant by start small. So, even though it failed at 80, it did balance a significant amount.  I went back and was  able to quickly increment up to about 70....then worked my way up to a 100% balance with no errors. Now I'm showing:

root@Server:~# btrfs fi show /mnt/cache
Label: none  uuid: 8df4175c-ffe2-44d7-91e2-fbb331319bed
        Total devices 1 FS bytes used 121.61GiB
        devid    1 size 232.89GiB used 125.02GiB path /dev/sdc1

Thanks again for all the help!

Link to comment
  • Spatial Disorder changed the title to [Solved] Cache Disk Full - Shows Plenty of Free Space
9 minutes ago, Spatial Disorder said:

Now I'm showing:

 

Looks good, I recommend you schedule a weekly balance, you can use the user scripts plugin, something like this is enough to keep it under control:

 

btrfs balance start -dusage=75 /mnt/cache

 

4 minutes ago, pwm said:

There are many nice things with btrfs, but the free-space administration seems developed by the bastard developer from hell. :(

 

It's almost impossible to move to btrfs without having to learn the hard way.

 

Yep, btrfs is kind of high maintenance, hopefully this specific issue should be fixed once we go to kernel 4.14.

Link to comment

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.

Guest
Reply to this topic...

×   Pasted as rich text.   Restore formatting

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.