Spatial Disorder Posted November 18, 2017 Share Posted November 18, 2017 Started getting a cache drive full error and docker/VMs stopping/pausing...however the cache disk shows plenty of free space. Server has been extremely stable in the current configuration since about Feb. Though, I did add musicbranz/headphones dockers maybe 4-6 weeks ago. I did a reboot this morning (sorry, I'm from the Windows Server world...when things get squirrely it's time for a reboot) and this changed nothing. I also expanded the docker vdisk from 20GB to 25GB which also didn't help. Cache shouldn't ever get full before mover runs...I don't download/move much data around on average. Diagnostics are attached. server-diagnostics-20171118-1010.zip Quote Link to comment
JorgeB Posted November 18, 2017 Share Posted November 18, 2017 The filesystem is fully allocated, see here: https://forums.lime-technology.com/topic/54533-624-cache-drive-reporting-quotno-space-left-on-diskquot/#comment-532931 Quote Link to comment
Spatial Disorder Posted November 18, 2017 Author Share Posted November 18, 2017 I'm confused.... Before I did anything else: root@Server:~# btrfs fi show /mnt/cache Label: none uuid: 8df4175c-ffe2-44d7-91e2-fbb331319bed Total devices 1 FS bytes used 131.82GiB devid 1 size 232.89GiB used 232.89GiB path /dev/sdc1 After looking at /mnt/cache I forgot I have downloads sitting on cache drive...I deleted that (~11GB) I then ran the below command as suggested in the linked post root@Server:/mnt/cache/system# btrfs balance start -dusage=5 /mnt/cache Done, had to relocate 1 out of 236 chunks I then get: root@Server:/mnt/cache/system# btrfs fi show /mnt/cache Label: none uuid: 8df4175c-ffe2-44d7-91e2-fbb331319bed Total devices 1 FS bytes used 120.47GiB devid 1 size 232.89GiB used 232.88GiB path /dev/sdc1 I only have 4 shares on /mnt/cache: root@Server:/mnt/cache# du -sh /mnt/cache/appdata/ 38G /mnt/cache/appdata/ root@Server:/mnt/cache# du -sh /mnt/cache/domains/ 45G /mnt/cache/domains/ root@Server:/mnt/cache# du -sh /mnt/cache/downloads/ 205M /mnt/cache/downloads/ root@Server:/mnt/cache# du -sh /mnt/cache/system/ 26G /mnt/cache/system/ Which should add up to ~110GB used... Quote Link to comment
JorgeB Posted November 18, 2017 Share Posted November 18, 2017 That balance didn't free up many chunks but it's best to start small or you can get a ENOSPC, and it did relocate one chunk. Now do a more complete balance, e.g.: btrfs balance start -dusage=80 /mnt/cache Quote Link to comment
Spatial Disorder Posted November 18, 2017 Author Share Posted November 18, 2017 Well.... root@Server:/mnt/user/james# btrfs balance start -dusage=80 /mnt/cache ERROR: error during balancing '/mnt/cache': No space left on device There may be more info in syslog - try dmesg | tail root@Server:/mnt/user/james# dmesg | tail [27506.319628] BTRFS info (device sdc1): relocating block group 58003030016 flags 1 [27511.777268] BTRFS info (device sdc1): found 25126 extents [27606.230821] BTRFS info (device sdc1): found 25126 extents [27606.418496] BTRFS info (device sdc1): relocating block group 56929288192 flags 1 [27627.136389] BTRFS info (device sdc1): found 30137 extents [27682.014305] BTRFS info (device sdc1): found 30137 extents [27682.216675] BTRFS info (device sdc1): relocating block group 55855546368 flags 1 [27707.130530] BTRFS info (device sdc1): found 30129 extents [27773.906438] BTRFS info (device sdc1): found 30127 extents [27774.372412] BTRFS info (device sdc1): 3 enospc errors during balance Not sure what to do next...do I need to clear more space? That would mean moving off docker data in appdata or domains (Win10 / Xubuntu) vdisks. Quote Link to comment
JorgeB Posted November 18, 2017 Share Posted November 18, 2017 That's why I said you need to start small, I though that since it reallocated one chunk with 5 you could go straight to a much higher number, try d-usage=10, then 20 and you should be able to go to 80, if it still fails, try smaller increasing values until it works or you'll need to clear more space, when it's done schedule a weekly balance so it doens't happen again. 1 1 Quote Link to comment
Spatial Disorder Posted November 19, 2017 Author Share Posted November 19, 2017 Thank you johnnie.black! I would have never figured this one out on my own...and I've learned a little more than I wanted about btrfs than I ever wanted to 15 hours ago, johnnie.black said: That's why I said you need to start small, I though that since it reallocated one chunk with 5 you could go straight to a much higher number, try d-usage=10, then 20 and you should be able to go to 80, if it still fails, try smaller increasing values until it works or you'll need to clear more space, when it's done schedule a weekly balance so it doens't happen again. I had misunderstood what you meant by start small. So, even though it failed at 80, it did balance a significant amount. I went back and was able to quickly increment up to about 70....then worked my way up to a 100% balance with no errors. Now I'm showing: root@Server:~# btrfs fi show /mnt/cache Label: none uuid: 8df4175c-ffe2-44d7-91e2-fbb331319bed Total devices 1 FS bytes used 121.61GiB devid 1 size 232.89GiB used 125.02GiB path /dev/sdc1 Thanks again for all the help! Quote Link to comment
pwm Posted November 19, 2017 Share Posted November 19, 2017 There are many nice things with btrfs, but the free-space administration seems developed by the bastard developer from hell. It's almost impossible to move to btrfs without having to learn the hard way. 1 Quote Link to comment
JorgeB Posted November 19, 2017 Share Posted November 19, 2017 9 minutes ago, Spatial Disorder said: Now I'm showing: Looks good, I recommend you schedule a weekly balance, you can use the user scripts plugin, something like this is enough to keep it under control: btrfs balance start -dusage=75 /mnt/cache 4 minutes ago, pwm said: There are many nice things with btrfs, but the free-space administration seems developed by the bastard developer from hell. It's almost impossible to move to btrfs without having to learn the hard way. Yep, btrfs is kind of high maintenance, hopefully this specific issue should be fixed once we go to kernel 4.14. Quote Link to comment
Recommended Posts
Join the conversation
You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.