Docker Image Full, Cache Filling Up, All Kinds of Fun

[Galactician]

Long time listener first time caller here. Thanks in advance for any assistance. I've been able to solve loads of problems through browsing this forum.

 

The last few days I've been seeing my cache filling up to full, and my docker doing same. I suspect maybe this is a logging issue but I am still a bit too much of a noob to know for sure. Today I got a Fatal Docker Image error when trying to access Apps from the GUI.

I am hereby doing what the error message says and including my diagnostics. Any thoughts?

galactiserver-diagnostics-20211101-1419.zip

Edited November 1, 2021 by Galactician

[itimpi]

Your docker problem is probably because your cache drive is full and this has resulted in corruption of the docker.img file.   

 

There also seem to be quite  few shares with the Use Cache=prefer seting which means that you want their contents moved to the cache if there is room.   Since you have not set any value for the Minimum Free Space value this means that these shares are allowed to use all the free space on the cache.  It is not infrequent for people to use the Prefer setting when what they really want is the Yes setting.

 

I would think you need to check that you really want the "Prefer" setting for these shares (the help built into the GUI explains how the options for this setting work).  You also want to set the Minimum Free Space value for the cache to be bigger than the largest file you expect to copy to the cache so that it never gets completely filled.

 

The docker.img file will need to be deleted and then recreated (via the Advanced view under Settings->Docker once you have made sure their is enough space for it on the cChe drive.  You can then re-install all the docker container binaries via the Previous Apps section on the Apps tab.  This will restore them with all settings intact except for any Custom Networks (which you will have to redo manually).

 

[Galactician]

Thanks for the reply.  Actually just changed all those shares to Prefer today as opposed to Yes in an effort to troubleshoot. They were mostly all set to YES before I started having issues, but now I am realizing I just created more issues by doing that. Whoops!

 

Will do as you say, fix the settings, and report back. Cheers.

[Galactician]

Things are back to stable again. Thanks again for the help! I have some cascading issues here, but I really extra hosed myself with that 'prefer' setting. Won't make that mistake again.

[trurl]

Typically you want only appdata, domains, system shares set as prefer at least until you get a better understanding and can decide if there are any other shares that you want to stay on cache.

[lovswr]

I'm at my wits end.  I have been having trouble with my docker for 2 or 3 months.  Mainly, the running dockers just stop & the error is typically the docker image is either full or corrupted.  I have done some of the things posted in this very thread such as deleting the image completely & changing the size.  This helps for a few days but then it will just crash again.  I keep the image on my 1 TB ssd cache & I have run tests on that drive that say it is ok.  I just don't know what else to do.  Docker was running fine last night & now this morning it has crashed.  Diagnostics attached.  

valhalla-diagnostics-20211128-0642.zip

[Squid]

Nov 28 03:50:23 valhalla kernel: BTRFS error (device sdb1): bdev /dev/sdb1 errs: wr 248, rd 778, flush 0, corrupt 0, gen 0

This is being caused by

Nov 25 03:46:18 valhalla kernel: ata1.00: failed command: WRITE FPDMA QUEUED
Nov 25 03:46:18 valhalla kernel: ata1.00: cmd 61/00:50:a0:61:bc/04:00:00:00:00/40 tag 10 ncq dma 524288 out
Nov 25 03:46:18 valhalla kernel:         res 41/40:48:08:4a:af/00:00:19:00:00/40 Emask 0x9 (media error)

 

Which is in turn related to either cabling to the cache drive (loose?) or alternatively 

197 Current_Pending_Sector  -O--CK   100   100   000    -    64

 

You can try running the extended SMART tests against the cache drive to see if it clears this stuff up.

 

On the other hand, since you only have a single cache drive in the pool you're going to have best results by reformatting it as XFS instead of BTRFS.  BTRFS has tendencies to not be very forgiving in certain situations (or it is buggy) whereas XFS is rock solid and can handle any weirdness

 

Converting will require you to stop all services (docker and virtual machines) from the settings tab and then moving everything off of the cache drive onto the array temporarily (set all the shares to be use cache:YES) and then running mover.  

 

After everything is finished, change the format of the drive to XFS start the array and set all the applicable shares to be use cache: PREFER and then run mover.  Afterwards,  you should be able to (hopefully) re-enable the services and you're back in business.

[trurl]

Also, since you have given 100G to docker.img, you probably have something misconfigured with one or more applications. 20G is usually more than enough. The usual reason for filling docker.img is an application writing to a path that isn't mapped. Each path specified in each application must correspond to a container path in its mappings, case-sensitive.