Docker Image becomes unwriteable every other Sunday

[RMichaelRae]

Running Unraid 6.9.2

 

Every other Sunday morning, at precisely 4:30a, I get an error from Fix Common Problems of "Unable to write to Docker Image". This has happened for about 3 months now. At first I thought it was just filling it, so I expanded it. This changed nothing. Spot checking a few times a day for weeks, I have never seen it larger than 15GB or so.

 

All of the errors have occurred at the same time, with the exception of the one this morning that occurred about 13 min later than I expected. I expect that this is likely related to some sort of scheduled task, but I have been unable to identify anything that runs remotely close to that time on a schedule. 

 

The screenshot is what I see when I view the docker image stats. The containers highlighted in red always run. Most of the others are always stopped. Notice the Writeable column says "nan B".

 

 

Also, on the Docker tab, this error is shown dozens of times... "Warning: mkdir(): Input/output error in /usr/local/emhttp/plugins/dynamix.docker.manager/include/DockerClient.php on line 331"

 

Thanks in advance for your guidance!

cassandra-diagnostics-20211107-0614.zip

[DigitalGoddess]

I would suggest looking to see if that isn't when your logs rotate or some other process starts/stops.  What do the server logs have to say that happening at that same time?  

[trurl]

You have completely filled your cache drive. The timing is due to the default mover schedule.

 

In addition to the usual cache-prefer shares appdata, domains, system, you have another share anonymized as t--p set to cache-prefer. What is the purpose of that share and why do you have it set to be moved to cache and stay on cache?

 

You also have a cache-no share, anonymized as n-s, with files on cache which will not be moved.

 

Mover ignores cache-no and cache-only, moves cache-yes shares from cache to array, and moves cache-prefer shares from array to cache.

 

Also noticed that some of your array disks are very full. Better if you leave some space on those.

[trurl]

Your full cache drive has corrupted your docker.img. After you get those 2 user shares configured correctly so you can get some of that moved off cache and have space again, you will have to delete and recreate your docker.img (20G will be more than enough)

 

https://wiki.unraid.net/Manual/Docker_Management#Re-Create_the_Docker_image_file

 

and reinstall your dockers

 

https://wiki.unraid.net/Manual/Docker_Management#Re-Installing_Docker_Applications

[RMichaelRae]

@trurl Thank you for the feedback. I will take a look at those as soon as I can, but the explanation makes sense. I appreciate you taking your time to assist.