Docker wont start and lost VM's after reboot - [SOLVED]

[1979]

Hello,

 

I've been using Unraid for a few months now and tonight I've had a serious issue.  I was having issues with a docker image, not being able to write to the HDD and reporting that the disk was read only, even though it has been working fine until today.

 

The docker log showed the following error:-

 

time="2021-05-06T00:30:20.508441397+01:00" level=error msg="Failed to log msg \"\" for logger json-file: write /var/lib/docker/containers/91e25451e514d61e59725f18205652b029d253abef36746569e2affd71ee2281/91e25451e514d61e59725f18205652b029d253abef36746569e2affd71ee2281-json.log: read-only file system"
time="2021-05-06T00:30:22.496091225+01:00" level=error msg="Failed to log msg \"\" for logger json-file: write /var/lib/docker/containers/91e25451e514d61e59725f18205652b029d253abef36746569e2affd71ee2281/91e25451e514d61e59725f18205652b029d253abef36746569e2affd71ee2281-json.log: read-only file system"
time="2021-05-06T00:30:22.496125858+01:00" level=error msg="Failed to log msg \"\" for logger json-file: write /var/lib/docker/containers/91e25451e514d61e59725f18205652b029d253abef36746569e2affd71ee2281/91e25451e514d61e59725f18205652b029d253abef36746569e2affd71ee2281-json.log: read-only file system"
time="2021-05-06T00:30:22.496143314+01:00" level=error msg="Failed to log msg \"\" for logger json-file: write /var/lib/docker/containers/91e25451e514d61e59725f18205652b029d253abef36746569e2affd71ee2281/91e25451e514d61e59725f18205652b029d253abef36746569e2affd71ee2281-json.log: read-only file system"
time="2021-05-06T00:30:22.496169171+01:00" level=error msg="Failed to log msg \"\" for logger json-file: write /var/lib/docker/containers/91e25451e514d61e59725f18205652b029d253abef36746569e2affd71ee2281/91e25451e514d61e59725f18205652b029d253abef36746569e2affd71ee2281-json.log: read-only file system"

 

I tried to stop the docker image and it seem to stop fine, and then went to remove it to try a fresh deployment.  When I tried to remove it first via the GUI and then via the command line.  But received an error, the error stated that docker was still using the image.  So shut down all the docker images I had running and then disabled docker and tried again but still got the same error.

 

I thought 'not a problem I'll leave docker disabled and just restart then I would be able to remove the old image and start again'. I paused the VM's I had running (1x Untangled FW and 2x 2016 Server) and rebooted.  However, when the Unraid server came back it was not showing any VM's  via the GUI and said one of the XML was clear.  

 

I've checked the Syslog output and cant see anything other than some warnings about 'page private not zero' and also that it still thinks docker is in use:-

 

May  6 21:22:52 UKBLTSRVUnraid1 kernel: BTRFS warning (device loop2): page private not zero on page 5865603072
May  6 21:22:52 UKBLTSRVUnraid1 kernel: BTRFS warning (device loop2): page private not zero on page 5865607168
May  6 21:22:52 UKBLTSRVUnraid1 kernel: BTRFS warning (device loop2): page private not zero on page 5865611264
May  6 21:22:52 UKBLTSRVUnraid1 kernel: BTRFS warning (device loop2): page private not zero on page 5865615360

 

May  6 21:25:13 UKBLTSRVUnraid1 emhttpd: shcmd (84432): /usr/local/sbin/mount_image '/mnt/user/system/docker/docker.img' /var/lib/docker 40
May  6 21:25:13 UKBLTSRVUnraid1 root: /mnt/user/system/docker/docker.img is in-use, cannot mount
May  6 21:25:13 UKBLTSRVUnraid1 emhttpd: shcmd (84432): exit status: 1

 

Now while losing all my docker images is not ideal, I can live with it but I need to get the VM's back up and running. I've check the file system and what looks like the needed files for the VM are all still there and I've also checked all the HDD's both within Unraid and also the IPMI and BIOS of the Dell server and again all are good so it's not a hardware issue.

 

I've attached the diagnostics to this post ukbltsrvunraid1-diagnostics-20210506-2128.zip in the hope that someone can help me get the VM's back and identify the cause of the issue.  The only thing I know which happen around the same time was that one of the VM systems used all of its allocatetted HDD space, but I cant see how that should/would affect the others.

 

Any help would be very welcome as my once wonderful Unraid server is now a paperweight.

[Squid]

Your cache drive is 100% full which caused the cascade effect. (btrfs doesn't respond well to completely filling it up)

 

Because the array is stopped, can't really tell what shares are existing on the cache drive right now.  

 

Ignore the docker problem right now, figure out why the cache drive is 100% full, take the appropriate steps (run mover / make sure that your VMs aren't over provisioned and the vdisks have expanded to more than the cache drive size etc)

[1979]

9 hours ago, Squid said:

Your cache drive is 100% full which caused the cascade effect. (btrfs doesn't respond well to completely filling it up)

 

Because the array is stopped, can't really tell what shares are existing on the cache drive right now.  

 

Ignore the docker problem right now, figure out why the cache drive is 100% full, take the appropriate steps (run mover / make sure that your VMs aren't over provisioned and the vdisks have expanded to more than the cache drive size etc)

Many thanks for the help.  I've not used 'mover' before is their a MAN/Help page for it, or is it just a case of clicking 'move' under the 'Main' menu?  Sorry if it a stupid question.

 

Edited May 7, 2021 by 1979

[1979]

Okay, so yes it was a stupid question 😁 

 

Move is simple and only needed me to make sure that the shares were set to 'Cache - Yes' and then run 'Move' and after some time it had freed up nearly all of the cache drive and as soon as this was done, and a quick reboot, all my VM's came back and Docker started working again with no further changes.

 

So moral of the story, always make sure you have you cache setting correct and space on your drive.

 

Thanks for the help everyone.