All my dockers disappeared after a restart

[JB4times4]

So I noticed my docker containers shut down last night, so I attempted a restart to fix it and now there are none showing up in my docker tab. I tried another restart to see if it would self resolve but no luck.

 

I've attached my diagnostics, but I'm new to unraid and unsure how to recover my dockers and fix the underlying problem.

 

Any help would be much appreciated.

tower-diagnostics-20200716-1258.zip

[JB4times4]

A quick update, further diagnosis shows that my appdata folder is completely empty. I restored it to a few days ago, and it seems to have restored my dockers but I'm really worried about the underlying problem that caused this in the first place.

[trurl]

Since the diagnostics you posted are without the array started, there are many things we can't know about your situation.

2 hours ago, JB4times4 said:

the underlying problem

Please attach new diagnostics with the array started to your NEXT post.

[JB4times4]

Didn't even know there was a difference! Here are the diagnostics with the array started.

 

Keep in mind I did do a restore of my appdata between these diagnostics and the last if that impacts anything in them.

tower-diagnostics-20200716-1738.zip

Edited July 16, 2020 by JB4times4

[trurl]

How old was that appdata backup your restored? Have you reconfigured the CA Backup plugin recently?

 

The last appdata backup with the CA Backup plugin in the syslog of your earlier diagnostics looks like this:

Jul 16 03:00:05 Tower CA Backup/Restore: Backing Up appData from /mnt/user/appdata/ to /[email protected]
Jul 16 03:00:05 Tower CA Backup/Restore: Using command: cd '/mnt/user/appdata/' && /usr/bin/tar -cvaf '/[email protected]' --exclude 'docker.img'  * >> /var/lib/docker/unraid/ca.backup2.datastore/appdata_backup.log 2>&1 & echo $! > /tmp/ca.backup2/tempFiles/backupInProgress
Jul 16 03:00:05 Tower CA Backup/Restore: Backup Complete
Jul 16 03:00:05 Tower CA Backup/Restore: Verifying backup
Jul 16 03:00:05 Tower CA Backup/Restore: Using command: cd '/mnt/user/appdata/' && /usr/bin/tar --diff -C '/mnt/user/appdata/' -af '/[email protected]' > /var/lib/docker/unraid/ca.backup2.datastore/appdata_backup.log & echo $! > /tmp/ca.backup2/tempFiles/verifyInProgress
Jul 16 03:00:06 Tower CA Backup/Restore: #######################
Jul 16 03:00:06 Tower CA Backup/Restore: appData Backup complete
Jul 16 03:00:06 Tower CA Backup/Restore: #######################

As you can see it completed almost instantly, so that seems to indicate appdata was already gone at that point.

 

I also notice your version of that plugin is out-of-date (as are some others)

 

Unfortunately, diagnostics has anonymized some things in that portion of the log so it isn't clear what the destination of the backup was. It makes me wonder if maybe you haven't misconfigured that plugin so that it was using the appdata folder as the destination of the backup and so wiped it out before beginning the copy.

[JB4times4]

I haven't changed any config since the initial setup. The backup I restored was from the 14th. I've attached my config for CA Backup.

 

I may just move on, and if it happens again try and get the diagnostics right away unless you've got other suggestions/thoughts on the matter.

 

Thanks for your help looking into this!

[trurl]

I also noticed many of your array disks are very full, and you have no cache. Dockers/VMs will perform better and not keep array disks spinning if appdata, domains, system shares are not on the array. Likely all your disks stay spinning since you have those shares all over the array.

[JB4times4]

I think you may have found the problem and I hadn't even noticed. I do actually have a cache drive, but it's not showing up in Unraid.

 

I guess I'll need to do some diagnostics on the drive, and potentially it failed, causing the problem.

[IrishFavor]

Iv had similar issues when my Docker.img files was to small i increased the size and then redeployed the docker container using the same config locations everything imported without issue and i was back up and running, Perhaps you are having a similar issue, also check to see do you have your dockers running from your storage array or your cache drive? i have hears similar issues having to do with disks going to sleep i have not had that issue but that could be another cause. 

[trurl]

15 hours ago, IrishFavor said:

had similar issues when my Docker.img files was to small i increased the size

I recommend NOT increasing docker image beyond 20G unless advised to do so. Filling docker image is almost always caused by misconfigured containers. The most common cause is an application writing to a path that isn't mapped.

 

In this case, losing cache was the cause of losing dockers, since cache is where they are normally configured to be. This hardware problem needs to be resolved before attempting any fix.

[JB4times4]

Forgot to report back and say thanks for the help diagnosing!

 

Somehow the sata cable for my cache drive became slightly unseated, reset that and restored my cache backup and looks like we're golden now!