Cache mounted RO / Docker Failed to start [SOLVED]

[1activegeek]

Not sure what happened last night but something went wonky in the middle of the night. I got notification that my services weren't responding then were, but my site pings were failing. Checked this AM, nothing seems to be up, and docker daemon not responding. Restarted the host and now it seems the cache is in a RO mounted state and docker won't start (obviously). Diagnostics posted below - any help would be greatly appreciated to get this back up and running. 

atlantis-diagnostics-20200213-0955.zip

Edited March 1, 2020 by 1activegeek
adding solved notation

[1activegeek]

Just in case, this is also the diagnostics I grabbed after running myself manually after the boot with RO mode

atlantis-diagnostics-20200213-1014.zip

[JorgeB]

Docker image is corrupt, delete and re-create, cache filesystem looks OK for now.

 

P.S. there were several sync errors during last parity check, were those expected?

[JorgeB]

Forgot to mention, there are several ATA errors on the parity disk, looks like a connection issue, check/replace cables.

[1activegeek]

I feel really blind here. I thought that would be the "quick" resolution. Unfortunately having deleted the docker image, restarted multiple times, unmount and mounting again - I still get the same message that the docker service failed to start. Updated diagnostics attached.

 

PS - is there something in particular I can start searching for if running into these issue? Perhaps a FAQ around troubleshooting for unRAID as a general rule? Hate relying on volunteers to problem solve. 

 

atlantis-diagnostics-20200213-2045.zip

[1activegeek]

It still appears to be mounting as a read only filesystem, thus the delete of the docker.img doesn't seem to be working. I checked locally via an ssh session to the server after attempting to delete the docker.img and it's still there. Manual attempt and it tell me it is a RO filesystem. 

[JorgeB]

I missed it earlier but there's also corruption on the cache filesystem, best way forward is to backup and reformat cache, then recreate docker image once more.

[1activegeek]

Thanks for the direction. Took a bit of work to make my way through it, but did get the disk wiped and reformatted. When I went to delete the docker image (from the backup I made of the cache and put back) there was a notice that the docker image was corrupted from something in a previous Beta. I should have screenshot the message. Deleted and now in the process of rebuilding my docker setup. Unfortunately somehow one of my custom networks was deleted so the containers defaulted to none for network. 🙁 But thank you again for the swift responses and assistance. I'm almost back up and running.

[Squid]

4 minutes ago, 1activegeek said:

process of rebuilding my docker setup.

Since you said this the way you did, you're probably doing it the hard way.  Easy way is here

https://forums.unraid.net/topic/57181-docker-faq/#comment-564309

6 minutes ago, 1activegeek said:

Unfortunately somehow one of my custom networks was deleted so the containers defaulted to none for network

That happens because the network information is stored within the docker.img.

 

7 minutes ago, 1activegeek said:

a notice that the docker image was corrupted from something in a previous Beta.

Known "issue" from moving around the docker.img file.  As you've already noticed, it's easiest to simply recreate it, and then reinstall everything.

[1activegeek]

@Squid The only thing I’m confused about though, is that my custom macvlan network survived, but the custom bridge did not. So that inconsistency threw me.
 

As for recreating, I did go that route - prob is the custom bridge wasn’t there and thus many of the containers didn’t have the right setup. So consequently the easiest option was Removing them all (manual work) and then re-create again with previous after I brought over backups of my container xml templates from a backup. Just lot of steps to get it right. Appreciate the guidance though, thank you!