Docker Containers Unstable - s6-sync hangs

[lqid10t]

This will be the third time I've attempted to reach out to support to address this issue, with no resolution and the last attempt I didn't even get a response.

 

Quite a few of my docker containers (which is about the only thing I use Unraid for now) get in a hung state, typically with s6-sync in a zombie state and maxing out a core on my CPUs. Sometimes I can get a day out of them, sometimes less. Sometimes I can get weeks. The instability is literally making me lose my mind, as I have investigated every single aspect outside Unraid (since I'm not good with troubleshooting this OS). Currently, the main docker container that is now doing this, is the nginx-proxy-manager container. I used to have issues with Sonarr/Radarr and anything that touched the SMB shares mounted using the unassigned devices plugin. I believe I have found the source of at least the issues with those containers, being that the shares those dockers used, were hosted on a TrueNAS server using SMB, with Asynchronous writes disabled. Once I re-enabled that, the issues at least on those dockers seems to have DRASTICALLY decreased.

 

Now, the docker I have the most issues with is nginx-proxy-manager. I cannot host my websites if this continually goes down, and requires a reboot of Unraid to resolve. Unless I'm mistaken, this docker should be nowhere near these SMB shares, which leads me to believe that was never the problem in the first place. If this was just happening with nginx-proxy-manager, I would reach out specifically to that dev, but this has been a problem for me for months with no resolution.

I initially had my array setup with 12 SAS drives, and thought that may be the cause. I have since replace those and have installed 4 SATA drives and still have the same issue. I run a cache drive to stage everything at a faster read/write speed, and to also store my dockers on. This reports no issues, nor can I find anything in the logs to suggest it with this device. I have shut down nearly all non-essential dockers at this point, and it still seems to occur. Short of attempting to migrate this to a new set of server hardware and starting the USB stick from scratch, I can't seem to figure out any leads on what may be causing this.

mut7app201-diagnostics-20201129-0129.zip

[JorgeB]

Not necessarily related but you're getting a lot of Power-on or device reset occurred on multiple disks, this is usually a power/connection problem, also a good idea to update the LSI firmware since it's very old.

[lqid10t]

I replaced the SAS drives and fully removed them I believe later on the 25th. Depending on what times you're looking at, it may show the removal of those drives and the failure to connect to them (hot-swappable bays). I pulled drives from the array multiple times as I made room and was able to remove them to install the newer SATA drives.

In doing this, I also ran my cache disk out of space and it never started dumping to the array, so I had to rebuild the docker.img file from scratch.

[lqid10t]

Also, I need to add that I've attempted to upgrade to the newest beta, but I absolutely need Active Directory to work, and it still didn't seem to connect the last version I tried.

[JorgeB]

1 hour ago, lqid10t said:

I replaced the SAS drives and fully removed them I believe later on the 25th.

The errors were from SATA array devices, e.g. disk1 and disk2 for a long periods of time (1 hour), also don't forget to update the LSI firmware.

[lqid10t]

I plan on moving these drives off the SAS controller anyways, so shouldn't be an issue. Went from having 12 drives to 4, so I no longer need the controller.

[JorgeB]

Then you should reboot and post new diags after that's fixed, difficult to look for issues in a syslog filled with other errors, related or not.

[Healzangels]

Hey lqid10t did you ever find a solution to Npm S6-SYNC hang. My instance is doing the exact same thing and going crazy trying to figure out why. Thanks!