6.5.0 Docker Tab Unresponsive

[bonienl]

Your container is declared "unhealthy", which is the only difference I see with my container.

"Health": {
                "Status": "unhealthy",

I am not sure how to get it "healthy", perhaps deleting the container+image and re-install may help?

 

[bonienl]

May I ask everybody who had the Docker tab hanging to execute the below command (just copy + paste) and see if this succeeds.

docker inspect --format='{{.State.Running}}#{{range $p,$c := .HostConfig.PortBindings}}{{$p}}:{{(index $c 0).HostPort}}|{{end}}#{{range $p,$c := .Config.ExposedPorts}}{{$p}}|{{end}}#{{range .NetworkSettings.Networks}}{{.IPAddress}}{{end}}#{{range $c := .HostConfig.Binds}}{{$c}}|{{end}}' $(docker ps -a --format='{{.Names}}')

Thanks

[Uncledome]

Just upgraded again to 6.5.0 to test this, but will have to downgrade before sunday evening because I am a week away and cannot restart the server.

Currently it looks to be running and your command just showed me all my containers on true except two, which had false.

 

 

[bonienl]

1 minute ago, Uncledome said:

Just upgraded again to 6.5.0 to test this, but will have to downgrade before sunday evening because I am a week away and cannot restart the server.

Currently it looks to be running and your command just showed me all my containers on true except two, which had false.

 

 

 

Ok, thanks for testing. Please try this command again when you have the "failing" page situation.

 

[Uncledome]

Just now, bonienl said:

 

Ok, thanks for testing. Please try this command again when you have the "failing" page situation.

 

Will do and will let you know when and what happens.

[cschanot]

I am having this issue as well, I am attempting to figure out what is causing the hanging, only recent system change besides upgrading to 6.5.0 was adding another nic and changing to active-backup. Same symptoms as everyone else.

 

 

[cschanot]

4 hours ago, bonienl said:

docker inspect --format='{{.State.Running}}#{{range $p,$c := .HostConfig.PortBindings}}{{$p}}:{{(index $c 0).HostPort}}|{{end}}#{{range $p,$c := .Config.ExposedPorts}}{{$p}}|{{end}}#{{range .NetworkSettings.Networks}}{{.IPAddress}}{{end}}#{{range $c := .HostConfig.Binds}}{{$c}}|{{end}}' $(docker ps -a --format='{{.Names}}')

 

This command and docker stats hang for me as well. Nothing unusually memory or cpu usage wise either. I will report back if I figure anything out.

 

 

[cschanot]

After running docker stats --no-stream on each container separately it only would hang on pi-hole. docker stop and docker kill would not work on the pi-hole container either so I had to kill the process. Once that was done everything was responsive again. I will try to figure out what the issue is with pi-hole tomorrow if I have time as I would like to continue running it. Thanks to bonienl and everyone else in this thread for helping to point me in the right direction.

 

So not sure if this will work long term but I will report back..for 2 days pi-hole has been running with no issues. I changed br0 to bond0 and that was it. I thought it might have something to do with networking after reading about a more general docker issue with hanging because of network settings in the docker. Bond0 works for me as I am running with two network cards bonded as active-backup. This was for the interface variable in pi-hole. Good luck to anyone else having this issue but everything that has been outlined will at least help you identify the docker causing your issue!

Edited March 26, 2018 by cschanot

[michaelmcq]

Just an update from me, with my troublesome docker turned off everything is working fine, it's not Pi-Hole for me but rather this : https://github.com/chrisns/docker-node-sonos-http-api perhaps there are similarilties, although not obviously to me!

I've not had a chance to update to RC2 yet.

[yippy3000]

For me after several rounds with reboots it seems to have fixed itself. The problem hasn't come back in the last 5 days.

[eubbenhadd]

FWIW I have the same timeout issue starting with 6.4.1. Didn't have it on 6.3.x and was hoping 6.5.0 would fix problem.

[local.bin]

On 24/03/2018 at 9:05 PM, bonienl said:

May I ask everybody who had the Docker tab hanging to execute the below command (just copy + paste) and see if this succeeds.

docker inspect --format='{{.State.Running}}#{{range $p,$c := .HostConfig.PortBindings}}{{$p}}:{{(index $c 0).HostPort}}|{{end}}#{{range $p,$c := .Config.ExposedPorts}}{{$p}}|{{end}}#{{range .NetworkSettings.Networks}}{{.IPAddress}}{{end}}#{{range $c := .HostConfig.Binds}}{{$c}}|{{end}}' $(docker ps -a --format='{{.Names}}')

Thanks

 

Just hangs for me on 6.5 and I confirm the other reported observations too.

 

Should I update to rc2 or thats confirmed to still have the issue?

 

Thanks

[bonienl]

The issue is with docker itself. 6.5.1-rc2 uses the same docker version as 6.5.0.

 

The temporary workaround is to find out which container is causing docker to hang and stop using this container.

 

Btw it is worthwhile to update to rc2 because it does have several corrections.

 

Edited March 30, 2018 by bonienl

[local.bin]

Ok I see, thanks, so is there a shortened command of yours above to see which are running and being able to shut them down individually?

 

Mmh ok, I am using the dvb kernel for 6.5 and they don't do rc versions so would prefer to stay on 6.5 unless the updates in rc2 are critical?

 

Thanks in advance

 

Edit: I will just use docker stop <container> && docker ps if thats all thats needed...

 

Edit: I am not able to stop dockers individually, the stop command just hangs there. So will stop the array and see if I can isolate the docker with the issue.

 

Edit: Seems to have all hung, unclean shutdown ....

 

Edit: Going back to 6.4.1 again, as its too laggy to resolve quickly and its a production system, so will keep an eye on further updates, resolving this issue

 

 

Edited March 30, 2018 by local.bin

[Uncledome]

 

On 3/24/2018 at 10:15 PM, bonienl said:

 

Ok, thanks for testing. Please try this command again when you have the "failing" page situation.

 

 

So, after a week away from home and letting the server just run, I have the same issue again.

 

I am currently running 6.5.0 since a week,

here is the error I am receiving in /var/log/syslog

 

Quote

Mar 30 12:00:11 Tower nginx: 2018/03/30 12:00:11 [error] 6328#6328: *817760 upstream timed out (110: Connection timed out) while reading response header from upstream, client: 192.168.XXX.XXX, server: , request: "POST /plugins/dynamix.docker.manager/include/ContainerManager.php HTTP/1.1", upstream: "fastcgi://unix:/var/run/php5-fpm.sock", host: "192.168.XXX.XXX", referrer: "http://192.168.XXX.XXX/Docker"

 

When I run the command you have stated,

On 3/24/2018 at 10:05 PM, bonienl said:

May I ask everybody who had the Docker tab hanging to execute the below command (just copy + paste) and see if this succeeds.

docker inspect --format='{{.State.Running}}#{{range $p,$c := .HostConfig.PortBindings}}{{$p}}:{{(index $c 0).HostPort}}|{{end}}#{{range $p,$c := .Config.ExposedPorts}}{{$p}}|{{end}}#{{range .NetworkSettings.Networks}}{{.IPAddress}}{{end}}#{{range $c := .HostConfig.Binds}}{{$c}}|{{end}}' $(docker ps -a --format='{{.Names}}')

Thanks

 

I get a true on every line of my docker containers.

[bonienl]

14 minutes ago, Uncledome said:

I get a true on every line of my docker containers.

 

Can you show the output? I am not sure what your statement means.

# docker inspect --format='...
true##19999/tcp|#10.0.105.194#/proc:/host/proc:ro|/sys:/host/sys:ro|/mnt/cache/appdata/netdata:/etc/netdata:rw|
true#443/tcp:8443|80/tcp:8080|#443/tcp|80/tcp|#172.17.0.2#/mnt/cache/appdata/heimdall:/config:rw|
true##8080/tcp|#10.0.105.192#/var/lib/docker/:/var/lib/docker/:ro|/:/rootfs:ro|/var/run:/var/run:ro|/sys:/sys:ro|
true##443/tcp|80/tcp|#10.0.101.80#/mnt/cache/appdata/Nginx:/config:rw|
true##53/tcp|53/udp|80/tcp|#10.0.101.99#/mnt/cache/appdata/pihole/pihole/:/etc/pihole/:rw|/mnt/cache/appdata/pihole/dnsmasq.d/:/etc/dnsmasq.d/:rw|
true##8080/tcp|8081/tcp|8443/tcp|8843/tcp|8880/tcp|#10.0.101.7#/mnt/cache/appdata/unifi/:/config:rw|
true##8181/tcp|#10.0.101.192#/mnt/cache/appdata/tautulli:/logs:rw|/mnt/cache/appdata/tautulli:/config:rw|
true##10001/tcp|#10.0.101.193#/mnt/user/:/media:ro|/mnt/cache/appdata/tonido:/config:rw|
true###10.0.105.193#/mnt/user/films/:/Movies:ro|/mnt/cache/handbrake/:/Output:rw|/mnt/cache/appdata/handbrake:/nobody/.config/ghb:rw|
true##1900/udp|7359/udp|8096/tcp|8920/tcp|#10.0.101.82#/mnt/user:/media:ro|/mnt/cache/appdata/Emby:/config:rw|
true##1900/udp|32400/tcp|32400/udp|32469/tcp|32469/udp|5353/udp|#10.0.101.81#/mnt/cache/appdata/plex/:/config:rw|/tmp:/transcode:rw|/mnt/cache/appdata/plex/Backup/:/backup:rw|/mnt/user/:/media:ro|

The first "true" means the container is in a running state. The remainder are the settings & status read for each container.

[Uncledome]

31 minutes ago, bonienl said:

 

Can you show the output? I am not sure what your statement means.

# docker inspect --format='...
true##19999/tcp|#10.0.105.194#/proc:/host/proc:ro|/sys:/host/sys:ro|/mnt/cache/appdata/netdata:/etc/netdata:rw|
true#443/tcp:8443|80/tcp:8080|#443/tcp|80/tcp|#172.17.0.2#/mnt/cache/appdata/heimdall:/config:rw|
true##8080/tcp|#10.0.105.192#/var/lib/docker/:/var/lib/docker/:ro|/:/rootfs:ro|/var/run:/var/run:ro|/sys:/sys:ro|
true##443/tcp|80/tcp|#10.0.101.80#/mnt/cache/appdata/Nginx:/config:rw|
true##53/tcp|53/udp|80/tcp|#10.0.101.99#/mnt/cache/appdata/pihole/pihole/:/etc/pihole/:rw|/mnt/cache/appdata/pihole/dnsmasq.d/:/etc/dnsmasq.d/:rw|
true##8080/tcp|8081/tcp|8443/tcp|8843/tcp|8880/tcp|#10.0.101.7#/mnt/cache/appdata/unifi/:/config:rw|
true##8181/tcp|#10.0.101.192#/mnt/cache/appdata/tautulli:/logs:rw|/mnt/cache/appdata/tautulli:/config:rw|
true##10001/tcp|#10.0.101.193#/mnt/user/:/media:ro|/mnt/cache/appdata/tonido:/config:rw|
true###10.0.105.193#/mnt/user/films/:/Movies:ro|/mnt/cache/handbrake/:/Output:rw|/mnt/cache/appdata/handbrake:/nobody/.config/ghb:rw|
true##1900/udp|7359/udp|8096/tcp|8920/tcp|#10.0.101.82#/mnt/user:/media:ro|/mnt/cache/appdata/Emby:/config:rw|
true##1900/udp|32400/tcp|32400/udp|32469/tcp|32469/udp|5353/udp|#10.0.101.81#/mnt/cache/appdata/plex/:/config:rw|/tmp:/transcode:rw|/mnt/cache/appdata/plex/Backup/:/backup:rw|/mnt/user/:/media:ro|

The first "true" means the container is in a running state. The remainder are the settings & status read for each container.

 

Sure, this is the output of the command:

 

true#8181/tcp:8181|#8181/tcp|#172.17.0.10#/mnt/user/appdata/tautulli:/config:rw|/mnt/user/appdata/PlexMediaServer/Library/Application Support/Plex Media Server/Logs/:/plex_logs:rw|
true##1900/udp|3005/tcp|32400/tcp|32410/udp|32412/udp|32413/udp|32414/udp|32469/tcp|8324/tcp|##/mnt/user/appdata/PlexMediaServer/transcode:/transcode:rw|/mnt/disks/DS22_Media:/data:ro,slave|/mnt/user/appdata/PlexMediaServer:/config:rw|
true#8989/tcp:8989|#8989/tcp|#172.17.0.8#/mnt/disks/DS22_Media//Downloads/NZBs/Complete:/downloads:rw,slave|/mnt/user/appdata/sonarr:/config:rw|/dev/rtc:/dev/rtc:ro|/mnt/disks/DS22_Media/:/tv:rw,slave|
true#7878/tcp:7878|#7878/tcp|#172.17.0.9#/mnt/disks/DS22_Media//Downloads/NZBs/Complete:/downloads:rw,slave|/mnt/disks/DS22_Media/Movies:/movies:rw,slave|/mnt/user/appdata/radarr:/config:rw|
true#443/tcp:444|80/tcp:8383|#443/tcp|80/tcp|#172.17.0.13#/mnt/user/appdata/heimdall:/config:rw|
true#8090/tcp:8090|#8090/tcp|8091/tcp|#172.17.0.14#/mnt/user/appdata/confluence:/var/atlassian/application-data/confluence:rw,slave|
true#80/tcp:8282|#443/tcp|80/tcp|#172.17.0.4#/mnt/user/appdata/organizr:/config:rw|
true#5076/tcp:5076|#5075/tcp|5076/tcp|#172.17.0.6#/mnt/user/appdata/hydra2:/config:rw|/mnt/disks/DS22_Media/Downloads/NZBs/Complete:/downloads:ro,slave|
true##58846/tcp|58946/tcp|58946/udp|8112/tcp|##/mnt/user/Torrent Downloads/:/downloads:rw|/mnt/user/appdata/deluge:/config:rw|
true##10011/tcp|30033/tcp|41144/tcp|9987/udp|##/mnt/user/appdata/teamspeak3:/config:rw|
true#8080/tcp:8080|9090/tcp:9090|#8080/tcp|9090/tcp|#172.17.0.7#/mnt/disks/DS22_Media/Downloads/NZBs/Complete:/downloads:rw,slave|/mnt/disks/DS22_Media/Downloads/NZBs/Incomplete:/incomplete-downloads:rw,slave|/mnt/user/appdata/sabnzbd:/config:rw|
true#8181/tcp:8182|#8181/tcp|#172.17.0.2#/mnt/disks/DS22_Media/Downloads/NZBs/Complete:/downloads:rw,slave|/mnt/disks/DS22_Media/Music:/music:rw,slave|/mnt/user/appdata/headphones:/config:rw|
true#5800/tcp:7803|5900/tcp:7903|#5800/tcp|5900/tcp|#172.17.0.11#/mnt/user/Handbrake/:/storage:ro|/mnt/user/Handbrake/watch:/watch:rw|/mnt/user/Handbrake/output:/output:rw|/mnt/user/appdata/HandBrake:/config:rw|
true#8080/tcp:8082|#3389/tcp|8080/tcp|#172.17.0.12#/mnt/user/JDownloader Downloads/:/downloads:rw|/mnt/user/appdata/JDownloader2:/config:rw|
true#5432/tcp:5432|#5432/tcp|#172.17.0.5#/mnt/cache/appdata/postgresql:/var/lib/postgresql:rw|
true#8080/tcp:8081|#8080/tcp|#172.17.0.3#/:/rootfs:ro|/var/run:/var/run:rw|/sys:/sys:ro|/var/lib/docker/:/var/lib/docker/:ro|

is there a way to restart the server without forcing it down through the power button?

Web UI is not accessible at the moment, only SSH.

 

reboot & powerdown -r as well as shutdown -hP now does not work.

Edited March 30, 2018 by Uncledome

[bonienl]

To restart the GUI try

/etc/rc.d/rc.nginx restart
/etc/rc.d/rc.php-fpm restart

If you don't manage to bring the GUI back, can you SSH in and do diagnostics before you reboot?

Edited March 30, 2018 by bonienl

[Uncledome]

4 minutes ago, bonienl said:

To restart the GUI try

/etc/rc.d/rc.nginx restart
/etc/rc.d/rc.php-fpm restart

 

 

Tried both, first command told me nginx was not running so I used the start parameter. Afterwards it told me that the daemon is starting, status parameter confirmed this.

the php-fpm could not be restarted, even with a force-quit, it just failed.

[bonienl]

14 minutes ago, Uncledome said:

 

Tried both, first command told me nginx was not running so I used the start parameter. Afterwards it told me that the daemon is starting, status parameter confirmed this.

the php-fpm could not be restarted, even with a force-quit, it just failed.

 

Need diagnostics to investigate further, but not certain the cause can be revealed.

 

There is a kernel patch included in unRAID 6.5.1-rc2 which solves a TCP bug. This bug caused some inexplainable application hangs. It is worthwhile to upgrrade to rc2.

 

[Uncledome]

12 minutes ago, bonienl said:

 

Need diagnostics to investigate further, but not certain the cause can be revealed.

 

There is a kernel patch included in unRAID 6.5.1-rc2 which solves a TCP bug. This bug caused some inexplainable application hangs. It is worthwhile to upgrrade to rc2.

 

 

How can I send you the diagnostic without having access to WebUI?

I still can access with SSH & WinSCP.

[MyKroFt]

I am not home atm to post diags, but when this happens to me, if I manually kill the docker daemon, everything comes back, as I get this hang on community apps looking for updates and the log complains about php-fpm as well as the docker page

 

Myk

[bonienl]

18 minutes ago, Uncledome said:

 

How can I send you the diagnostic without having access to WebUI?

I still can access with SSH & WinSCP.

 

When you SSH in type:

diagnostics

This will create a zip file in the /logs folder on you USB device.

Normally the USB device is remotely accessible with SMB, e.g. //tower/flash

 

[Uncledome]

50 minutes ago, bonienl said:

 

When you SSH in type:

diagnostics

This will create a zip file in the /logs folder on you USB device.

Normally the USB device is remotely accessible with SMB, e.g. //tower/flash

 

 

Okay, I started the diagnostics a few minutes ago when you posted how to do it.

Looks like it us currently still running so I am guessing this might take a while. (at the moment: Starting diagnostics collection...)

 

I'll edit this post once it is finished and I got the zip folder.

 

//EDIT1:

Okay, I have waited  a few minutes and still nothing is happening.

Is there a possibility to force the server to restart without losing the validity of my parity?

Otherwise I'll have to force it down using the power button.

 

//EDIT2:

Okay, I've went with the force reboot. Had to reboot it afterwards again because the Passphrase Box was missing...

The Diagnostic should be attached. (Made the Diag after reboot)

tower-diagnostics-20180330-0552.zip

Edited March 30, 2018 by Uncledome

[Uncledome]

Okay, Server has issues again. Guess I'll be downgrading to 6.4.1 as I had no issues there.

 

Just for your info:

Quote

Mar 30 08:40:12 Tower nginx: 2018/03/30 08:40:12 [error] 6209#6209: *28369 upstream timed out (110: Connection timed out) while reading response header from upstream, client: 192.168.XXX.XXX, server: , request: "POST /webGui/include/DashUpdate.php HTTP/1.1", upstream: "fastcgi://unix:/var/run/php5-fpm.sock", host: "192.168.XXX.XXX", referrer: "http://192.168.XXX.XXX/Dashboard"

 

Diagnostic won't work, wether trying through WebUI nor directly from CLI. Just hangs right after "Starting diagnostic collection".

 

As for the Docker command:

true#8181/tcp:8181|#8181/tcp|#172.17.0.9#/mnt/user/appdata/tautulli:/config:rw|/mnt/user/appdata/PlexMediaServer/Library/Application Support/Plex Media Server/Logs/:/plex_logs:rw|
true##1900/udp|3005/tcp|32400/tcp|32410/udp|32412/udp|32413/udp|32414/udp|32469/tcp|8324/tcp|##/mnt/user/appdata/PlexMediaServer/transcode:/transcode:rw|/mnt/disks/DS22_Media:/data:ro,slave|/mnt/user/appdata/PlexMediaServer:/config:rw|
true#8989/tcp:8989|#8989/tcp|#172.17.0.7#/mnt/disks/DS22_Media//Downloads/NZBs/Complete:/downloads:rw,slave|/mnt/user/appdata/sonarr:/config:rw|/dev/rtc:/dev/rtc:ro|/mnt/disks/DS22_Media/:/tv:rw,slave|
true#7878/tcp:7878|#7878/tcp|#172.17.0.8#/mnt/disks/DS22_Media//Downloads/NZBs/Complete:/downloads:rw,slave|/mnt/disks/DS22_Media/Movies:/movies:rw,slave|/mnt/user/appdata/radarr:/config:rw|
true#443/tcp:444|80/tcp:8383|#443/tcp|80/tcp|#172.17.0.12#/mnt/user/appdata/heimdall:/config:rw|
true#8090/tcp:8090|#8090/tcp|8091/tcp|#172.17.0.13#/mnt/user/appdata/confluence:/var/atlassian/application-data/confluence:rw,slave|
true#80/tcp:8282|#443/tcp|80/tcp|#172.17.0.3#/mnt/user/appdata/organizr:/config:rw|
true#5076/tcp:5076|#5075/tcp|5076/tcp|#172.17.0.5#/mnt/user/appdata/hydra2:/config:rw|/mnt/disks/DS22_Media/Downloads/NZBs/Complete:/downloads:ro,slave|
false##58846/tcp|58946/tcp|58946/udp|8112/tcp|##/mnt/user/Torrent Downloads/:/downloads:rw|/mnt/user/appdata/deluge:/config:rw|
true##10011/tcp|30033/tcp|41144/tcp|9987/udp|##/mnt/user/appdata/teamspeak3:/config:rw|
true#8080/tcp:8080|9090/tcp:9090|#8080/tcp|9090/tcp|#172.17.0.6#/mnt/disks/DS22_Media/Downloads/NZBs/Complete:/downloads:rw,slave|/mnt/disks/DS22_Media/Downloads/NZBs/Incomplete:/incomplete-downloads:rw,slave|/mnt/user/appdata/sabnzbd:/config:rw|
false#8181/tcp:8182|#8181/tcp|#172.17.0.2#/mnt/disks/DS22_Media/Downloads/NZBs/Complete:/downloads:rw,slave|/mnt/disks/DS22_Media/Music:/music:rw,slave|/mnt/user/appdata/headphones:/config:rw|
true#5800/tcp:7803|5900/tcp:7903|#5800/tcp|5900/tcp|#172.17.0.10#/mnt/user/Handbrake/:/storage:ro|/mnt/user/Handbrake/watch:/watch:rw|/mnt/user/Handbrake/output:/output:rw|/mnt/user/appdata/HandBrake:/config:rw|
true#8080/tcp:8082|#3389/tcp|8080/tcp|#172.17.0.11#/mnt/user/JDownloader Downloads/:/downloads:rw|/mnt/user/appdata/JDownloader2:/config:rw|
true#5432/tcp:5432|#5432/tcp|#172.17.0.4#/mnt/cache/appdata/postgresql:/var/lib/postgresql:rw|
true#8080/tcp:8081|#8080/tcp|#172.17.0.2#/:/rootfs:ro|/var/run:/var/run:rw|/sys:/sys:ro|/var/lib/docker/:/var/lib/docker/:ro|

 

Deluge and Headphones were offline from the start as they do not automatically start with server startup (i disabled them).

Edited March 30, 2018 by Uncledome