6.5.0 Docker Tab Unresponsive


Recommended Posts

8 minutes ago, rix said:

Its really all about not being displayed in the frontend.

 

The frontend calls the docker service to obtain status information. It seems the docker service itself gets stuck. I have no clue why, because I can't recreate this on any of my systems.

 

Hence my question if you stop all containers from running, do you experience still the same issue?

 

Link to comment

Is there a docker command to stop them all? I can’t do that at the minute, but will try over the weekend at some point.

 

As an aside I ran “docker stats” from ssh and it doesn’t return anything, have to ctrl c to get a prompt back

 

“docker ps” does return successfully  

Link to comment

For the last 3 hours I have been watching the episodes of Planet Earth 2, streamed to my tv using Plex. At the same time another Plex video stream goes to my PC and a third Emby video stream goes to yet another PC. Meanwhile 10 container web GUIs are opened to serve requests.

 

And of course the unRAID GUI itself is open. No crashes, no hangups, all works perfect when clicking thru the GUI and obtaining all kinds of page information ¬¬

 

Link to comment

Can’t stop them all, one of them, a node docker for the Sonos api that I added wouldn’t stop. The others all have.

 

I’ve subsequently tried to restart the docker service with 

/etc/rc.d/rc.docker restart

but that’s not working, looks like I might be doing a forced reboot in a bit and then disabling that particular docker

Link to comment
9 hours ago, bdillahu said:

 

All but one of them.. "minio-unraid" won't stop...

 

I'm not docker expert, so not quite sure how to debug, but willing to try if you can tell me things to look for.

For me it is diginc/pi-hole 

 

Last login: Fri Mar 23 07:25:31 2018
Linux 4.14.26-unRAID.
root@ninja:~# docker stop $(docker ps -a --filter status='running' --format='{{.Names}}')
Ubooquity
Sonarr-Processing
SABnzbd-VPN
Plex
Organizr
Jackett
Headphones
Duplicati
HandBrake
Deluge-VPN
HomeAssistant
Tautulli
RSScrawler
MKVToolNix
JDownloader
Avidemux
DNScrypt
Verarbeitung
SyncLounge
Ripper
^C
root@ninja:~# docker ps
CONTAINER ID        IMAGE               COMMAND             CREATED             STATUS                PORTS               NAMES
80680360fa8f        diginc/pi-hole      "/s6-init"          3 weeks ago         Up 2 days (healthy)                       Pi-Hole
root@ninja:~# docker inspect Pi-Hole
^C
root@ninja:~#

 

 

pkill -f docker 

Has worked, now updated / rebooting to the latest rc

Edited by rix
Link to comment

Just as I posted it happened again.

 

Mar 24 11:46:26 ninja nginx: 2018/03/24 11:46:26 [error] 6302#6302: *15078 upstream timed out (110: Connection timed out) while reading response header from upstream, client: XX.XX.XX.XX, server: , request: "POST /webGui/include/DashboardApps.php HTTP/2.0", upstream: "fastcgi://unix:/var/run/php5-fpm.sock", host: "XXYYZZ.unraid.net", referrer: "https://XXYYZZ.unraid.net/Dashboard"

This time running the 6.5.1 RC2

Link to comment
5 minutes ago, rix said:

Just as I posted it happened again.

 


Mar 24 11:46:26 ninja nginx: 2018/03/24 11:46:26 [error] 6302#6302: *15078 upstream timed out (110: Connection timed out) while reading response header from upstream, client: XX.XX.XX.XX, server: , request: "POST /webGui/include/DashboardApps.php HTTP/2.0", upstream: "fastcgi://unix:/var/run/php5-fpm.sock", host: "XXYYZZ.unraid.net", referrer: "https://XXYYZZ.unraid.net/Dashboard"

This time running the 6.5.1 RC2

 

If you wait long enough, does the list of Docker containers eventually appear or you keep a spinner on that page?

 

Edited by bonienl
Link to comment
13 minutes ago, bonienl said:

 

If you wait long enough, does the list of Docker containers eventually appear or you keep a spinner on that page?

 

Infinite spinner across Firefox/Edge,.. also checked using private tabs.

 

The same error (see above) appeared four times now.

 

To me it seems that after a minute of timeout it fails to connect again:

Mar 24 12:00:19 ninja nginx: 2018/03/24 12:00:19 [error] 6302#6302: *17385 upstream timed out (110: Connection timed out) while reading response header from upstream, client: XX.XX.XX.XX, server: , request: "POST /plugins/dynamix.docker.manager/include/DockerUpdate.php HTTP/2.0", upstream: "fastcgi://unix:/var/run/php5-fpm.sock", host: "XXYYZZ.unraid.net", referrer: "https://XXYYZZ.unraid.net/Docker"
Mar 24 12:00:19 ninja nginx: 2018/03/24 12:00:19 [error] 6302#6302: *17385 upstream timed out (110: Connection timed out) while reading response header from upstream, client: XX.XX.XX.XX, server: , request: "POST /plugins/ca.docker.autostart/include/exec.php HTTP/2.0", upstream: "fastcgi://unix:/var/run/php5-fpm.sock", host: "XXYYZZ.unraid.net", referrer: "https://XXYYZZ.unraid.net/Docker"
Mar 24 12:02:20 ninja nginx: 2018/03/24 12:02:20 [error] 6302#6302: *18397 upstream timed out (110: Connection timed out) while reading response header from upstream, client: XX.XX.XX.XX, server: , request: "POST /plugins/dynamix.docker.manager/include/DockerUpdate.php HTTP/2.0", upstream: "fastcgi://unix:/var/run/php5-fpm.sock", host: "XXYYZZ.unraid.net", referrer: "https://XXYYZZ.unraid.net/Docker"
Mar 24 12:02:20 ninja nginx: 2018/03/24 12:02:20 [error] 6302#6302: *18397 upstream timed out (110: Connection timed out) while reading response header from upstream, client: XX.XX.XX.XX, server: , request: "POST /plugins/ca.docker.autostart/include/exec.php HTTP/2.0", upstream: "fastcgi://unix:/var/run/php5-fpm.sock", host: "XXYYZZ.unraid.net", referrer: "https://XXYYZZ.unraid.net/Docker"
Mar 24 12:03:02 ninja nginx: 2018/03/24 12:03:02 [error] 6302#6302: *18528 upstream timed out (110: Connection timed out) while reading response header from upstream, client: XX.XX.XX.XX, server: , request: "POST /plugins/dynamix.docker.manager/include/DockerUpdate.php HTTP/2.0", upstream: "fastcgi://unix:/var/run/php5-fpm.sock", host: "XXYYZZ.unraid.net", referrer: "https://XXYYZZ.unraid.net/Docker"
Mar 24 12:03:02 ninja nginx: 2018/03/24 12:03:02 [error] 6302#6302: *18528 upstream timed out (110: Connection timed out) while reading response header from upstream, client: XX.XX.XX.XX, server: , request: "POST /plugins/ca.docker.autostart/include/exec.php HTTP/2.0", upstream: "fastcgi://unix:/var/run/php5-fpm.sock", host: "XXYYZZ.unraid.net", referrer: "https://XXYYZZ.unraid.net/Docker"

Should I try restarting the php5 process as described above?

 

EDiT1: Currently running 2 transcodes inside my containers so I wont reboot immediately, but I now know how to reboot cleanly (killing all docker-related processes, then rebooting).

 

EDiT2: Related JS-Error:

The connection to https://XXYYZZ.unraid.net/sub/var was interrupted while the page was loading. (dynamix.js:34:10380)

error.thumb.png.cc2dbdf4d66e67118d52fa28ef41e394.png

Edited by rix
Link to comment

No the php-fpm process is not the problem.

 

If you don't mind some 'hacking'.

In the file  /usr/local/emhttp/plugins/dynamix.docker.manager/include/DockerClient.php is the following statement at the near top.

return exec("timeout 20 /usr/bin/docker $cmd 2>/dev/null",$var);

Can you change the timeout value of 20 to 5.

After the change simply reload the Docker page in your browser.

Edited by bonienl
Link to comment
24 minutes ago, bonienl said:

No the php-fpm process is not the problem.

 

If you don't mind some 'hacking'.

In the file  /usr/local/emhttp/plugins/dynamix.docker.manager/include/DockerClient.php is the following statement at the near top.


return exec("timeout 20 /usr/bin/docker $cmd 2>/dev/null",$var);

Can you change the timeout value of 20 to 5.

After the change simply reload the Docker page in your browser.

 

I tweaked the timeout and saved the file:

changed_php.JPG.04bc5e845b7ee19d7a6b7099e513f3be.JPG

 

The spinner persists.

 

Further: The php5-fpm.sock error does not pop up on loading the Docker page, even though it seems overall related. It appeared ~3 minutes after loading the Docker page.

 

Here are the latest entries in my log:

Mar 24 12:30:29 ninja kernel: TCP: request_sock_TCP: Possible SYN flooding on port 8181. Sending cookies. Check SNMP counters.
Mar 24 12:32:47 ninja kernel: mdcmd (53): spindown 4
Mar 24 12:35:07 ninja sshd[9358]: Accepted password for root from XX.XX.XX.XX port 52543 ssh2
Mar 24 12:41:02 ninja nginx: 2018/03/24 12:41:02 [error] 6302#6302: *23512 upstream timed out (110: Connection timed out) while reading response header from upstream, client: XX.XX.XX.XX, server: , request: "POST /plugins/dynamix.docker.manager/include/DockerUpdate.php HTTP/2.0", upstream: "fastcgi://unix:/var/run/php5-fpm.sock", host: "XXYYZZ.unraid.net", referrer: "https://XXYYZZ.unraid.net/Docker"
Mar 24 12:41:02 ninja nginx: 2018/03/24 12:41:02 [error] 6302#6302: *23512 upstream timed out (110: Connection timed out) while reading response header from upstream, client: XX.XX.XX.XX, server: , request: "POST /plugins/ca.docker.autostart/include/exec.php HTTP/2.0", upstream: "fastcgi://unix:/var/run/php5-fpm.sock", host: "XXYYZZ.unraid.net", referrer: "https://XXYYZZ.unraid.net/Docker"

 

The SYN flooding one is new to me.

Link to comment
3 hours ago, rix said:

The SYN flooding one is new to me.

 

Harmless, this comes from Tautulli, I see this as well.

 

I have installed & configured pi-hole, but can not recreate the issue. Pi-hole is happily running (and blocking ads), I can start/stop it without issue. Docker page stays accessible all the time.

Link to comment
4 hours ago, bonienl said:

 

Harmless, this comes from Tautulli, I see this as well.

 

I have installed & configured pi-hole, but can not recreate the issue. Pi-hole is happily running (and blocking ads), I can start/stop it without issue. Docker page stays accessible all the time.

OK, I have kept pi-hole stopped for now. The issue has since been resolved. 

 

That doesn't solve its cause but I have no idea how to get to the bottom of this as of now. 

 

Thanks for your support! 

Link to comment

Sure thing:

 

root@ninja:~# docker inspect Pi-Hole
[
    {
        "Id": "80680360fa8faa1bc88db1c2734502f98adfd4bea62f261673e24f4bf6533b01",
        "Created": "2018-02-27T03:08:50.116377106Z",
        "Path": "/s6-init",
        "Args": [],
        "State": {
            "Status": "exited",
            "Running": false,
            "Paused": false,
            "Restarting": false,
            "OOMKilled": false,
            "Dead": false,
            "Pid": 0,
            "ExitCode": 0,
            "Error": "",
            "StartedAt": "2018-03-24T14:44:41.38719067Z",
            "FinishedAt": "2018-03-24T14:45:27.583023854Z",
            "Health": {
                "Status": "unhealthy",
                "FailingStreak": 0,
                "Log": [
                    {
                        "Start": "2018-03-24T11:42:11.763336777+01:00",
                        "End": "2018-03-24T11:42:22.104227867+01:00",
                        "ExitCode": 0,
                        "Output": "\n; <<>> DiG 9.10.3-P4-Debian <<>> @127.0.0.1 pi.hole\n; (1 server found)\n;; global options: +cmd\n;; Got answer:\n;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 56472\n;; flags: qr aa rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 1\n\n;; OPT PSEUDOSECTION:\n; EDNS: version: 0, flags:; udp: 4096\n;; QUESTION SECTION:\n;pi.hole.\t\t\tIN\tA\n\n;; ANSWER SECTION:\npi.hole.\t\t300\tIN\tA\t10.10.20.40\n\n;; Query time: 1 msec\n;; SERVER: 127.0.0.1#53(127.0.0.1)\n;; WHEN: Sat Mar 24 11:42:22 CET 2018\n;; MSG SIZE  rcvd: 52\n\n"
                    },
                    {
                        "Start": "2018-03-24T11:42:52.531040003+01:00",
                        "End": "2018-03-24T11:42:52.625293238+01:00",
                        "ExitCode": 0,
                        "Output": "\n; <<>> DiG 9.10.3-P4-Debian <<>> @127.0.0.1 pi.hole\n; (1 server found)\n;; global options: +cmd\n;; Got answer:\n;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 29373\n;; flags: qr aa rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 1\n\n;; OPT PSEUDOSECTION:\n; EDNS: version: 0, flags:; udp: 4096\n;; QUESTION SECTION:\n;pi.hole.\t\t\tIN\tA\n\n;; ANSWER SECTION:\npi.hole.\t\t300\tIN\tA\t10.10.20.40\n\n;; Query time: 1 msec\n;; SERVER: 127.0.0.1#53(127.0.0.1)\n;; WHEN: Sat Mar 24 11:42:52 CET 2018\n;; MSG SIZE  rcvd: 52\n\n"
                    },
                    {
                        "Start": "2018-03-24T11:43:28.891350036+01:00",
                        "End": "2018-03-24T11:43:29.018972396+01:00",
                        "ExitCode": 0,
                        "Output": "\n; <<>> DiG 9.10.3-P4-Debian <<>> @127.0.0.1 pi.hole\n; (1 server found)\n;; global options: +cmd\n;; Got answer:\n;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 7335\n;; flags: qr aa rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 1\n\n;; OPT PSEUDOSECTION:\n; EDNS: version: 0, flags:; udp: 4096\n;; QUESTION SECTION:\n;pi.hole.\t\t\tIN\tA\n\n;; ANSWER SECTION:\npi.hole.\t\t300\tIN\tA\t10.10.20.40\n\n;; Query time: 3 msec\n;; SERVER: 127.0.0.1#53(127.0.0.1)\n;; WHEN: Sat Mar 24 11:43:29 CET 2018\n;; MSG SIZE  rcvd: 52\n\n"
                    },
                    {
                        "Start": "2018-03-24T11:44:00.160101955+01:00",
                        "End": "2018-03-24T11:44:30.1610434+01:00",
                        "ExitCode": -1,
                        "Output": "Health check exceeded timeout (30s)"
                    },
                    {
                        "Start": "2018-03-24T15:45:11.387398931+01:00",
                        "End": "2018-03-24T15:45:11.434955272+01:00",
                        "ExitCode": 0,
                        "Output": "\n; <<>> DiG 9.10.3-P4-Debian <<>> @127.0.0.1 pi.hole\n; (1 server found)\n;; global options: +cmd\n;; Got answer:\n;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 14818\n;; flags: qr aa rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 1\n\n;; OPT PSEUDOSECTION:\n; EDNS: version: 0, flags:; udp: 4096\n;; QUESTION SECTION:\n;pi.hole.\t\t\tIN\tA\n\n;; ANSWER SECTION:\npi.hole.\t\t300\tIN\tA\t10.10.20.40\n\n;; Query time: 0 msec\n;; SERVER: 127.0.0.1#53(127.0.0.1)\n;; WHEN: Sat Mar 24 15:45:11 CET 2018\n;; MSG SIZE  rcvd: 52\n\n"
                    }
                ]
            }
        },
        "Image": "sha256:bb640716ae33ec36f3c2d359443e4afa40461dc5981c1716987655d8b3987628",
        "ResolvConfPath": "/var/lib/docker/containers/80680360fa8faa1bc88db1c2734502f98adfd4bea62f261673e24f4bf6533b01/resolv.conf",
        "HostnamePath": "/var/lib/docker/containers/80680360fa8faa1bc88db1c2734502f98adfd4bea62f261673e24f4bf6533b01/hostname",
        "HostsPath": "/var/lib/docker/containers/80680360fa8faa1bc88db1c2734502f98adfd4bea62f261673e24f4bf6533b01/hosts",
        "LogPath": "/var/lib/docker/containers/80680360fa8faa1bc88db1c2734502f98adfd4bea62f261673e24f4bf6533b01/80680360fa8faa1bc88db1c2734502f98adfd4bea62f261673e24f4bf6533b01-json.log",
        "Name": "/Pi-Hole",
        "RestartCount": 0,
        "Driver": "btrfs",
        "Platform": "linux",
        "MountLabel": "",
        "ProcessLabel": "",
        "AppArmorProfile": "",
        "ExecIDs": null,
        "HostConfig": {
            "Binds": [
                "/mnt/user/appdata/pihole/pihole/:/etc/pihole/:rw",
                "/mnt/user/appdata/pihole/dnsmasq.d/:/etc/dnsmasq.d/:rw"
            ],
            "ContainerIDFile": "",
            "LogConfig": {
                "Type": "json-file",
                "Config": {}
            },
            "NetworkMode": "br0",
            "PortBindings": {},
            "RestartPolicy": {
                "Name": "no",
                "MaximumRetryCount": 0
            },
            "AutoRemove": false,
            "VolumeDriver": "",
            "VolumesFrom": null,
            "CapAdd": [
                "NET_ADMIN"
            ],
            "CapDrop": null,
            "Dns": [],
            "DnsOptions": [],
            "DnsSearch": [],
            "ExtraHosts": null,
            "GroupAdd": null,
            "IpcMode": "shareable",
            "Cgroup": "",
            "Links": null,
            "OomScoreAdj": 0,
            "PidMode": "",
            "Privileged": false,
            "PublishAllPorts": false,
            "ReadonlyRootfs": false,
            "SecurityOpt": null,
            "UTSMode": "",
            "UsernsMode": "",
            "ShmSize": 67108864,
            "Runtime": "runc",
            "ConsoleSize": [
                0,
                0
            ],
            "Isolation": "",
            "CpuShares": 0,
            "Memory": 0,
            "NanoCpus": 0,
            "CgroupParent": "",
            "BlkioWeight": 0,
            "BlkioWeightDevice": [],
            "BlkioDeviceReadBps": null,
            "BlkioDeviceWriteBps": null,
            "BlkioDeviceReadIOps": null,
            "BlkioDeviceWriteIOps": null,
            "CpuPeriod": 0,
            "CpuQuota": 0,
            "CpuRealtimePeriod": 0,
            "CpuRealtimeRuntime": 0,
            "CpusetCpus": "",
            "CpusetMems": "",
            "Devices": [],
            "DeviceCgroupRules": null,
            "DiskQuota": 0,
            "KernelMemory": 0,
            "MemoryReservation": 0,
            "MemorySwap": 0,
            "MemorySwappiness": null,
            "OomKillDisable": false,
            "PidsLimit": 0,
            "Ulimits": null,
            "CpuCount": 0,
            "CpuPercent": 0,
            "IOMaximumIOps": 0,
            "IOMaximumBandwidth": 0
        },
        "GraphDriver": {
            "Data": null,
            "Name": "btrfs"
        },
        "Mounts": [
            {
                "Type": "bind",
                "Source": "/mnt/user/appdata/pihole/dnsmasq.d",
                "Destination": "/etc/dnsmasq.d",
                "Mode": "rw",
                "RW": true,
                "Propagation": "rprivate"
            },
            {
                "Type": "bind",
                "Source": "/mnt/user/appdata/pihole/pihole",
                "Destination": "/etc/pihole",
                "Mode": "rw",
                "RW": true,
                "Propagation": "rprivate"
            }
        ],
        "Config": {
            "Hostname": "80680360fa8f",
            "Domainname": "",
            "User": "",
            "AttachStdin": false,
            "AttachStdout": true,
            "AttachStderr": true,
            "ExposedPorts": {
                "53/tcp": {},
                "53/udp": {},
                "80/tcp": {}
            },
            "Tty": false,
            "OpenStdin": false,
            "StdinOnce": false,
            "Env": [
                "TZ=Europe/Berlin",
                "HOST_OS=unRAID",
                "TCP_PORT_53=53",
                "UDP_PORT_53=53",
                "TCP_PORT_80=80",
                "ServerIP=10.10.20.40",
                "DNS1=10.10.20.30",
                "DNS2=10.10.20.30",
                "IPv6=False",
                "TZ=Europe/Berlin",
                "WEBPASSWORD=m2ayAKZntvzyDR5nRR23A8JvGS85Bp",
                "PATH=/opt/pihole:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin",
                "TAG=debian",
                "ARCH=amd64",
                "setupVars=/etc/pihole/setupVars.conf",
                "PIHOLE_INSTALL=/tmp/ph_install.sh",
                "S6OVERLAY_RELEASE=https://github.com/just-containers/s6-overlay/releases/download/v1.21.2.2/s6-overlay-amd64.tar.gz",
                "PHP_ENV_CONFIG=/etc/lighttpd/conf-enabled/15-fastcgi-php.conf",
                "PHP_ERROR_LOG=/var/log/lighttpd/error.log",
                "S6_LOGGING=0",
                "S6_KEEP_ENV=1",
                "S6_BEHAVIOUR_IF_STAGE2_FAILS=2"
            ],
            "Cmd": null,
            "Healthcheck": {
                "Test": [
                    "CMD-SHELL",
                    "dig @127.0.0.1 pi.hole || exit 1"
                ]
            },
            "ArgsEscaped": true,
            "Image": "diginc/pi-hole",
            "Volumes": null,
            "WorkingDir": "",
            "Entrypoint": [
                "/s6-init"
            ],
            "OnBuild": null,
            "Labels": {
                "image": "diginc/pi-hole:debian_amd64",
                "maintainer": "[email protected]",
                "url": "https://www.github.com/diginc/docker-pi-hole"
            }
        },
        "NetworkSettings": {
            "Bridge": "",
            "SandboxID": "e5b92ca209a817a17c2d91687b361ca7449de6e3d037748e6eea93dd6de57528",
            "HairpinMode": false,
            "LinkLocalIPv6Address": "",
            "LinkLocalIPv6PrefixLen": 0,
            "Ports": {},
            "SandboxKey": "/var/run/docker/netns/e5b92ca209a8",
            "SecondaryIPAddresses": null,
            "SecondaryIPv6Addresses": null,
            "EndpointID": "",
            "Gateway": "",
            "GlobalIPv6Address": "",
            "GlobalIPv6PrefixLen": 0,
            "IPAddress": "",
            "IPPrefixLen": 0,
            "IPv6Gateway": "",
            "MacAddress": "",
            "Networks": {
                "br0": {
                    "IPAMConfig": {
                        "IPv4Address": "10.10.20.40"
                    },
                    "Links": null,
                    "Aliases": [
                        "80680360fa8f"
                    ],
                    "NetworkID": "3d68d516eab65c46e225eb8150b3d4a93cc55dd540038c845d149e13d086e602",
                    "EndpointID": "",
                    "Gateway": "",
                    "IPAddress": "",
                    "IPPrefixLen": 0,
                    "IPv6Gateway": "",
                    "GlobalIPv6Address": "",
                    "GlobalIPv6PrefixLen": 0,
                    "MacAddress": "",
                    "DriverOpts": null
                }
            }
        }
    }
]

 

Link to comment

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.

Guest
Reply to this topic...

×   Pasted as rich text.   Restore formatting

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.