MammothJerk Posted June 9, 2024 Posted June 9, 2024 I don't edit my docker containers that often so i couldve had this issue for weeks/months without realizing it, but it has definitely been noticeable in the past couple of days that updating docker containers is taking FOREVER and most of the times its not even working at all. It also seems to take ages to stop the containers and sometimes it even goes past the set timeout and forces an unclean shutdown. This is a very typical error when updating a container: things i've tried: rebooting server changing dns deleting the docker directory and reinstalling all containers from CA (reinstalling the containers felt a bit slow but it didn't freeze the entire server). uninstall the ipmi plugin turn off autostart from all containers the only thing that seems to have helped is to turn off autostart from all containers, which let me edit a couple of them without freezing the server. But when i got to the 5th container or so it started freezing again. heres a snippet from the syslog when trying to edit the sonarr and then sonarr4k containers. Jun 9 06:55:45 VeNASaur kernel: usb 1-6: reset full-speed USB device number 4 using xhci_hcd Jun 9 06:55:45 VeNASaur kernel: usb 1-13: reset high-speed USB device number 6 using xhci_hcd Jun 9 06:59:36 VeNASaur kernel: docker0: port 5(vethc279c8b) entered blocking state Jun 9 06:59:36 VeNASaur kernel: docker0: port 5(vethc279c8b) entered disabled state Jun 9 06:59:36 VeNASaur kernel: device vethc279c8b entered promiscuous mode Jun 9 06:59:47 VeNASaur kernel: eth0: renamed from vetha0f7368 Jun 9 06:59:47 VeNASaur kernel: IPv6: ADDRCONF(NETDEV_CHANGE): vethc279c8b: link becomes ready Jun 9 06:59:47 VeNASaur kernel: docker0: port 5(vethc279c8b) entered blocking state Jun 9 06:59:47 VeNASaur kernel: docker0: port 5(vethc279c8b) entered forwarding state Jun 9 07:25:33 VeNASaur nginx: 2024/06/09 07:25:33 [error] 12249#12249: *13611 upstream timed out (110: Connection timed out) while reading response header from upstream, client: 192.168.1.100, server: , request: "GET /plugins/ipmi/include/ipmi_temp.php?unit=C&dot=. HTTP/2.0", subrequest: "/auth-request.php", upstream: "fastcgi://unix:/var/run/php5-fpm.sock", host: "192.168.1.111", referrer: "https://192.168.1.111/Docker/UpdateContainer?xmlTemplate=edit:/boot/config/plugins/dockerMan/templates-user/my-Sonarr.xml" Jun 9 07:25:33 VeNASaur nginx: 2024/06/09 07:25:33 [error] 12249#12249: *13611 auth request unexpected status: 504 while sending to client, client: 192.168.1.100, server: , request: "GET /plugins/ipmi/include/ipmi_temp.php?unit=C&dot=. HTTP/2.0", host: "192.168.1.111", referrer: "https://192.168.1.111/Docker/UpdateContainer?xmlTemplate=edit:/boot/config/plugins/dockerMan/templates-user/my-Sonarr.xml" Jun 9 07:25:33 VeNASaur nginx: 2024/06/09 07:25:33 [error] 12249#12249: *13611 upstream timed out (110: Connection timed out) while reading response header from upstream, client: 192.168.1.100, server: , request: "GET /plugins/unassigned.devices.preclear/assets/sweetalert2.js?_=1717910096010 HTTP/2.0", subrequest: "/auth-request.php", upstream: "fastcgi://unix:/var/run/php5-fpm.sock", host: "192.168.1.111", referrer: "https://192.168.1.111/Docker/UpdateContainer?xmlTemplate=edit:/boot/config/plugins/dockerMan/templates-user/my-Sonarr.xml" Jun 9 07:25:33 VeNASaur nginx: 2024/06/09 07:25:33 [error] 12249#12249: *13611 auth request unexpected status: 504 while sending to client, client: 192.168.1.100, server: , request: "GET /plugins/unassigned.devices.preclear/assets/sweetalert2.js?_=1717910096010 HTTP/2.0", host: "192.168.1.111", referrer: "https://192.168.1.111/Docker/UpdateContainer?xmlTemplate=edit:/boot/config/plugins/dockerMan/templates-user/my-Sonarr.xml" Jun 9 07:25:46 VeNASaur nginx: 2024/06/09 07:25:46 [error] 12249#12249: *13611 upstream timed out (110: Connection timed out) while reading upstream, client: 192.168.1.100, server: , request: "POST /Docker/UpdateContainer?xmlTemplate=edit:/boot/config/plugins/dockerMan/templates-user/my-Sonarr.xml HTTP/2.0", upstream: "fastcgi://unix:/var/run/php5-fpm.sock:", host: "192.168.1.111", referrer: "https://192.168.1.111/Docker/UpdateContainer?xmlTemplate=edit:/boot/config/plugins/dockerMan/templates-user/my-Sonarr.xml" Jun 9 07:31:46 VeNASaur kernel: br-ee7a2a4499c0: port 1(vethf120004) entered blocking state Jun 9 07:31:46 VeNASaur kernel: br-ee7a2a4499c0: port 1(vethf120004) entered disabled state Jun 9 07:31:46 VeNASaur kernel: device vethf120004 entered promiscuous mode Jun 9 07:32:47 VeNASaur kernel: eth0: renamed from vethf1a01fc Jun 9 07:32:47 VeNASaur kernel: IPv6: ADDRCONF(NETDEV_CHANGE): vethf120004: link becomes ready Jun 9 07:32:47 VeNASaur kernel: br-ee7a2a4499c0: port 1(vethf120004) entered blocking state Jun 9 07:32:47 VeNASaur kernel: br-ee7a2a4499c0: port 1(vethf120004) entered forwarding state Jun 9 07:32:47 VeNASaur kernel: IPv6: ADDRCONF(NETDEV_CHANGE): br-ee7a2a4499c0: link becomes ready Jun 9 08:43:53 VeNASaur nginx: 2024/06/09 08:43:53 [error] 12249#12249: *24791 upstream timed out (110: Connection timed out) while reading response header from upstream, client: 192.168.1.100, server: , request: "GET /plugins/ipmi/include/ipmi_temp.php?unit=C&dot=. HTTP/2.0", subrequest: "/auth-request.php", upstream: "fastcgi://unix:/var/run/php5-fpm.sock", host: "192.168.1.111", referrer: "https://192.168.1.111/Docker/UpdateContainer?xmlTemplate=edit:/boot/config/plugins/dockerMan/templates-user/my-Sonarr4K.xml" Jun 9 08:43:53 VeNASaur nginx: 2024/06/09 08:43:53 [error] 12249#12249: *24791 auth request unexpected status: 504 while sending to client, client: 192.168.1.100, server: , request: "GET /plugins/ipmi/include/ipmi_temp.php?unit=C&dot=. HTTP/2.0", host: "192.168.1.111", referrer: "https://192.168.1.111/Docker/UpdateContainer?xmlTemplate=edit:/boot/config/plugins/dockerMan/templates-user/my-Sonarr4K.xml" Jun 9 08:43:53 VeNASaur nginx: 2024/06/09 08:43:53 [error] 12249#12249: *24791 upstream timed out (110: Connection timed out) while reading response header from upstream, client: 192.168.1.100, server: , request: "GET /plugins/unassigned.devices.preclear/assets/sweetalert2.js?_=1717914796079 HTTP/2.0", subrequest: "/auth-request.php", upstream: "fastcgi://unix:/var/run/php5-fpm.sock", host: "192.168.1.111", referrer: "https://192.168.1.111/Docker/UpdateContainer?xmlTemplate=edit:/boot/config/plugins/dockerMan/templates-user/my-Sonarr4K.xml" Jun 9 08:43:53 VeNASaur nginx: 2024/06/09 08:43:53 [error] 12249#12249: *24791 auth request unexpected status: 504 while sending to client, client: 192.168.1.100, server: , request: "GET /plugins/unassigned.devices.preclear/assets/sweetalert2.js?_=1717914796079 HTTP/2.0", host: "192.168.1.111", referrer: "https://192.168.1.111/Docker/UpdateContainer?xmlTemplate=edit:/boot/config/plugins/dockerMan/templates-user/my-Sonarr4K.xml" Jun 9 08:43:56 VeNASaur nginx: 2024/06/09 08:43:56 [error] 12249#12249: *24791 upstream timed out (110: Connection timed out) while reading upstream, client: 192.168.1.100, server: , request: "POST /Docker/UpdateContainer?xmlTemplate=edit:/boot/config/plugins/dockerMan/templates-user/my-Sonarr4K.xml HTTP/2.0", upstream: "fastcgi://unix:/var/run/php5-fpm.sock:", host: "192.168.1.111", referrer: "https://192.168.1.111/Docker/UpdateContainer?xmlTemplate=edit:/boot/config/plugins/dockerMan/templates-user/my-Sonarr4K.xml" Jun 9 08:44:01 VeNASaur nginx: 2024/06/09 08:44:01 [error] 12249#12249: *24791 upstream timed out (110: Connection timed out) while reading response header from upstream, client: 192.168.1.100, server: , request: "POST /plugins/unassigned.devices.preclear/Preclear.php HTTP/2.0", subrequest: "/auth-request.php", upstream: "fastcgi://unix:/var/run/php5-fpm.sock", host: "192.168.1.111", referrer: "https://192.168.1.111/Docker/UpdateContainer?xmlTemplate=edit:/boot/config/plugins/dockerMan/templates-user/my-Sonarr.xml" Jun 9 08:44:01 VeNASaur nginx: 2024/06/09 08:44:01 [error] 12249#12249: *24791 auth request unexpected status: 504 while sending to client, client: 192.168.1.100, server: , request: "POST /plugins/unassigned.devices.preclear/Preclear.php HTTP/2.0", host: "192.168.1.111", referrer: "https://192.168.1.111/Docker/UpdateContainer?xmlTemplate=edit:/boot/config/plugins/dockerMan/templates-user/my-Sonarr.xml" the webui never got past the "progress" page and i had to close the page, and still 10+ minutes after i closed that tab the webui is still not responding. but if i then reload php with /etc/rc.d/rc.php-fpm reload the webui instantly starts responding again. here are my docker settings: I recently updated to a 14900k and i did not have this issue before so im not sure what the issue might be, though i did a bit of troubleshooting when trying to set the CPU pinning so i mightve changed something... Any ideas? Quote
JonathanM Posted June 9, 2024 Posted June 9, 2024 Does it act the same way with a docker image instead of using docker directory? Quote
MammothJerk Posted June 10, 2024 Author Posted June 10, 2024 (edited) On 6/9/2024 at 11:48 PM, JonathanM said: Does it act the same way with a docker image instead of using docker directory? tried this, 3 hours later its still barely installed half of the containers from CA, so i doubt thats the fix. On 6/10/2024 at 1:25 AM, Squid said: You should post your diagnostics I was unable to download diagnostics without first rebooting the server, the download webui log would show up but would not populate anything, it would only show the three dots animation and the rest of the dialog was blank. After rebooting the download worked in just a couple seconds. going to try recreating diagnostics after docker containers are installed, will edit ------------------- Installed all containers from CA without issue, but i got a lot of "Execution error" dialogs when trying to start them from the docker webui. all dockers running, webUI noticeably sluggish, 20-30% CPU usage. trying to add a container takes over 10 minutes, the webui is completely unresponsive while the container is being added. Here are the diagnostics: Edited June 14, 2024 by MammothJerk Quote
SkilledAlpaca Posted June 10, 2024 Posted June 10, 2024 Following since I have the same issues. It used to only be slow and freezing behind my local ssl cert, but now even doing direct IP, Docker comes to a stand still and makes it impossible to do anything on the Unraid system. Quote
Solution MammothJerk Posted June 14, 2024 Author Solution Posted June 14, 2024 It would appear (at least in my case) that the cache SSD (kingston a400) was not of sufficient quality for my usecase. i switched to a pair of kingston kc3000 nvme SSDs and the issue is completely gone. My dockers feel more responsive than ever. Although they are hitting pretty high temps (60-70c) so i might have to pick up some heatsinks for them 😬 Quote
SkilledAlpaca Posted January 28 Posted January 28 I would like to add to this as I just solved the issue I was having. My 'docker.img' and 'libvirt.img' on the Unraid "system" share were stored on the array. The system share was set to 'Cache > Array' which caused both of those files to move from the NVME drive onto the array. Disabling Docker and VMs, changing the system share to "Cache < Array", running the mover, and then re-enable both. I personally set the system share to Cache only since it appears only Docker and VMs use it. This has solved my slowness with pulling and updating containers. Quote
tjb_altf4 Posted January 28 Posted January 28 39 minutes ago, SkilledAlpaca said: I would like to add to this as I just solved the issue I was having. My 'docker.img' and 'libvirt.img' on the Unraid "system" share were stored on the array. The system share was set to 'Cache > Array' which caused both of those files to move from the NVME drive onto the array. Disabling Docker and VMs, changing the system share to "Cache < Array", running the mover, and then re-enable both. I personally set the system share to Cache only since it appears only Docker and VMs use it. This has solved my slowness with pulling and updating containers. From memory, this gets defaulted this way only if you initially configure Unraid without a cache drive. If you create with a cache drive when first setting up, system share will default to cache drive. 1 Quote
Recommended Posts
Join the conversation
You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.