Docker / Community Applications Connection Time Out - Can't install new containers [SOLVED]


Recommended Posts

SOLVED: Turned out to be address via a later firmware update to the router.


Hello! Thanks in advance for checking out my question.

TLDR; I installed a new router and now my plugins and community applications pages are very slow to load and pulling new docker containers does not work / takes a very long time. I tried many steps to resolve, but the issue persists.

Before yesterday I was running my unraid server (version 6.9.0-rc2) on my home network. I purchased a new router (QNAP QHora-301W w/ up-to-date firmware) and plugged my unraid server into a 10Gbit LAN port. Nothing else was changed. I updated the LAN IP range in the router (192.168.100.X > 10.20.30.X), set my unraid server to the same static IP it was on prior to the new router (10.20.30.222; managed by the router), and set DNS to 1.1.1.1 (was 8.8.8.8 before). I did NOT bring over my old port forwarding mappings, because I felt I could use a fresh start on that (I probably had more open ports than I really needed). Everything ISP-related stayed exactly the same. I access some docker containers via the internet using Nginx Proxy Manager (NPM), and so I did forward port 80 > 1880 and 443 > 18443.

At this point, everything on the server is seemingly working great. However, I tried to log into the NPM web GUI and discovered that I could not, with the error of "Bad Gateway". I tried to Google that error and didn't find any good leads. I figured I'd just backup my appdata folder for NPM, delete the docker container, and start fresh to see if that fixed anything. I then went to Community Applications (CA) and tried to re-download the NPM container. CA (and also the "Plugins" tab) was much slower than usual to load. I also could not pull down any containers (NPM or anything else). These were the errors from Tools>Diagnostics:
 

time="2021-04-07T08:39:55.878436185-07:00" level=error msg="Download failed, retrying: read tcp 10.20.30.222:33386->104.18.123.25:443: read: connection timed out"
time="2021-04-07T08:39:55.878487095-07:00" level=error msg="Download failed, retrying: read tcp 10.20.30.222:33382->104.18.123.25:443: read: connection timed out"
time="2021-04-07T08:39:55.878441744-07:00" level=error msg="Not continuing with pull after error: error pulling image configuration: read tcp 10.20.30.222:33384->104.18.123.25:443: read: connection timed out"
time="2021-04-07T08:39:56.390430850-07:00" level=error msg="Download failed, retrying: read tcp 10.20.30.222:33392->104.18.123.25:443: read: connection timed out"
time="2021-04-07T08:40:49.127467179-07:00" level=error msg="Not continuing with pull after error: error pulling image configuration: read tcp 10.20.30.222:56906->104.18.125.25:443: read: connection timed out"
time="2021-04-07T08:40:57.830529626-07:00" level=error msg="Not continuing with pull after error: error pulling image configuration: read tcp 10.20.30.222:56916->104.18.125.25:443: read: connection timed out"
time="2021-04-07T08:41:50.054561692-07:00" level=error msg="Not continuing with pull after error: error pulling image configuration: read tcp 10.20.30.222:46292->104.18.124.25:443: read: connection timed out"


And on the docker pull page from community applications:
 

IMAGE ID [1610037955]: Pulling from jlesage/nginx-proxy-manager.
IMAGE ID [df20fa9351a1]: Already exists.
IMAGE ID [c29f2a9687c5]: Already exists.
IMAGE ID [f2b10fbfc380]: Already exists.
IMAGE ID [529722c2e3cf]: Already exists.
IMAGE ID [f0cf5f38d987]: Already exists.
IMAGE ID [21fb739242f4]: Already exists.
IMAGE ID [b17e90563eea]: Pulling fs layer. Downloading 92% of 532 KB.
IMAGE ID [bff4d859ae50]: Pulling fs layer. Downloading 67% of 4 MB.
IMAGE ID [fd1567abff3c]: Pulling fs layer. Downloading 92% of 34 MB.
IMAGE ID [90a3e8820aa7]: Pulling fs layer.
IMAGE ID [4704d454c63e]: Pulling fs layer.
IMAGE ID [138ba29e4057]: Pulling fs layer.

 

It is strange to me that some data is coming through (92% of 34MB), but not enough! Those downloads stall at those percentages after about 5-10 seconds. After about 1 minute elapses, this appears:

 

root@localhost:# /usr/local/emhttp/plugins/dynamix.docker.manager/scripts/docker run -d --name='NginxProxyManager' --net='bridge' -e TZ="America/Los_Angeles" -e HOST_OS="Unraid" -e 'USER_ID'='99' -e 'GROUP_ID'='100' -e 'UMASK'='000' -e 'APP_NICENESS'='' -e 'DISABLE_IPV6'='0' -p '7818:8181/tcp' -p '1880:8080/tcp' -p '18443:4443/tcp' -v '/mnt/user/appdata/NginxProxyManager':'/config':'rw' 'jlesage/nginx-proxy-manager'
Please wait ...

Unable to find image 'jlesage/nginx-proxy-manager:latest' locally
latest: Pulling from jlesage/nginx-proxy-manager
df20fa9351a1: Already exists
c29f2a9687c5: Already exists
f2b10fbfc380: Already exists
529722c2e3cf: Already exists
f0cf5f38d987: Already exists
21fb739242f4: Already exists
b17e90563eea: Pulling fs layer
bff4d859ae50: Pulling fs layer
fd1567abff3c: Pulling fs layer
90a3e8820aa7: Pulling fs layer
4704d454c63e: Pulling fs layer
138ba29e4057: Pulling fs layer
4704d454c63e: Waiting
90a3e8820aa7: Waiting
138ba29e4057: Waiting


And then we wait for a long, long time. These are the abbreviated steps I took next, more or less in this order:
 

  • Unraid is able to ping Google just fine
  • All docker containers (except NPM's Bad Gateway) are working perfectly fine via local ports (but not outside network, because NPM is gone now)
  • Checked "Fix Common Problems" and removed preclear plugin (probably unrelated); no other problems show up; Problem persists
  • Googled around and determined it might be a DNS issue; tried openDNS (208.67.222.222), Google DNS server (8.8.8.8), as well as a different CloudFlare DNS (1.0.0.1) all set via the router (unraid set to get automatically); Problem persists
  • Checked that Docker Hub was online and functional
  • Checked that GitHub was online and functional
  • Shutdown server, shutdown router, started router, started server; Problem persists
  • Turned off port forwarding of 80 and 443 to nginx ports (since the docker container is gone)

  • Shutdown server, shutdown router, started router, started server; Problem persists

  • Opened up some high-valued ports near those that I saw referenced in the docker diagnostics (~30000-50000); Problem persists

  • Switch to a 1Gbit port on router (was 10Gbit port); Problem persists

  • Settings > Docker > Scrub docker btrfs with correct errors checked. No errors found.; Problem persists

  • Reboot server in GUI Safe Mode, installed CA, try again; Problem persists
     

In GUI Safe Mode, I decided to just leave the docker container download going for a long time (~1 hour). I came back to this:
 

root@localhost:# /usr/local/emhttp/plugins/dynamix.docker.manager/scripts/docker run -d --name='NginxProxyManager' --net='bridge' -e TZ="America/Los_Angeles" -e HOST_OS="Unraid" -e 'USER_ID'='99' -e 'GROUP_ID'='100' -e 'UMASK'='000' -e 'APP_NICENESS'='' -e 'DISABLE_IPV6'='0' -p '7818:8181/tcp' -p '1880:8080/tcp' -p '18443:4443/tcp' -v '/mnt/user/appdata/NginxProxyManager':'/config':'rw' 'jlesage/nginx-proxy-manager'
Please wait .

Unable to find image 'jlesage/nginx-proxy-manager:latest' locally
latest: Pulling from jlesage/nginx-proxy-manager
df20fa9351a1: Already exists
c29f2a9687c5: Already exists
f2b10fbfc380: Already exists
529722c2e3cf: Already exists
f0cf5f38d987: Already exists
21fb739242f4: Already exists
b17e90563eea: Pulling fs layer
bff4d859ae50: Pulling fs layer
fd1567abff3c: Pulling fs layer
90a3e8820aa7: Pulling fs layer
4704d454c63e: Pulling fs layer
138ba29e4057: Pulling fs layer
138ba29e4057: Waiting
90a3e8820aa7: Waiting
4704d454c63e: Waiting
b17e90563eea: Retrying in 5 seconds
bff4d859ae50: Retrying in 5 seconds
b17e90563eea: Retrying in 4 seconds
bff4d859ae50: Retrying in 4 seconds
b17e90563eea: Retrying in 3 seconds
bff4d859ae50: Retrying in 3 seconds
fd1567abff3c: Retrying in 5 seconds
b17e90563eea: Retrying in 2 seconds
bff4d859ae50: Retrying in 2 seconds
fd1567abff3c: Retrying in 4 seconds
b17e90563eea: Retrying in 1 second
bff4d859ae50: Retrying in 1 second
fd1567abff3c: Retrying in 3 seconds
fd1567abff3c: Retrying in 2 seconds
bff4d859ae50: Download complete
b17e90563eea: Verifying Checksum
b17e90563eea: Download complete
b17e90563eea: Pull complete
bff4d859ae50: Pull complete
fd1567abff3c: Retrying in 1 second
fd1567abff3c: Download complete
138ba29e4057: Verifying Checksum
138ba29e4057: Download complete
fd1567abff3c: Pull complete


and in docker logs via diagnostics:
 

time="2021-04-07T18:43:31.729223056-07:00" level=error msg="Download failed, retrying: read tcp 10.20.30.222:43780->104.18.124.25:443: read: connection timed out"
time="2021-04-07T18:43:31.729239234-07:00" level=error msg="Download failed, retrying: read tcp 10.20.30.222:43782->104.18.124.25:443: read: connection timed out"
time="2021-04-07T18:43:34.289240096-07:00" level=error msg="Download failed, retrying: read tcp 10.20.30.222:43784->104.18.124.25:443: read: connection timed out"
time="2021-04-07T18:48:39.953230819-07:00" level=error msg="Download failed, retrying: read tcp 10.20.30.222:43930->104.18.124.25:443: read: connection timed out"
time="2021-04-07T18:48:40.978206276-07:00" level=error msg="Download failed, retrying: read tcp 10.20.30.222:43928->104.18.124.25:443: read: connection timed out"


Amazingly, the docker container is installed at this point. I guess it constantly tried different ports until it could make a connection to 104.18.124.25:443? That IP address (104.18.124.25:443) appears to be a cloudflare server, not sure if there's significance to that, or that's just what docker hub uses. However, NPM still displays the "Bad Gateway" error from before (even with new appdata path).

 

So, I got my short-term problem of getting access to my containers via the internet fixed after re-forwarding ports for NPM. However, I am unable to install new docker containers (at least in a reasonable amount of time), I probably can't update them (not tested yet), nor add new subdomains to NPM (due to "Bad Gateway" problem). I don't see much in the NPM logs except:
 

[4/7/2021] [8:03:10 PM] [Migrate ] › ℹ info Current database version: none


Nothing else on my network is having any issues with this new router. What have I missed on the Unraid side?

Thanks so much!!

 

Edited by njdowdy
wrote the NPM port wrong, SOLVED
Link to comment
  • njdowdy changed the title to Docker / Community Applications Connection Time Out - Can't install new containers [SOLVED]

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.

Guest
Reply to this topic...

×   Pasted as rich text.   Restore formatting

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.