Docker service failed to start


Recommended Posts

New to UnRAID and I love it, but I've been having some stability issues that I hope folks can help me with.  Usually I can search enough to work things out on my own, but I've started drawing enough blanks here and there that it's time to reach out for help.

 

I've seen this issue one other time, but I need this system to be stable, so I want to figure out what the root cause is.

 

This morning I sat down to check on things and saw the following errors when I went to work with my Docker containers.  This is after just 21 hours of uptime.

 

image.thumb.png.2bcabf4c3e1d357a021c5f30b2e7b576.png

 

I don't see anything relevant in the syslog, but here it is anyways.

 

Quote

Nov 21 15:04:46 Tower login[2218]: ROOT LOGIN  on '/dev/pts/1'
Nov 21 16:30:45 Tower webGUI: Successful login user root from 192.168.1.105
Nov 21 16:32:58 Tower dhcpcd[1625]: br0: failed to renew DHCP, rebinding
Nov 21 16:44:58 Tower sshd[24492]: Accepted password for root from 192.168.1.105 port 60299 ssh2
Nov 21 16:45:40 Tower sshd[25272]: Accepted password for root from 192.168.1.105 port 60318 ssh2
Nov 21 16:50:43 Tower webGUI: Successful login user root from 192.168.1.105
Nov 21 18:14:11 Tower kernel: perf: interrupt took too long (2505 > 2500), lowering kernel.perf_event_max_sample_rate to 79000
Nov 21 18:45:39 Tower webGUI: Successful login user root from 192.168.1.223
Nov 21 20:40:09 Tower kernel: TCP: request_sock_TCP: Possible SYN flooding on port 8181. Sending cookies.  Check SNMP counters.
Nov 21 20:41:05 Tower kernel: mdcmd (40): nocheck Cancel
Nov 21 20:41:05 Tower kernel: md: recovery thread: exit status: -4
Nov 21 20:42:01 Tower sSMTP[18408]: Creating SSL connection to host
Nov 21 20:42:01 Tower sSMTP[18408]: SSL connection using TLS_AES_256_GCM_SHA384
Nov 21 20:42:01 Tower sSMTP[18408]: Authorization failed (534 5.7.9  https://support.google.com/mail/?p=InvalidSecondFactor j8sm4672763oif.55 - gsmtp)
Nov 21 20:49:21 Tower emhttpd: cmd: /usr/local/emhttp/plugins/dynamix/scripts/tail_log syslog
Nov 21 21:15:46 Tower emhttpd: cmd: /usr/local/emhttp/plugins/dynamix/scripts/tail_log syslog
Nov 21 21:39:01 Tower webGUI: Successful login user root from 192.168.1.105
Nov 21 21:41:06 Tower kernel: mdcmd (41): spindown 0
Nov 21 21:41:06 Tower kernel: mdcmd (42): spindown 1
Nov 21 21:47:08 Tower sshd[32627]: Accepted password for root from 192.168.1.115 port 50452 ssh2
Nov 21 22:45:21 Tower nginx: 2020/11/21 22:45:21 [alert] 6605#6605: worker process 6606 exited on signal 6
Nov 22 00:43:34 Tower kernel: mdcmd (43): spindown 2
Nov 22 02:40:09 Tower crond[1711]: exit status 1 from user root /usr/local/sbin/mover &> /dev/null
Nov 22 03:49:48 Tower kernel: mdcmd (44): spindown 1
Nov 22 04:30:01 Tower root: Fix Common Problems Version 2020.10.21
Nov 22 04:30:01 Tower root: Fix Common Problems: Warning: Plugin file.activity.plg is not up to date
Nov 22 04:30:01 Tower root: Fix Common Problems: Warning: Plugin unassigned.devices.plg is not up to date
Nov 22 04:30:07 Tower root: Fix Common Problems: Warning: Syslog mirrored to flash
Nov 22 04:30:07 Tower sSMTP[22792]: Creating SSL connection to host
Nov 22 04:30:07 Tower sSMTP[22792]: SSL connection using TLS_AES_256_GCM_SHA384
Nov 22 04:30:08 Tower sSMTP[22792]: Authorization failed (534 5.7.9  https://support.google.com/mail/?p=InvalidSecondFactor w6sm4667340otj.12 - gsmtp)
Nov 22 05:01:15 Tower root: /etc/libvirt: 920.2 MiB (964894720 bytes) trimmed on /dev/loop3
Nov 22 05:01:15 Tower root: /var/lib/docker: 186.7 GiB (200447295488 bytes) trimmed on /dev/loop2
Nov 22 05:01:15 Tower root: /mnt/cache: 1.5 TiB (1606462537728 bytes) trimmed on /dev/sdd1
Nov 22 05:39:37 Tower kernel: mdcmd (45): spindown 1
Nov 22 09:43:24 Tower webGUI: Successful login user root from 192.168.1.20
Nov 22 10:00:44 Tower webGUI: Successful login user root from 192.168.1.113
Nov 22 10:01:28 Tower emhttpd: cmd: /usr/local/emhttp/plugins/dynamix/scripts/tail_log syslog
Nov 22 10:43:15 Tower webGUI: Successful login user root from 192.168.1.115
Nov 22 10:43:25 Tower emhttpd: cmd: /usr/local/emhttp/plugins/dynamix/scripts/tail_log syslog
Nov 22 11:47:44 Tower kernel: mdcmd (46): spindown 1
Nov 22 11:52:16 Tower sshd[25020]: Accepted password for root from 192.168.1.115 port 51644 ssh2

 

And I've attached the diagnostics.

 

I appreciate any and all help you can send my way.

tower-diagnostics-20201122-1154.zip

Link to comment

Why do you have 200G allocated to docker.img? Have you had problems filling it? Making it larger won't fix that, it will only make it take longer to fill. 20G is usually much more than enough. I have 17 containers and they are using less than half of 20G.

 

Filling docker.img is usually a sign that you have one or more apps writing to a path that isn't mapped. Common mistakes are specifying a path within an app that doesn't exactly match the mapped container path in upper/lower case, or specifying a relative path.

 

Were those diagnostics taken while you were having the problem in the screenshot?

Link to comment
1 hour ago, trurl said:

Why do you have 200G allocated to docker.img? Have you had problems filling it? Making it larger won't fix that, it will only make it take longer to fill. 20G is usually much more than enough. I have 17 containers and they are using less than half of 20G.

 

Filling docker.img is usually a sign that you have one or more apps writing to a path that isn't mapped. Common mistakes are specifying a path within an app that doesn't exactly match the mapped container path in upper/lower case, or specifying a relative path.

 

Were those diagnostics taken while you were having the problem in the screenshot?

To be honest, when I was setting this up for the first time and I didn't know any better, more seemed like it would be better.  I've since read that it's literally 10x what it really needs to be, but I haven't bothered to fix it because it seems like a bit of a chore and it hasn't caused me any problems.  Do I need to get it back to 20G?  As far as I know I've never actually filled it, but then again I've never checked either.

 

Those diags are indeed from before rebooting after the problem showed up.

Link to comment

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.

Guest
Reply to this topic...

×   Pasted as rich text.   Restore formatting

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.