unRAID PRO 6.9.2, Asrock x370, ryzen 5 1600x (CSTATES DISABLED), 16gb ram

 

Suspicions started when Home Assistant was showing sudden drops to zero on all my sensors, including those sourced from scripts and MQTT from local sources.

 

So I got Uptime Kuma to track as many services as I can. Sure just about every day at 3AM and/or midnight the 60-second health checks from Uptime Kuma say everything is down, generally "ECONNREFUSED" for local services (e.g. MariaDB, Mosquitto, Nginx) and "Client network socket disconnected before secure TLS connection was established" for external services (e.g. google.com, 1.1.1.1). Sometimes uptime kuma misses the downtime spikes maybe because it only checks every 60 seconds, but Home Assistant sees them every single time because my sensors all post every 5 seconds.

 

There are no cronjobs at this time, and there is nothing in syslog, srsly the most boring syslog ever. Actually nothing in syslog pertaining to this at all. Always midnight, 3AM, or both.

 

Below are a bunch of screenshots showing stuff, as well as syslog from late last night to now; where things all went offline like normal.

 

Does anyone have any ideas?

 

root@rack:~# tail -n50 /var/log/syslog
Mar 16 22:00:12 rack emhttpd: spinning down /dev/sdg
Mar 16 22:00:14 rack emhttpd: spinning down /dev/sdj
Mar 16 22:00:14 rack emhttpd: spinning down /dev/sdh
Mar 16 22:00:14 rack emhttpd: spinning down /dev/sde
Mar 16 22:00:14 rack emhttpd: spinning down /dev/sdb
Mar 16 22:00:14 rack emhttpd: spinning down /dev/sdc
Mar 16 22:00:14 rack emhttpd: spinning down /dev/sdi
Mar 17 00:27:48 rack root: /etc/libvirt: 23.9 MiB (25100288 bytes) trimmed on /dev/loop3
Mar 17 00:27:48 rack root: /var/lib/docker: 118.1 GiB (126809567232 bytes) trimmed on /dev/loop2
Mar 17 00:27:48 rack root: /mnt/disks/Kingston240SSD: 113.2 GiB (121536778240 bytes) trimmed on /dev/sdk1
Mar 17 02:00:17 rack emhttpd: read SMART /dev/sdj
Mar 17 02:00:19 rack emhttpd: read SMART /dev/sde
Mar 17 02:00:53 rack emhttpd: read SMART /dev/sdb
Mar 17 02:01:08 rack emhttpd: read SMART /dev/sdi
Mar 17 02:01:18 rack emhttpd: read SMART /dev/sdg
Mar 17 02:02:01 rack emhttpd: read SMART /dev/sdc
Mar 17 02:02:30 rack emhttpd: read SMART /dev/sdh
Mar 17 03:14:10 rack emhttpd: spinning down /dev/sdh
Mar 17 03:14:12 rack emhttpd: spinning down /dev/sdg
Mar 17 03:14:12 rack emhttpd: spinning down /dev/sde
Mar 17 03:14:12 rack emhttpd: spinning down /dev/sdc
Mar 17 03:14:12 rack emhttpd: spinning down /dev/sdi
Mar 17 03:14:33 rack emhttpd: spinning down /dev/sdb
Mar 17 03:14:35 rack emhttpd: spinning down /dev/sdj
Mar 17 03:40:16 rack crond[1914]: exit status 1 from user root /usr/local/sbin/mover &> /dev/null
Mar 17 05:50:20 rack emhttpd: read SMART /dev/sdh
Mar 17 06:37:33 rack emhttpd: read SMART /dev/sde
Mar 17 06:38:19 rack emhttpd: read SMART /dev/sdj
Mar 17 06:38:19 rack emhttpd: read SMART /dev/sdb
Mar 17 06:38:19 rack emhttpd: read SMART /dev/sdc
Mar 17 06:38:42 rack emhttpd: read SMART /dev/sdg
Mar 17 06:38:42 rack emhttpd: read SMART /dev/sdi
Mar 17 07:23:04 rack webGUI: Successful login user root from [redacted]
Mar 17 07:38:44 rack emhttpd: spinning down /dev/sdj
Mar 17 07:38:44 rack emhttpd: spinning down /dev/sdg
Mar 17 07:38:44 rack emhttpd: spinning down /dev/sde
Mar 17 07:38:44 rack emhttpd: spinning down /dev/sdb
Mar 17 07:38:44 rack emhttpd: spinning down /dev/sdc
Mar 17 07:38:44 rack emhttpd: spinning down /dev/sdi
Mar 17 07:40:26 rack emhttpd: spinning down /dev/sdh

 

Edited March 16, 20224 yr by andrew207

It seems like I have the same problem you have. Not every night but like every 2 or 3 days my docker containers loose connection for a short time.

For me I noticed it when my bigger copy jobs on Krusader did not finish and the UI reset itself to the default location the next morning.  

 

I know the post is a bit old now but as it is the only one with a similar problem I found I would like to ask if you found a solution for the problem?

At some point it changed from every night to once a week, I have no idea why. It still happens reliably once a week at ~3AM same as before.

 

My specific problem with this downtime was that a few automations would kick off in Home Assistant when sensors dropped to zero as the Mosquitto container died, I was able to resolve that through application-logic that ignores triggers when everything is reporting zero.

 

So no I never resolved it, but the problem has eased up and I worked around it.

 

Thank you very much for the update! Makes me feel less like I'm the only one with this weird problem. 

 

I also know now that the timing has to have something to do with it. Otherwise my server wouldn't do the exact same thing at basically the same time. (Never 100% checked the timing but I noticed that it had to be between like 2-4 in the morning)

 

If I find something on my system I will update here. 

 

Edited October 31, 20223 yr by HQToast

Could it be related to dhcp allocation?

Interesting idea. I was quite excited for a moment because I have not even thought about DHCP possibly causing this.

 

Unfortunately this does not seem to be the problem.

I am using a range of static IPs for all my Server stuff and have a dedicated space for DHCP addresses which is not even close to being full, causing problems with the allocation. 

So in theory I can't see a reason why it would cause the problems. 

 

But just to be sure I will decrease the lease time and address space dramatically to try causing allocation problems intentionally. Lets see if this could actually be a cause 👍

Well I think I found my problem... And I do feel quite stupid now for basically looking at it and not really using my brain to process the obvious problem.

 

It is the "CA Backup / Restore Appdata" plugin and it is doing exactly what I told it to do 2 years ago. It even tells you what it is doing every time you check the settings.

 

 

Well. And in the settings the again obvious trigger for the problem is right there. Weekly backup at 4am in the morning.

 

I just found it so I can not 100% say that this is the problem but looking at it I am pretty sure that this is the solution for my specific problem. 

 

God dammit you're right @HQToast. It certainly looks like the answer.

 

Might suggest the devs write a message to syslog :P. Cheers.