6.11.5 Crashing Every 4-7 Days

[Tytan McBride]

Hi there!

 

I recently upgraded from 6.9.2 to 6.11.5, and since doing that, it seems like my server is crashing roughly every 4-7 days. I tried changing the Docker Network Type to ipvlan already as I saw that mentioned quite a few times in previous posts, but it is still occurring. I attached my diagnostics file, any help/suggestions is beyond appreciated!

element22-diagnostics-20230331-1125.zip

[JorgeB]

Enable the syslog server and post that, also another common issue is this one, so check if you are affected.

[Tytan McBride]

54 minutes ago, JorgeB said:

Enable the syslog server and post that, also another common issue is this one, so check if you are affected.

I'll enable it and keep an eye on it. As for that other issue, I'm not doing anything torrent related on it, just Usenet stuff via SABnzbd.

[Tytan McBride]

@JorgeB I attached a syslog. It happened the morning of April 5th (today). I took a look myself but can't see anything, hopefully you can? 

syslog (3) element22-diagnostics-20230405-0422.zip

[JorgeB]

Unfortunately there's nothing relevant logged, this usually points to a hardware problem, one thing you can try is to boot the server in safe mode with all docker/VMs disabled, let it run as a basic NAS for a few days, if it still crashes it's likely a hardware problem, if it doesn't start turning on the other services one by one.

[Tytan McBride]

So far, it's looking like a software issue. I downgraded to my previous version of 6.9.2, and haven't had any issues or reboots since. Gonna give it time, but it's looking like something in the update is making something somewhere incompatible. Might have to hold off and then try 6.12 when it comes out to get it more up to date.

[Tytan McBride]

Update 5 (May 11): Alright, so it crashed again a while ago, so I switched from SWAG to NGINX, and ever since then it's been flawless (uptime is currently at 15+ days). If anyone else sees this, maybe give this a shot if you're in a similar situation. I'm not sure why or what caused this, but hey, I'll take it.

 

Update 4: So far since I deleted my custom Docker network and undid all custom network changes, it seems to be more stable. I had to go in and manually delete some configuration files, but so far so good. If it keeps working, I'll probably switch from SWAG to NGINX and see if that helps prevent it going forward since I don't trust SWAG anymore.

 

Update 3: Still crashing at around that same time, and the other system I had open decided to reboot and close the log windows (thanks Windows Update). I did notice that SWAG tries to renew its security certificates at around the time every day where the last syslog is made, exactly an hour before it's crashed multiple times. Disabling it and deleting every trace of a VPN tunnel, put everything that was routed through it back to bridge mode, and give that a shot.

 

Update 2: I decided to go back to 6.11.5 and have another system open and was going to leave that running constantly, but with logs open for all Docker containers so I would be able to monitor those, and I noticed that suddenly a lot of containers were giving an error. Rebuilding the Docker image now and going to see if maybe that's what was going funky.

 

Update: 6.9.2 continued to seem to be just fine, I gave 6.12-rc3 a shot, and it ended up doing the same thing. I did notice that it was also within a few minutes of the time at which it's crashed before, so I'm thinking whatever is running at around 3am is causing it...now to find out what it is.

 

Syslog also continued to be of no use, so it's gotta be a Docker or plugin causing it (if that makes sense).

Edited May 12, 2023 by Tytan McBride
Update