Server becoming unaccessable forcing unclean shutdown/restart

[colev14]

Hello, I recently started having an issue where my server will lock up and no longer be accessible via the GUI. I also can't ping it or ssh or anything. I started writing the logs to the flash drive and attached the syslog and syslog-previous. I'm not really sure what the issue is. My server has been running for 20+ days consistently for the past few months. But the past couple weeks it will only run for 2-3 days before locking up. I've also added the diagnostics, but those were from after the most recent restart.

syslog syslog-previous unraid-diagnostics-20231219-0858.zip

[Heffe]

I'm having a similar issue with my system. You're not alone.

I looked through you logs, it looks like there is a misbehaving docker container. In the "syslog-previous" it looks like something is "flip-flopping" which might be causing some issues. Have you tried booting in "safe-mode"?

[colev14]

Is there any way to determine which one it is? I'm not sure how to read the logs properly. I have not tried safe mode before. What should I do after booting to safe mode?

[JorgeB]

Unfortunately there's nothing relevant logged, this usually points to a hardware issue, one thing you can try is to boot the server in safe mode with all docker/VMs disabled, let it run as a basic NAS for a few days, if it still crashes it's likely a hardware problem, if it doesn't start turning on the other services one by one.

[colev14]

On 12/19/2023 at 9:41 AM, JorgeB said:

Unfortunately there's nothing relevant logged, this usually points to a hardware issue, one thing you can try is to boot the server in safe mode with all docker/VMs disabled, let it run as a basic NAS for a few days, if it still crashes it's likely a hardware problem, if it doesn't start turning on the other services one by one.

I was able to get the server to run for 8 days without locking up running it in safe mode with docker disabled. I started up plex for 3 days and it worked fine, so I started up all the Arrs along with cloudflared docker and after 2 days it locked up again. Is there anything in the syslog that points to one of the docker containers being the issue? I'm thinking about running the Arrs/Plex on a 2nd PC in proxmox and mounting my unraid server as an NFS share and using it that way in safe mode.

unraid_server_logs.log

[JorgeB]

Nothing that I can see about that, there are some strange network related errors which I don't remember seeing before, not sure if they can be a problem:

 

2024-01-09T13:30:23+00:00 Unraid dhcpcd[1721]: eth0: 00:e0:4c:0e:2a:3d(00:e0:4c:0e:2a:3d) claims 192.168.50.112
2024-01-09T13:35:21+00:00 Unraid dhcpcd[1721]: eth0: 00:e0:4c:0e:2a:3d(00:e0:4c:0e:2a:3d) claims 192.168.50.112

 

[colev14]

On 1/10/2024 at 5:08 AM, JorgeB said:

Nothing that I can see about that, there are some strange network related errors which I don't remember seeing before, not sure if they can be a problem:

 

2024-01-09T13:30:23+00:00 Unraid dhcpcd[1721]: eth0: 00:e0:4c:0e:2a:3d(00:e0:4c:0e:2a:3d) claims 192.168.50.112
2024-01-09T13:35:21+00:00 Unraid dhcpcd[1721]: eth0: 00:e0:4c:0e:2a:3d(00:e0:4c:0e:2a:3d) claims 192.168.50.112

 

 

I was able to have the server work for a week straight before it locked up again, followed by another week of uptime before locking up, and now it locked up after 3 days. Is there anything in these logs that show anything? I know I keep asking the same thing, I'm just not sure what else to do. I am now back in safe mode with only plex running. I moved everything else over to another proxmox server. My last resort after this will be to move plex to proxmox and just use the unraid server as an nfs server.

 

Edit: decided to downgrade to 6.12.4 and take it out of safe mode and see if that does anything. If it locks up again, I'll go back to safe mode with only Plex running, then safe mode with nothing running again.

unraid_server_logs.log

Edited January 23 by colev14

[JorgeB]

Don't see anything relevant logged.

[colev14]

I swapped out my motherboard/cpu/ram to a backup PC I have. I switched from my x570, 3900x to an itx mobo and 3400G with a different ram kit. The server ran perfect for 2 weeks. I use that second system for backups though. I figured the issue might be the motherboard. I bought a new B550 board and put my old cpu and ram back in it. The server locked up again after 3 days. I am going to try swapping the ram from the backup system into this one and see if that does anything. This ram is new from January 2023 and was working fine previously. I don't really understand why that would be the issue. I'm thinking maybe it's the cpu? I had been using the 3900X in my gaming PC for 2 years with no issues. I upgraded to the 5900X and swapped the 3900X into my server. So I don't really think that's the issue. At think point, I'm not sure what to think. Swapping the ram and if that doesn't work, I'll swap my old Ryzen 1600AF into the system and see if that does anything.

 

My only other thought is it could be the GPU. But I have an old amd rx540 that is in there right now and I have an old nvidia gpu. I don't remember what model. Super old, basically just useful for display out. And the server crashes with both of them. So I don't think it's the GPU.

[JorgeB]

16 hours ago, colev14 said:

I'm thinking maybe it's the cpu?

It's a possiblity.