Hello, I'm not sure why, but about every 7-10 days it seems my system crashes and I can't tell why. I have downloaded my diagnostics logs, but I'm not sure how to read them really.

I've also previously enabled syslog server, but that log file seems to be missing stuff based on my last crash. According to the top right in Unraid, my up-time is 2 days, but I don't have any logs on Oct 20th on the syslog file.

 

I've seen other posts similar to this where it was determined to be a failing RAM stick. My server has 18 4GB sticks so I'd much rather not have to pull each one of these sticks and wait a week to see if the system will crash again.

 

Could someone read my diagnostics and tell me if there is something that would lead me on the right path for troubleshooting? I'd also be greatly appreciative if you could explain how you reached that conclusion so I can try my best to understand where to go in the future.

 

Thank you so much for your help!

ryanshomeserver-diagnostics-20241022-1319.zip

The regualr syslog starts over after every boot, post a screenshot of the syslog server settings.

Hey JorgeB, my syslog does include information from days before the last system crash, but not from the day that the last crash occurred.

 

Here are my syslog settings.

13 hours ago, Rs31412 said:

but not from the day that the last crash occurred.

Do you mean that there's just nothing logged from that day? Seems unusual to not have at least a few logged lines.

9 hours ago, JorgeB said:

just nothing logged from that day? Seems unusual to not have at least a few logged lines.

You're correct. It is very weird. I have logs from Oct 10, Oct 14, Oct 15, Oct 17, then Oct 22nd. My current uptime is 3 days 23 hours so I'd expect to see logs on Oct 20th.

 

The only logs available on Oct 17th were: 

Oct 17 13:06:17 RyansHomeServer monitor: Stop running nchan processes
Oct 17 22:25:53 RyansHomeServer monitor: Stop running nchan processes

 

This is why I thought the diagnostics logs would be the best place to find what could be causing crashes, but I don't know where to start to look. Is there potentially anywhere else where I could get information on what is causing my server to crash?

The persistent syslog is the one that should have the more info, the syslog in the diags starts over after every boot, so it won't help after a crash.

Ok, I just was told that my server crashed again while I was out this morning. Again, what's weird is there are no logs within the persistent syslog during the approximate time of the crash (11/3/2024 sometime between 8am and 11:30am):

 

Oct 22 15:43:10 RyansHomeServer kernel: kvm: already loaded vendor module 'kvm_intel'
Oct 22 17:18:28 RyansHomeServer monitor: Stop running nchan processes
Oct 22 21:05:43 RyansHomeServer kernel: md: sync done. time=27829sec
Oct 22 21:05:43 RyansHomeServer kernel: md: recovery thread: exit status: 0
Oct 23 15:21:14 RyansHomeServer monitor: Stop running nchan processes
Oct 24 13:09:24 RyansHomeServer monitor: Stop running nchan processes
Oct 24 16:53:33 RyansHomeServer monitor: Stop running nchan processes
Oct 25 16:13:20 RyansHomeServer monitor: Stop running nchan processes
Oct 28 14:13:40 RyansHomeServer monitor: Stop running nchan processes
Oct 29 13:34:50 RyansHomeServer monitor: Stop running nchan processes
Oct 29 16:53:54 RyansHomeServer monitor: Stop running nchan processes
Oct 30 17:08:40 RyansHomeServer monitor: Stop running nchan processes
Oct 31 14:04:21 RyansHomeServer monitor: Stop running nchan processes
Oct 31 16:33:26 RyansHomeServer monitor: Stop running nchan processes
Nov  1 20:53:43 RyansHomeServer webGUI: Successful login user root from 192.168.1.234
Nov  1 21:41:08 RyansHomeServer monitor: Stop running nchan processes
Nov  3 12:40:56 RyansHomeServer root: Delaying execution of fix common problems scan for 10 minutes
Nov  3 12:40:56 RyansHomeServer unassigned.devices: Mounting 'Auto Mount' Devices...
Nov  3 12:40:57 RyansHomeServer unassigned.devices: Mounting partition 'sdc1' at mountpoint '/mnt/disks/cctv'...
Nov  3 12:40:57 RyansHomeServer unassigned.devices: Mount cmd: /sbin/mount -t 'xfs' -o rw,relatime '/dev/sdc1' '/mnt/disks/cctv'
Nov  3 12:40:57 RyansHomeServer kernel: XFS (sdc1): Mounting V5 Filesystem
Nov  3 12:40:57 RyansHomeServer kernel: XFS (sdc1): Starting recovery (logdev: internal)
Nov  3 12:40:57 RyansHomeServer kernel: XFS (sdc1): Ending recovery (logdev: internal)

 

I can provide further logs after Nov 3 12:40:57, but at that time, I had logged in to Unraid, restarted the array and pulled the syslog.

 

Is there any other way I should be troubleshooting this?

Looks like the server restarted by itself, that is almost always a hardware issue.

Hey @JorgeB, is there any way to limit the type of hardware that may be failing? I recently moved my drives and GPUs to an R720 so I have a new motherboard, new RAM, new CPUs, but I'm still getting crashes. So it has to be one of the components that existed in both systems (unless I got lucky and the new system happens to have another separate failing part too). If the drives are surviving the parity check and come back up as "normal operation, drive is active" (Green dot), then does that mean that the drives aren't the problem?

11 hours ago, Rs31412 said:

motherboard, new RAM, new CPUs

If all this is new, and it keeps rebooting, could be the PSU, or where the server is connected to, we've had users with similar issues caused by a UPS, an even a smart plug.

idrac should log any hardware issues, I would start there.