I built my first unraid machine in early October and everything was humming along great until right around Christmas.  I went to watch some plex and it said my server was not available.  I wasnt able to get to the web UI and ssh timed out trying to connect.  I checked the physical machine and the fans were all off as well as the status light on the motherboard.  The RGB was going strong on the motherboard though so got that going for me.  Pressing the power button doesn't start it back up so I believe its still on or running in some capacity.  I have to hold the power button down for maybe 30 seconds and then I can press it again and it will turn on, fans spin up, and I ma able to log in to the web UI shortly after.  This has happened maybe 5 or 6 times since Christmas and seems to be happening more and more frequnelty.  I mainly use my server as a plex machine, and thought perhaps Plex was memory leaking or something and about 3 days ago after the last event I added a script to restart the plex docker at 3 am daily.  Unfortunately today it locked up again.  I dont see anything that stands out in the logs unfortunately.  From googling around it seems I need to go into the BIOS and check any overclock/energy management settings but I didnt turn anything on when I set it up, all I did was update to the latest BIOS.  But before I pull the machine out and hook it up to a monitor and stuff tomorrow I was hoping to get some advice on what to look for.  I have an i5 13500/Gigabyte z790 Aorus Elite/Seasonic PSU/ Corsair 32gb DDR5 6000/ 2x Samsung 980 1TB nvme/ 2x Samsung 870 SSD/ 3 12TB drives from SPD and an LSI 9300-16i HBA.

peacemillion-diagnostics-20250118-1732 2.zip

Have you done memtest?

11 minutes ago, trurl said:

Have you done memtest?

When I built the machine I did a quick pass with the built in memory test and passed with no issues.  Have not run anything since.

10 hours ago, shoomba said:

I checked the physical machine and the fans were all off as well as the status light on the motherboard. 

This is almost always a hardware problem.

4 hours ago, JorgeB said:

This is almost always a hardware problem.

So just start replacing components until it stops crashing? What if it stops crashing and then starts up again 2 months later like it did after being first built.

You can first enable the syslog server and post that after an event just to confirm there's not something/somewhere invoking a shutdown, it's been known to happen due to bad power cable/button or even a cat or child, if there's no shutdown event it's hardware for sure.

1 hour ago, JorgeB said:

You can first enable the syslog server and post that after an event just to confirm there's not something/somewhere invoking a shutdown, it's been known to happen due to bad power cable/button or even a cat or child, if there's no shutdown event it's hardware for sure.

The log from the syslog server from the last event is in the attached zip on the first post.  But Ill throw it here as well as the one before. But nothing looks like a shutdown event in there to me.

peacemillion-diagnostics-20250118-1732 2.zip peacemillion-diagnostics-20250118-1732.zip

Persistent syslog doesn't come with the diags, unless the save to flash drive is enabled, and you also enable the save last syslog option.

10 hours ago, JorgeB said:

Persistent syslog

Looks like it was added into the zip of the "2" version

10 minutes ago, trurl said:

Looks like it was added into the zip of the "2" version

Thanks, missed that.

 

There's no shutdown event logged, so assuming one occurred during that time, it must be hardware related.

So far I have replaced the RAM, had an unresponsive episode maybe 3 days after I replaced it.  I pinned a couple cores to plex based on another suggestion and unplugged the usb from my UPS in case that could be causing some kind of issue.  That ran for about 4 days.  I checked the logs and saw a lot of these errors.  Pretty much constant throughout the 4 days it was on.

 

Jan 21 08:54:39 PeaceMillion kernel: pcieport 0000:00:1d.4: AER: Corrected error message received from 0000:0b:00.0 
Jan 21 08:54:39 PeaceMillion kernel: nvme 0000:0b:00.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, (Receiver ID) 
Jan 21 08:54:39 PeaceMillion kernel: nvme 0000:0b:00.0: device [144d:a80a] error status/mask=00000001/0000e000 
Jan 21 08:54:39 PeaceMillion kernel: nvme 0000:0b:00.0: [ 0] RxErr

I have since replaced the PSU as well as turned off some bios settings, basically anything that seemed like it was involved in increasing performance and speed, XMP was already off.  The error has not shown up again so far.  Seems perhaps like one of my NVME drives may be having a problem?  Could this result in the server becoming unresponsive?  They are two Samsung 980 Pro 1TBs in a ZFS Raid1 pool, they are used as my cache.

Edited January 27, 20251 yr by shoomba

Try this first for the PCIe errors:

https://forums.unraid.net/topic/118286-nvme-drives-throwing-errors-filling-logs-instantly-how-to-resolve/?do=findComment&comment=1165009

 

I replaced the power supply, stopped a couple dockers that I wasnt really using, and turned off a couple bios settings, including c states and anything that seemed to indicate overclocking increased performance and its been about 8 days so hopefully my issue is solved.