Hi all, hoping someone can help.

 

Came home to a powered-off server.

 

I troubleshooted the following,

PSU checked out fine. Tested server with alternate PSUs.

I removed the battery on the mobo to clear bios.

The original USB memory stick seemed faulty from the dirty power off, I'm not sure tbh if that was the cause of the USB memory card acting up but it has since been replaced.

Backup of Unraid installed on new USB memory stick.

Once I got Unraid back up and running, it would start crashing, getting Call Trace errors up the wazoo. I'm not sure why all of a sudden it started happening. I recently installed the Satisfactory dedicated server in Docker which may have been the cause/catalyst, though that docker app ran fine for a few days with zero issues.

I was on macvlan but followed the instructions in the 6.12.4 release notes and switched to ipvlan.

I still got call trace errors. tinkered further with docker and network settings. Currently, my network settings are:

Docker Settings: ipvlan
Settings > Network Settings > eth0 > Enable Bonding = No.
Settings > Network Settings > eth0 > Enable Bridging = Yes.
Settings > Docker > Host access to custom networks = Disabled
 

I also deleted the docker image file, and reinstalled all dockers apps again on a new image.
I had a bit of stability for almost a day but then noticed a mce: [Hardware Error]: Machine check events logged, but I was unable to figure out how to investigate further. A little later on another call trace.
Many dirty power offs and powering on back to Unraid to try and solve the issue but unfortunately, I'm still getting call trace errors which forces me to do another dirty power off. Lost count now.

I ran a memtest overnight and completed over 4 passes, zero errors.

 

I've exhausted my ideas to further troubleshoot, hence why I'm here. I have attached some images and diagnostics which I got immediately after powering up from another crash.

Any help is much appreciated.

apollo-diagnostics-20241101-1700.zip

Difficult to say by the call traces if it's hardware or software, one thing you can try is to boot the server in safe mode with all docker containers/VMs disabled, let it run as a basic NAS for a few days, if it still crashes it's likely a hardware problem, if it doesn't start turning on the other services one by one, including the docker containers. 

Just because the PSU tested clean doesnt mean that the issue couldnt be beyond it. Im willing to guess that you can boot to unraid on another box just fine. 

 

IMO... remove everything from the box so it's just a CPU/1 RAM stick/UNRAID usb/and video card (if you need it) and see if you still get the error. If you do, you know it's one of these things that's causing the error. From here start adding a part/device and boot till you get the error. Then you know what's causing your issues

Edited November 1, 20241 yr by mathomas3

You might also try booting into another 'live' OS via USB and see if anything is amiss there as well

54 minutes ago, JorgeB said:

Difficult to say by the call traces if it's hardware or software, one thing you can try is to boot the server in safe mode with all docker containers/VMs disabled, let it run as a basic NAS for a few days, if it still crashes it's likely a hardware problem, if it doesn't start turning on the other services one by one, including the docker containers. 

Hi JorgeB, thanks for the advice. I'll go ahead and do as you recommended and post the findings here.
 

 

12 minutes ago, mathomas3 said:

Just because the PSU tested clean doesnt mean that the issue couldnt be beyond it. Im willing to guess that you can boot to unraid on another box just fine. 

 

IMO... remove everything from the box so it's just a CPU/1 RAM stick/UNRAID usb/and video card (if you need it) and see if you still get the error. If you do, you know it's one of these things that's causing the error. From here start adding a part/device and boot till you get the error. Then you know what's causing your issues

Hi mathomas3, thanks for the reply. I decided to change out the PSU entirely. Originally it was a 750W. Now I have a different PSU at 1000W in the system. Booting to Unraid isn't an issue, unless you're suggesting I try migrating the system to a different CPU, MOBO, RAM etc which I can do.

 

When I was troubleshooting previously, I would boot up into Unraid and immediately go and disable the Docker service as I was convinced one of the containers was causing Call Traces. Hard to say so with certainty after seeing the hardware error in the log. The mobo, CPU, ram in the system currently are over 10 years old, so it could just be their time. I also thought maybe it was the Satisfactory dedicated server container in docker causing issues but I had that running for almost a whole day yesterday. Then I went to update a password in Vaultwarden and got another Call Trace error.

I have the system in Safe Mode currently doing a Parity Check. I got a 3 day weekend ahead, so I'll turn docker services back on Monday and turn on individual containers one by one and see what happens.

 

18 minutes ago, mathomas3 said:

You might also try booting into another 'live' OS via USB and see if anything is amiss there as well

I'll try this as well if need be. Thanks again.

 

Ok. Glad that you took the time to include the PSU... it's overlooked when changing hardware IMO... 

Update.

 

I ran the server for 48hrs in safe mode. Did a couple of backups from my PC to Unraid. I also played some media files hosted on Unraid just so it wasn't sitting there doing nothing. Experienced no issues at all.

 

Decided to reboot Unraid with services and docker enabled and turned on one docker container at a time spaced out over long intervals. Had plans with the family, sleep, etc. Couldn't sit next to Unraid all day.

 

Everything so far has been stable. No trace errors. I have avoided turning on the Satisfactory Dedicated Server so far. But plan to do so once I've confirmed some stability with all the other containers up and running. 

 

I did get a hardware error again and have attached a syslog and diagnostics.

Is the ACHP/BIOS error and the mce Hardware error the same thing? Apologies, I have no idea how to read these logs.

 

How do I view the mcelog?

 

Thanks again for the help.
 

apollo-diagnostics-20241104-2017.zip apollo-syslog-20241104-1117.zip

I may have spoken too soon, went into one of the containers to change the settings as I only seeing the port number and not the whole IP address on the docker page and it crashed Unraid. Took a picture of the monitor attached to the Unraid tower. When I rebooted, the container was completely removed from my docker page. Went to reinstall the container and it auto-started. Worried it would crash again, I tried to go into the container settings but didn't have enough time unchange the edit I made before it crashed again. This time a Call trace error.

1st Crash

 

2nd Crash

 

Python segfaulted on the diags, looks more like a hardware issue to me, same for those call traces, if you have multiple sticks try using the server with just one, if the same try with a different one, that will basically rule out bad RAM.

2 hours ago, JorgeB said:

Python segfaulted on the diags, looks more like a hardware issue to me, same for those call traces, if you have multiple sticks try using the server with just one, if the same try with a different one, that will basically rule out bad RAM.

Hi JorgeB, thanks for the reply.

So I've got 32gb split 4x 8gb. I've booted Unraid with individual 8gb sticks and am still getting Call Trace errors.

 

Would this photo (sorry, not sure why I didn't take a screenshot. Very very tired) indicate that the hardware issue is with my CPU?

34 minutes ago, Aganan said:

indicate that the hardware issue is with my CPU?

Could be.