I'm at my wits end with these daily reboots! Please Help!

[Dro]

Hello All,

Hopefully i can get some direction here. I have been experiencing constant, daily reboots, pretty much every 24hrs and sometimes twice a day. Its at random times but consistently at least once a day. It's been going on for quite a long time now. It seems ever since updating from like 4 versions ago of unRAID it started.. Don't remember the exact version when it started.. My server was rock solid, with all the old hardware. At one point i had almost a 4 month uptime.. Then the reboots started and i slowly started replacing parts.

I have pretty much built an entirely new server replacing each part at a time to see if it would fix the issue and its still occurring! Its driving me absolutely crazy! Please see attached diagnostics and syslog. 

Here is everything I have done.. And mind you, everything i have done, i have let it run before replacing the next part waiting for a random reboot. The New RAM actually made the server stay up for about 3 days before a reboot. Now I'm back to daily reboots.

 

Currently on 16.12.10

New CPU (no overclocking)

New Motherboard (latest BIOS)

New RAM (no overclocking)
New NIC - I run 10gb SFP+

New DAC Cables

Tried various switch ports, different switches

Tried the 1GB Ethernet Port

New NVME Cache Drive

Replaced Multiple USB Sticks

Used Multiple USB Ports

I'm using IPVLAN

I have no Temp issues, all fans work, everything is clean
Turned off all external backups

Disabled all docker containers

Literally i have done pretty much everything. The only part that hasn't been replaced at this point is my PSU and all my HDD are about 5 years old but no errors or issues with those.. The PSU was purchased new, it is an EVGA 850w modular PSU.. Its a few years old.. Its possible that is my issue. I am planning to replace that soon. 

Desperately looking for some advice.. Thank you!

syslog-192.168.1.10.log mdronet-unraid1-diagnostics-20240626-0849.zip

Edited June 26 by Dro

[JorgeB]

Server rebooting by itself, instead of hanging or crashing is almost always a hardware problem, one thing you can try is to boot the server in safe mode with all docker containers/VMs disabled, let it run as a basic NAS for a few days, if it still crashes it's likely a hardware problem, if it doesn't start turning on the other services one by one. 

[Dro]

Just now, JorgeB said:

Server rebooting by itself, instead of hanging or crashing is almost always a hardware problem, one thing you can try is to boot the server in safe mode with all docker containers/VMs disabled, let it run as a basic NAS for a few days, if it still crashes it's likely a hardware problem, if it doesn't start turning on the other services one by one. 

 

I've replaced all hardware besides the PSU. Literally everything, including all cables.. What hardware problem can it possibly be? It must be the PSU then as I know a failing one can cause reboots and that's the only thing I haven't replaced.. Logs havent told me much but there may be other things you guys can see.

Let me try the safe mode and see what happens.. Thats a good test..

[JorgeB]

I can for sure be the PSU, that would even be one of the main suspects for spontaneous reboots.

[Dro]

5 minutes ago, JorgeB said:

I can for sure be the PSU, that would even be one of the main suspects for spontaneous reboots.

Yea i am starting to think I should have replaced that first🤣I needed an upgraded server anyways! Even though it was purchased new. I know it is a symptom of it.. 
I'm in safe mode now, lets see what happens.. Thanks!

[Dro]

On 6/26/2024 at 9:39 AM, JorgeB said:

I can for sure be the PSU, that would even be one of the main suspects for spontaneous reboots.

Server has not rebooted yet going on two days! The only thing I have running is Plex so we can use it internally. Besides that everything else is disabled. I won’t say it’s good yet until Saturday because if I make it another day without a reboot that hasn’t happened in a long time. But if it continues to stay up it sounds like it’s a potential plugin or container issue causing the reboots? I’m not using many containers nothing random or out of the ordinary. What do the logs say?

[JorgeB]

23 minutes ago, Dro said:

But if it continues to stay up it sounds like it’s a potential plugin or container issue causing the reboots?

Won't say it's impossible, but not very likely, unless there's an underlying issue and the plugin/container is exacerbating it.

[Dro]

3 hours ago, JorgeB said:

Won't say it's impossible, but not very likely, unless there's an underlying issue and the plugin/container is exacerbating it.

so, whatelse could it possible be if server continues to run with all dockers (besides plex) and all plugins disabled? Have you checked the logs to see anything? I checked myself and I can't spot anything out of the ordinary.. Server does seem rock solid so far.. Usually it would reboot around 8-12 hours of uptime but mostly random. Im actually hoping it does reboot so i can assume its the PSU because of it doesnt im even more lost.

[JorgeB]

There's nothing in the logs that I can see, and that's not surprising with spontaneous reboots, but try installing the plugins and containers one by one, to see if you can identify a culprit.

[Dro]

21 minutes ago, JorgeB said:

There's nothing in the logs that I can see, and that's not surprising with spontaneous reboots, but try installing the plugins and containers one by one, to see if you can identify a culprit.

I think that is my next step. So far, rock solid.. Ill enable one at a time and see what happens.. I did recall seeing some random errors from NPM a few days ago. 

[Dro]

On 6/28/2024 at 10:40 AM, JorgeB said:

There's nothing in the logs that I can see, and that's not surprising with spontaneous reboots, but try installing the plugins and containers one by one, to see if you can identify a culprit.

So far, zero reboots. All dockers are started now. Still in safe mode. Still haven’t enabled plugins. Let me ask you, what else does safe mode do? I would say at this point whatever it’s doing is obviously stopping whatever it is causing the reboot.  I wonder if some plugin has an issue. It would seem strange. I don’t run anything crazy. Not sure where to go from here. I haven’t gone this many days in over a year. 

[JorgeB]

8 hours ago, Dro said:

what else does safe mode do?

With v6.12 it just disables the plugins, nothing else.

[Dro]

37 minutes ago, JorgeB said:

With v6.12 it just disables the plugins, nothing else.

Thanks I’m going to take it out of safe mode today and see if the reboots comeback. If they do then I’d assume it’s some plugin. 

[Bfox135]

Shot in the dark. 
are you using macvlan or ipvlan in your docker?

[Dro]

19 hours ago, Bfox135 said:

Shot in the dark. 
are you using macvlan or ipvlan in your docker?

In the orginal post I mentioned IPLVAN in the list.

[Dro]

On 6/30/2024 at 4:52 AM, JorgeB said:

With v6.12 it just disables the plugins, nothing else.

Not 1 hour after taking it out of safe mode, it randomly rebooted unclean shutdown =( 

Ran for 6 days straight, all dockers enabled, no plugins.. So its obviously some kind of plugin issue.. Anyway to just disable them without removing them completely? I dont have anything out of the ordinary in there.

Here is a list. Maybe there is a known rebooting issue with one of these? Shot in the dark. 

Appdata Backup

CA Auto Turbo Write Mode

CA Auto Update Applications

CA Mover Tuning

Community Applications

Dynamix Date Time

Dynamix File Integrity

Dynamix File Manager

Dynamix System Buttons

Dynamix System Information

Dynamix System Temperature

Fix Common Problems

GPU Statistics

Intel GPU TOP

NerdTools

Parity Check Tuning

Recycle Bin

Theme Engine

Tips and Tweaks

Unassigned Devices

Unassigned Devices Plus

unbalanced

Unraid Connect

User Scripts

 

[itimpi]

You can disable a plugin by changing its file extension in the config/plugins folder on the flash drive to be something other than .plg and then rebooting the server.   Reverse the process to get it enabled again.

[Dro]

3 hours ago, itimpi said:

You can disable a plugin by changing its file extension in the config/plugins folder on the flash drive to be something other than .plg and then rebooting the server.   Reverse the process to get it enabled again.

Maybe for future updates add a slider to disable/enable 😃 Any thoughts if any of those listed plugins may be a potential issue? 

[JorgeB]

11 hours ago, Dro said:

GPU Statistics

Intel GPU TOP

NerdTools

These would be the ones I would start with, especially the GPU related ones

[Dro]

1 hour ago, JorgeB said:

These would be the ones I would start with, especially the GPU related ones

Thank you! Will start there. 

[guiper]

Any update here? Also going crazy as I have exactly the same issue. I bough an entire new PC trying to sort it out (it was due to be honest) and even with a completely new setup (only 4 disks and PSU cables remain from the old setup).

Every 24 to 48h the computer "reboots" alone and is waiting for the password to start the array. Nothing in syslog. 

Just tried to disable some power efficiency optimizations I added a few months ago, but in my case it must be software related, since everything is new hardware 
 

Edited July 3 by guiper

[Dro]

On 7/3/2024 at 4:06 AM, guiper said:

Any update here? Also going crazy as I have exactly the same issue. I bough an entire new PC trying to sort it out (it was due to be honest) and even with a completely new setup (only 4 disks and PSU cables remain from the old setup).

Every 24 to 48h the computer "reboots" alone and is waiting for the password to start the array. Nothing in syslog. 

Just tried to disable some power efficiency optimizations I added a few months ago, but in my case it must be software related, since everything is new hardware 
 

Nope, i am SO frustrated with it!!! Disabled those 3 plugins as suggested, still rebooting. Literally if i am doing one plugin a day ill be here weeks trying to see which one may or may not cause it.. Kind of ridiculous.. Has to be a better way to figure this out. All i know in safe mode i was running for 6+ days zero issues with all dockers enabled.. as soon as i returned to regular mode not a few hours later the reboots started happening again.. so it has to be some kind of plugin.. Maybe i will just remove everything and add things back as i go. Maybe that will be more efficient.. Either way i am so over it 😡

[Geck0]

PSU issue. I had the same problem a couple of years ago. I replaced it with a Corsair 1500. I purchased a spare one that is sitting in my cupboard, just in case. 

The issue you're describing is almost identical to what I was experiencing. Its either a load issue or your PSU is on its way out.

[Dro]

Just now, Geck0 said:

PSU issue. I had the same problem a couple of years ago. I replaced it with a Corsair 1500. I purchased a spare one that is sitting in my cupboard, just in case. 

The issue you're describing is almost identical to what I was experiencing. Its either a load issue or your PSU is on its way out.

But these issues dont happen in safe mode.. If it was hardware related I think it would still be happening. Only thing being disabled are the plugins in safe mode and system runs stable.. I would think PSU issue too and ill probably replace it just incase eventhough it was bought new a few years ago. EVGA 850w.

[Geck0]

Well, look at it this way. If it turns out to not be the PSU, then you have a spare. 

It does sound awfully familiar though. Mine was going down randomly. It started off as reboots and then eventually it became crashes.

It could be the case that you're running something like the dynamix file check and the load is tipping the PSU over the edge. Especially if the capacitors are giving out. 

Any coil whine or ticking noises?