Is there a BIOS upgrade available for the MB?  Are you using the GPU MB slot for the GPU?

51 minutes ago, Frank1940 said:

Is there a BIOS upgrade available for the MB?  Are you using the GPU MB slot for the GPU?

BIOS is up to date. To my knowledge, the bottom pcie slot in my mobo is the only one that does not support use of a graphics card (mob is Tyan S7012)

1 hour ago, itimpi said:

This strongly suggests something at the hardware level.  Culprit I would first look for would be inadequate cooling causing a thermal related shutdown or problems with the power supply.

@itimpi unfortunately this is the usual answer 🙂 among other things, in some ways it is also correct if it is really related to the GPU ... but why does it happen? it doesn't seem to be really an hardware problem. 

 

My CPU runs constanty at 5/15% of load with peeks of maximum 30% during monthly parity checks or activity like this the rest of the time it runs a handful of docker apps. The power consumption is 45/65W at the socket with a PSU of 750W it is far from being a problem.

 

also i am wondering, but really any problem causing a system crash would be logged? Is there anything that can escape this logging system but still caused by a software issue (interacting with a piece of hardware)? is it really impossible? 🤔

 

 

 

 

 

 

40 minutes ago, alexbn71 said:

also i am wondering, but really any problem causing a system crash would be logged? Is there anything that can escape this logging system but still caused by a software issue (interacting with a piece of hardware)? is it really impossible? 🤔

 

I seem to recall that a Docker can cause this type of problem.  Have you tried the Safe Mode with the GPU installed? 

2 minutes ago, Frank1940 said:

I seem to recall that a Docker can cause this type of problem.  Have you tried the Safe Mode with the GPU installed? 

I haven't tried it yet. I keep that in mind, thanks

 

16 hours ago, Frank1940 said:

I seem to recall that a Docker can cause this type of problem.  Have you tried the Safe Mode with the GPU installed? 

@2Piececombo have you ever suspected that docker might play a role with this issue? In some of my tests (not sure a 100%) I tried to stop docker service and the issue did not arise in this case. however, I have not done tests that lasted many days. I've also tried with docker service up and running and all docker apps stopped and it happened again

Edited August 20, 20223 yr by alexbn71

7 hours ago, alexbn71 said:

@2Piececombo have you ever suspected that docker might play a role with this issue? In some of my tests (not sure a 100%) I tried to stop docker service and the issue did not arise in this case. however, I have not done tests that lasted many days. I've also tried with docker service up and running and all docker apps stopped and it happened again

I have not run in safe mode with the GPU installed, but I guess I should give it a go.

 

@Frank1940 suppose this is the cause, are you aware of any remedies for this problem? Or what causes it? I've never heard of an issue like this with docker, but I'm clearly no expert. 

2 minutes ago, 2Piececombo said:

suppose this is the cause, are you aware of any remedies for this problem? Or what causes it? I've never heard of an issue like this with docker, but I'm clearly no expert. 

 

I am no expert either.  I just do a lot of reading of threads on the forum and this gives me some insight into various problems and I often remember some details about their solution.  (I have a very simple NAS only setup and I don't have a lot of in-depth knowledge of about the interworkings of the more complex setups that many folks are using.  But isolating these types of problems to one of the subsystems is the first step to a solution.  For example, there seem to be hundreds of Docker Containers for Unraid.  I suspect that many are barely supported for many reasons...)

12 hours ago, Frank1940 said:

I suspect that many are barely supported for many reasons

I do not agree. The concept behind docker makes it a universal system in itself. I don't see why docker should run smoothly on a Raspberry Pi or any other hardware/linux distribution (and windows) and not on Unraid. I've never had issue like this with Docker on other linux systems.

 

...and I also want to remind you that I only kept the docker service active without any of my apps active and the problem occurred anyway. This should rule out any of them as guilty, right?

Edited August 21, 20223 yr by alexbn71

Powered down my server and reisntalled the GPU, booted back up normally, and within 5 mins it shutdown again. It's 100%, without a doubt, related to the GPU. I will try again but booted into safe mode and post results.

 

@alexbn71 I wanted to ask you, when you have a GPU isntalled, does your onboard graphics display the unraid boot process/login screen normally? My BIOS is set to force display out the onboard graphics (so unraid does display login/console out the GPU display) and it works as normal, up until the boot process is complete and it goes to the login screen. I just get a black screen and am not able to use the local console. The webui is still fine and accessible, but not locally. Do you have this same behavior? I thought it might be something to do with the nvidia driver plugin, so I posted on the support page. The author claimed the issue was due to me still being on 6.9 and said the issue would go away once I upgraded to 6.10.3.. Well, it didn't. Curious if you have this issue as well

Small follow up, I've booted into safe mode, started array and docker, and it has not crashed, and the onboard graphics is displaying the console as normal. This makes me think that the nvidia driver plugin is somehow to blame for my issues.

 

Thoughts?

7 hours ago, 2Piececombo said:

@alexbn71 I wanted to ask you, when you have a GPU isntalled, does your onboard graphics display the unraid boot process/login screen normally?

My currently MB doesn't have an integrated graphics. I have it on a new MB+CPU+RAM that I prepered as last chance to resolve this issue

8 hours ago, 2Piececombo said:

This makes me think that the nvidia driver plugin is somehow to blame for my issues.

So it should be a combination of factors unless there are hundreds of people out there with the same problem ... isn't that right? 

On 8/22/2022 at 2:55 AM, alexbn71 said:

So it should be a combination of factors unless there are hundreds of people out there with the same problem ... isn't that right? 

Possibly, unless it's somehow specific to certain GPUs. Both the GPUs I had wer older, a Quadro 600, and then a Quadro P600. I dont recall if you specified what card you have, but maybe it's related to this. I'm going to try a different card at some point and see what happens, though I have a very interesting update..

 

My server ran for at least 12 hours in safe mode with array started and docker running. Only thing disabled were plugins I believe. Then at some point I heard the beeps my mobo makes when it reboots. I presume it booted back to normal (not safemode) and then booted/crashed repeatedly til I just turned it off. I haven't had any time to look into why it might have shut off in safe mode, but when I power it back on ill grab diags and the mirror to USB syslog and post them here.

 

Well I'm at a complete loss now... Finally had some time to look into what going on. My IPMI is now showing over 400 events. (see pic below)  The same two things over and over again.. But the timestamps make absolutely no sense. I booted up the server, and when I tired to access the webui i get an nginx error. I log into the terminal and the aspect ratio is super wierd, it shows I have no dockers isntalled, all my cache drives are showing as unassigned, and it freezes, then reboots. This is with no GPU installed, btw. So where before I would only have issues WITH a gpu, it's not just gone to absolute shit. I think im going to have to backup all my data and assume this server is going to fail completely sooner than later. I dont think its going to stay alive long enough for me to grab diags anymore, so maybe syslog will give some idea.... Question, does the syslog include any sensitive info? Or am I goo to post it? I've not seen anything sensitive as I scrolled through it, but I want to be sure before I post it. 

 

Also, I watched the entire boot up process, and something about CPU errors. I have another pair of CPUs im going to drop in and see what happens.

Wish me luck...

 

8 hours ago, 2Piececombo said:

I dont recall if you specified what card you have,

GTX650... 10 years ago?

 

I'm sorry for the latest disaster 😔 It happened to me with multiple reboot that something broke. Luckily I stopped the loop after 3 or 4 restarts and the damage had been slight

I've found another even poorer video card "ASUS EAH 5450". After 10 days without a GPU and no issue on my NAS I've installed this "new" card... let's see what happens

I thought the CPU error I was seeing was related to memory that needed to be reseated. During the CPU swap a RAM module must have developed a poor connection, because after re-seating them all it was fine and the server ran for about 9 hours without issue, then about 2 minutes ago it rebooted out of nowhere again. Now Im seeing the CPU errors again. Im going to have to assume the motherboard is faulty, I don't know what else it could be. The strangest part to me is that it has never once crashed while booted into windows/linux/etc. ONLY unraid. Regardless, I'm going to replace the mobo. I said previously that the mobo had been replaced, and that's sorta true. I had ordered another, installed it in this server, and still experienced a problem (this was 8-10 months ago) so I put the original mobo back in and built a second server with the new mobo. It's a Tyan S7012, which are still available new on ebay. Im going to pick up yet another one, swap the board out, and see what happens. If I still have problems im going to nuke this install of unraid completely and start over. I dont have the space to completely back up my media library, but oh well that can always be.... reacquired

Not an expert, but i was facing the same issue. Basically under high load my server was randomly shutting down. I have connected the server to a APC UPS. When i dug deeper into my UPS settings, i realised that i had set the system to shut down when only 10 mins of runtime was left. However upon tracking my UPS status, under high load, my UPS was giving me a runtime of 8 minutes (less than 10 minutes). I believe this is what was triggering the shutdown. 

I disabled the APC daemon and this seemed to fix the issue. 

If you are using a UPS, maybe this could be the problem. Let me know if you would like more details ! 

Edited November 23, 20241 yr by Astrosterone
Missed important info

5 minutes ago, Astrosterone said:

disabled the APC daemon

Not the best approach.

 

The best way to configure your server with UPS is to have the server shutdown when the power has been off for a very short time. The purpose isn't to allow you to run on battery, the purpose is to allow a clean shutdown.

 

 

One more thing to check is to actually determine the time required for the array to stop.  You do this clicking on the STOP bottom on the Main tab and measuring the time.

 

Then go to    SETTINGS   >>>   Disk Settings    and look for the       setting and make sure that it is at least 25% larger than the time you measured.