Good day, ever since upgrading to 6.11 the server has experienced sporadic shutdowns. I was eventually able to see the behaviour while sitting at the screen and the culprit seems to be CPU overheat.

 

I have checked fans and cleaned filters, but today the server shutdown again. 

 

A change I noticed is that the asus motherboard temperature sensors, now come through to unraid. Not sure it has any bearing. 

 

Any guidance is much appreciated.

 

syslog-192.168.99.100.log unraid-diagnostics-20221022-1353.zip

Are you over-clocking the CPU?

 

Over-clocking is not recommended for Unraid. 

@Frank1940

Thank you for your response. I'm not overclocking, i dont even game.

1. install the "Dynamix System Temp" plugin and perl (via nerdTools) to be able to detect correctly your motherboard sensors.

2. give us a reading of the cpu temp e.g. at half load.

3. as quick fix you can install "Tips and Tweaks" plugin and set "Normal CPU Scaling Governor" and "Power Saving CPU Scaling Governor" to "Powersave".

@doobyns

Thank you for your help. 

I have the dynamix system temp plugin installed and perl comes installed by default in 6.11. Nerd tools has been deprecated

I also have tips and tweaks and will try the governor scaling tip. It was set to performance, so I'll modify and monitor.

How did you determine you are overheating. I don't see anything in the information provided that indicates that. By chance did you set anything in your bios that may do automatic overclocking. 

 

It is kind of unlikely to be the problem unless something is very wrong with the system. Modern CPU's have things built into them that should make this very difficult unless you have a extremally insufficient cooling solution and even then unlikely. Modern cpu's should speed up and slow down the clock based on usage and thermal headroom.  Simply put you should be able to even leave your cooler off and the system would run just extremely throttled. Though i will admit i would never want to test this.

 

Have you checked your power supply. What is it rated to run at, and how old is it, and what is the normal system power consumption. Do you have a GPU in the system?

2 hours ago, juan11perez said:

Nerd tools has been deprecated

It is NerdPack that is deprecated.   Nerd Tools is a replacement for the 6.11.x Unraid releases.

@itimpi thank you for the correctiont. Nevertheless pearl is installed.

 

@mavrrick thank you for your help. I determined it overheat, because is saw tctl escalate pass 90 degrees and got distracted with something else; following which the unit shutdown.

I do have 3 GPUs. 1  on my daily driver, 1 on full transcoding, 1 for the occasional spare desktop duty.

Thing is, this has all been runing fine for 2 years. Then again "all it's fine until it isn't".

 

 

Video cards can be particularly harsh on power supplies. Where any of them doing any activity during the shutdown.  Transients loads made by modern video cards can really bring a Power supply to it's knees. What video cards do you have.  What teir power supply do you have. What is the powersupply brand/model and supported wattage of the device. It would also be nice to know what the CPU load at the time of the shutdown was. 

@mavrrick once again thank you for your time.

I have the following cards:

 

GeForce GTX 1050 Ti - transcoding plex,frigate (on 24hr)

GeForce GTX 1650 - archlinux daily driver (on 24hr)

Radeon RX 470/480 - spare vm (on ocassionally)

 

Also i have Coral Edge TPU - working on frigate.

 

Power supply is EVGA 850 B3, 80 Plus BRONZE 850W,

 

Difficult to say exact demand at the time, but I dont believe cpu was over 30% utilisation.

 

OKay, So the power supply is looks to be near the middle to the top middle of the tier list. So that is pretty good. 

 

The typical power consumption of the graphics cards under load appears to be around 335 + the TPU. Not sure of the best way to look at up. I don't remember hearing to much about transient load prior to the rtx 20 series so that may not be related. 

 

It would be nice to see the dashboard section for the CPU close to this event. A few snapshots that show system at idle vs the load of the system at around the time this happens.

 

I have noticed on my system the time the system hits the hightest temps is when I have a few cores boosting to max clock. That will drive the CPU up as high as it can't go. When the entire system is under load it settles much lower. Here is what I mean.

 

This is the system at relative idle os just doing simple tasks. Hovering around 33-34 degrees

 

 

Under significant load it settles around 46 degrees. This is load created from setiathome using all of the cores

Okay, so as I was trying to create the condition I know I observed previously my system is making a liar out of me. I have confirmed it is boosting all the way to 4.5Ghz on 8 threads and the temps are staying below the highest I have seen of 46.  My fans are ramping up allot though so one question i would ask is what is your fan profile like. Do you hear your fans ramp up, or have you made keeping it all as quite as possible a priority. The reason i saw this before could be simply related to the fan profile i had i guess. I think in the past i did try to put a focus on noise reduction.

 

Something else that has come to mind is have you disabled Global C-States in your bios. That is something I needed to do for stability as well when using my GPU and has anything changed recently that could have enabled it. 

 

I still think it could be worth trying a different power supply. It isn't a hard stretch that under heavy load even 850 could be stretched thin with that quantity of hardware. Even though it is rated at 850 over time power supplies do loose a bit of their headroom. it is possible you have enough spikes in load that cause it to kick in overload protection and drop. I had a similar issue some 12 years ago with a WHS Home server i built with what i thought was a top teir power supply at the time. It ran great until it didn't. 

 

Also out of curiosity what is the cooling solution you have for your CPU Air cooled/Water cooled? 

Edited October 22, 20223 yr by mavrrick

try the following:

* create a VM

* from inside the VM download something (over the LAN, not from UNRAID)

* watch CPU Temp

 

I have almost the same machine (5700G instead of your 3900) and it is perfectly going through the roof (85° and more) if there is a VM doing some heavy net transfers like downloads.

 

check it out.

 

@mavrrick

@MAM59

For CPU cooling I have a "be quiet" tower cooler.

Thank you very much for taking the time to write your recommendations. It is much appreciated. 

 

I have reviewed all these points and I think the answer appears to be much simpler than anticipated.

 

I have had this server in this configuration for about 2 years. About 1 year ago I moved to a new place and the server  to a spot where I now realised it was not getting proper air flow. It had not started shutting down until recently but apparently the issue had been providing signs i couldn't work out until now.

 

I live in a hot climate and the server is in my office, which is airconditioned during the day. At night the AC is off and the door is shut. Pretty much every day I noticed in the glances panel that at around 3am the iowait would increase which I now presume was related to  throttling back. This has been going on for months and I really had no clue.

 

After following the recommendations I decided to change the server location. The immediate result was a temperature drop of about 5 degrees; but more revealing the above mentioned iowait error disappeared.

 

I'll keep monitoring and hopefully this is the solution. Thank you again for taking the time to look into my issue.

 

On 10/22/2022 at 2:14 PM, MAM59 said:

try the following:

* create a VM

* from inside the VM download something (over the LAN, not from UNRAID)

* watch CPU Temp

 

I have almost the same machine (5700G instead of your 3900) and it is perfectly going through the roof (85° and more) if there is a VM doing some heavy net transfers like downloads.

 

check it out.

 

i actually have been having a similar problem except without even any VMs. i have an intel pentium gold 8505 in a ugreen NAS, running unraid, and my CPU has been going to 90+ when all i'm doing is qbittorrent and sabznbd running/downloading. I've also been experiencing a weird problem where my twingate connector randomly goes offline and i think it may have to do with this CPU overheating. I'm not sure how to fix this, maybe my bios fan settings are messed up but i never changed them.