[SOLVED] Unraid keeps crashing

[aneelley]

I just built a new server.  I have experienced three crashes after I have the system up and running with the following Docker containers:

nzbget

Plex

sonarr

radarr

 

I suspect the CPU is overheating because I have seen three of its threads constantly at 100% when on the Dashboard.  Also, the FixCommonProblems plugin notified me of the following:

 

 

"CPU possibly will not throttle down frequency at idle - Your CPU is running constantly at 100% and will not throttle when it's idle (to save heat /  power). This is because there is currently no CPU Scaling Driver Installed.  Seek assistance on the unRaid forums with this issue.

 

So that brings here here this evening.

 

I also see the following two issues related to cache:

 

 

unraid-diagnostics-20200430-2021.zip

[aneelley]

I should also mention that I can reproduce the issue at will by just starting the nzbget container.  It will peg one or more of the threads on the CPU and after a few minutes, the array will lock up to where I have to press the reset button.

[trurl]

1 hour ago, aneelley said:

I just built a new server.

Is this a different server than the one I was helping you with last week?

[aneelley]

@trurl Yes it is.

[trurl]

You have your appdata, domains, and system shares setup wrong.

1 hour ago, aneelley said:

I suspect the CPU is overheating

Do you have Dynamix System Temp plugin setup? Or do you have some other way to monitor the temp, such as IPMI?

[aneelley]

The disks (except for the cache disk) are the same.  The flash drive for booting Unraid is the same.  The rest has been upgraded.

[trurl]

10 hours ago, aneelley said:

The flash drive for booting Unraid is the same.

So is this a new install on the same flash? Why did you change the Use cache setting on appdata, domains, and system shares from the default cache-prefer?

[aneelley]

2 minutes ago, trurl said:

You have your appdata, domains, and system shares setup wrong.

Do you have Dynamix System Temp plugin setup? Or do you have some other way to monitor the temp, such as IPMI?

I am not monitoring it at all.  I just installed Dynamix System Temp plugin.  

[aneelley]

Just now, trurl said:

So is the a new install on the same flash? Why did you change the Use cache setting on appdata, domains, and system shares from the default cache-prefer?

I kept filling up the cache (only 400GB) so I removed it.  I now have a new nvme 2TB cache in there.

[trurl]

4 minutes ago, trurl said:

You have your appdata, domains, and system shares setup wrong

We didn't have to go into fixing that in your other threads because you hadn't changed them from the default cache-prefer, and those shares were all already all on cache where they belonged. Now you have some of each of them on the array where they will impact your docker / VM performance due to the slower parity writes, and where they will keep array disks spinning.

 

And it is a multi-step process getting them all on cache, possibly with some additional cleanup.

 

But I guess we can go through all that later.

 

Have you taken care of these Ryzen tweaks?

 

 

[aneelley]

1 minute ago, trurl said:

We didn't have to go into fixing that in your other threads because you hadn't changed them from the default cache-prefer, and those shares were all already all on cache where they belonged. Now you have some of each of them on the array where they will impact your docker / VM performance due to the slower parity writes, and where they will keep array disks spinning.

 

And it is a multi-step process getting them all on cache, possibly with some additional cleanup.

 

But I guess we can go through all that later.

 

Have you taken care of these Ryzen tweaks?

 

 

I have not and I will do that stat.  I think that is the main thing.  I am cleaning up the cache right now.  Got the mover running.  Just a little bit of stuff on there.  34MB maybe.

[aneelley]

I just completed this:

Ryzen on Linux can lock up due to issues with c-states, and while this should mostly affect 1st gen Ryzen there are reports that 2nd and even 3rd gen can be affected in some cases, make sure bios is up to date, then look for "Power Supply Idle Control" (or similar) and set it to "typical current idle" (or similar).

 

Also, everything is moved off cache and I have all shares set to No for using it.  I am going to turn up the nzbget container and see if I can make Unraid crash again.  If I can get past that, the crashing is solved and I just need to properly set up the shares for using the cache drive.

[aneelley]

I typically see the following on the CPU just before unraid crashes:

 

I captured this with only nzbget running.  It is uncompressing a movie.

[aneelley]

I should also state that I stopped the container (took it a few minutes to stop) before it crashed the system.  The CPU is barely doing anything (1%) now.

 

[trurl]

33 minutes ago, aneelley said:

I typically see the following on the CPU just before unraid crashes:

 

I captured this with only nzbget running.  It is uncompressing a movie.

That doesn't look too bad. You typically would want a lot of processing power for post-processing downloads. That shouldn't make it crash though. I don't have nearly that many cores but it is not uncommon for some to go red like that.

 

Did you install this CPU and cooler yourself?

[aneelley]

My buddy and I installed the CPU and cooler.  I have built many systems over the years.  I now have nzbget running.  So far, no crashes.  I am going to let it run through the night but I believe this fixed the issue with the crash:

Ryzen on Linux can lock up due to issues with c-states, and while this should mostly affect 1st gen Ryzen there are reports that 2nd and even 3rd gen can be affected in some cases, make sure bios is up to date, then look for "Power Supply Idle Control" (or similar) and set it to "typical current idle" (or similar).

 

If it goes through the night, I will call it successful.

[aneelley]

Here is a new diag.  We can tackle the cache setup tomorrow.

unraid-diagnostics-20200430-2336.zip

[JorgeB]

Also check the FAQ link for max RAM speeds, you're currently overclocking your RAM.

[aneelley]

Ok, I did see that but was not sure if I was or not.  I'll have to go into the BIOS and set it.  I could not tell which one I needed as I really don't know what generation Ryzen that I have.  I bought it new this week.

 

The system is still running this morning.

Edited May 1, 2020 by aneelley

[aneelley]

I set the cache on the shares.  Hopefully I got it right.  Would appreciate any optimization or correction advice.  New diags attached.

unraid-diagnostics-20200501-0845.zip

[trurl]

No that is not ideal. Your appdata, though you have set it to cache-only, currently has all of its files on the array. Ideally all its files would be on cache. And your domains and system shares are cache-no with all their files on the array, but ideally they would have all their files on cache and set to cache-only.

 

As I said, it will be a multi-step process to get this changed, with possibly some manual cleanup down at the disk level. Mover ignores cache-only and cache-no shares. It moves cache-prefer files from array to cache, and moves cache-yes shares from cache to array.

 

To get those appdata, domains, and system shares moved to cache they will have to be set to cache-prefer. But, mover also cannot move open files. So before they can be moved, dockers and VMs will have to be disabled; not the individual dockers and VMs, but the services themselves in Settings - Docker and Settings - VM Manager. Then finally, mover won't move duplicates, so if some of these exist on both cache and array you will have to delete the duplicates from the disk level.

 

I was glad when we were working through your problems last week that you did not have these problems at that time. Now you have made them and will have to clean them up if you want the ideal setup for your dockers and VMs.

[trurl]

1. Go to Settings - Docker and disable. Go to Settings - VM Manager and disable.
2. Set appdata, domains and system shares to cache-prefer. Probably better set your downloads share to cache-no at least for now.
3. Run mover. Wait for it to complete.

 

Post new diagnostics.

[aneelley]

11 hours ago, johnnie.black said:

Also check the FAQ link for max RAM speeds, you're currently overclocking your RAM.

Hey @johnnie.black Can you tell me what the proper memory speed is for my system?  I can't understand that guide.

[aneelley]

6 hours ago, trurl said:

1. Go to Settings - Docker and disable. Go to Settings - VM Manager and disable.
2. Set appdata, domains and system shares to cache-prefer. Probably better set your downloads share to cache-no at least for now.
3. Run mover. Wait for it to complete.

 

Post new diagnostics.

Mover is still running.  I will follow up once done.

[aneelley]

Quick question, should I have left the docker and VM services stopped before running the mover?  Just making sure I didn't need to start them up first.