I have had trouble with my server staying stable; it randomly hard crashes and I set up a syslog server to try and capture the reason as diagnostics can't be taken until after a hard reboot; they are full of segfault 'likely on CPU'.  Upon researching the segfault error others have reported it is their CPU.  Memory tests have passed before, but I will launch another one.  

 

Not going to provide the other logs as their last entries was last night, but the kernel.log has entries from today and its all segfaults.  I can provide any other logs if wanted; I just don't want to replace a CPU for no reason.  I presume CPU would be replaced first before mobo or at least I hope so.  As for the other weirdness ive had a lot of problems with Docker image becoming constantly corrupt/failing to stop when told to, or the *arr databases corrupting as well.

 

Anything else I should do to troubleshoot?

tower-diagnostics-20241210-1031.zip

Edited December 10, 20241 yr by codexus
add diagnostics

memtest is only definitive if it finds errors, since you have multiple sticks try using the server with just one, if the same try with a different one, that will basically rule out bad RAM.

another thing to consider is power... have you added anything new to the system recently? new HDD?

 

something to consider

35 minutes ago, mathomas3 said:

another thing to consider is power... have you added anything new to the system recently? new HDD?

 

something to consider

I have not, recently I replaced the array with new drives as the previous ones were giving SMART errors and I had thought that was contributing to the crashes, but they were replaced with a like number and the hard crashes have not changed one way or the other.  I have a 500w thermaltake PSU currently, I am not sure if that is considered insufficient for five drives.  No other add-on hardware just my mobo, processor, ram, and five drives and of course system fans.

If you have a spare PSU to try... it might be worth your time to try it... else you could try the stress the system and see if it crashes then... 

 

Without creating a new USB bootdisk... you could run a parity check, should it crash then... try it one more time just to CYA... if it crashes the second time... I would say it's good to assume that you need a bigger PSU... 

 

It's been over ten years... but I did have a 500/550 Watt??? PSU... and that was with 6hdd I think... it's so long ago I cant recall correctly... but I feel you are on that edge

1 hour ago, mathomas3 said:

If you have a spare PSU to try... it might be worth your time to try it... else you could try the stress the system and see if it crashes then... 

 

Without creating a new USB bootdisk... you could run a parity check, should it crash then... try it one more time just to CYA... if it crashes the second time... I would say it's good to assume that you need a bigger PSU... 

 

It's been over ten years... but I did have a 500/550 Watt??? PSU... and that was with 6hdd I think... it's so long ago I cant recall correctly... but I feel you are on that edge

Thats a great idea... parity is currently running and has five hours left.  Tonight I will run it again and see if it crashes overnight, that may just give my answer if it does.

Unfortunately I dont have spare CPU or PSU so it will be buy one then the other if the problem isn't resolved.  Sucks, but if the Parity Check(s) don't crash then I have to start replacing hardware, which ever is cheapest first then work my way out.  Was hoping the logs would be definitive about the problem.

 

Edit:  I have in the past replaced the Unraid OS USB stick too just in case that was the issue.  

Edited December 10, 20241 yr by codexus

21 hours ago, mathomas3 said:

If you have a spare PSU to try... it might be worth your time to try it... else you could try the stress the system and see if it crashes then... 

 

Without creating a new USB bootdisk... you could run a parity check, should it crash then... try it one more time just to CYA... if it crashes the second time... I would say it's good to assume that you need a bigger PSU... 

 

It's been over ten years... but I did have a 500/550 Watt??? PSU... and that was with 6hdd I think... it's so long ago I cant recall correctly... but I feel you are on that edge

First parity check went fine, the second one overnight caused a crash.  So... power supply it looks like.  

 

Would a 700w be good to go, or should I go to 800 or more?

23 hours ago, codexus said:

First parity check went fine, the second one overnight caused a crash.  So... power supply it looks like.  

 

Would a 700w be good to go, or should I go to 800 or more?

Meh... what ever you can afford... personally... I went overkill and got a 1000 watt PSU... I never wanted to question the PSU again... given that you are running a 500 watt currently... and you want to have room to expand... IMO... 600 is enough if you dont intend to expand/add HDD to the system but I would go with a 750 watt IMO... I seem to remember that per each hdd you want to budget 25 watts

 

anyways... IMO get a new PSU and stress/parity check the system a number of times... if it crashes report back here and return the 'newish' psu.... 

A 500W PSU should be plenty for 6 HDDs... A bigger PSU isn't necessary better for more HDDs if your PSU doesn't have enough 12v rails to handle them.

 

You need to look at the +12V rail on the current PSU sticker to see how many Amps are on it.

 

This example PSU has 600W on the 12V Rail and is rated for 600W. That's up to 60 10W HDDs as long as you don't exceed the Amps. (Look at drive start up amps)

Edited December 12, 20241 yr by MowMdown

Just now, MowMdown said:

A 500W PSU should be plently for 6 HDDs... A bigger PSU isn't necessarly better for more HDDs if your PSU doesn't have enough 12v rails to handle them.

 

You need to look at the +12V rail on the current PSU sticker to see how many Amps are on it.

User could also have a GPU... just saying

36 minutes ago, mathomas3 said:

User could also have a GPU... just saying

 

I mean yeah if you were putting a gaming load or LLM load on it sure it could be using 100+ Watts but if you're just doing encoding/decoding say in plex, it probably pulling <50W and I can't imagine it would be enough to cause crashes. But it's something to 100% consider too.

1 minute ago, MowMdown said:

 

I mean yeah if you were putting a gaming load or LLM load on it sure it could be using 100+ Watts but if you're just doing encoding/decoding say in plex, it probably pulling <50W and I can't imagine it would be enough to cause crashes. But it's something to 100% consider too.

Funny thing is that 1000 watt PSU I mentioned... it's 13 years old and I intend to reuse it in a AM5 build... just have to put the hardware together... so yea... well worth the money having a quality PSU that isnt just enough... 

 

that was 2011 and I spent something like $250 at the time from the colorado springs best buy... not cheap but yea... when you need something... you need it

1 hour ago, MowMdown said:

A 500W PSU should be plenty for 6 HDDs... A bigger PSU isn't necessary better for more HDDs if your PSU doesn't have enough 12v rails to handle them.

 

You need to look at the +12V rail on the current PSU sticker to see how many Amps are on it.

 

This example PSU has 600W on the 12V Rail and is rated for 600W. That's up to 60 10W HDDs as long as you don't exceed the Amps. (Look at drive start up amps)

 

PSU says 35 amps on the 12v rail... maddeningly I cant find startup amperage on the drives, but the spec sheet says 11.3 watts for read/write and dividing that by the voltage gives me about 1amp per drive reading/writing.  That could be wildly off vs startup, but I cant find anything about amperage let alone startup amps.  

 

Presuming read/write is max-ish usage were looking at 5 out of 35 amps running the drives.  

 

 

1 hour ago, mathomas3 said:

User could also have a GPU... just saying

No GPU, just the fans, mobo, processor and drives sucking up power.

 

Nothing particularly power hungry to my knowledge, old tech.

 

Processor: Intel Pentium Gold G-6400 4ghz.

Motherboard: ASROCK Z590 PRO4

Memory: G.Skill Aegis 2x8GB

 

Edited December 12, 20241 yr by codexus

When looking at PSU current ratings for each voltage buss, you have to look beyond the first one you that you find!  For example, @MowMdown posted up this one:

 

Now look as you go down the rating chart.  If you pull 600W (50A), you can only draw 11.5W from both the +3.3V and 5V buss combined-- virtually no power for the MB.  Rosewill is not the only manufacturer to do this type of specmanship.  Every PSU I have ever looked at does it.  By the way, all PSU's have overcurrent shutdown circuitry designed in.  Exceed that setpoint current for a few milliseconds that the PSU will normally completely kill that buss' output!

21 hours ago, Frank1940 said:

When looking at PSU current ratings for each voltage buss, you have to look beyond the first one you that you find!  For example, @MowMdown posted up this one:

 

Now look as you go down the rating chart.  If you pull 600W (50A), you can only draw 11.5W from both the +3.3V and 5V buss combined-- virtually no power for the MB.  Rosewill is not the only manufacturer to do this type of specmanship.  Every PSU I have ever looked at does it.  By the way, all PSU's have overcurrent shutdown circuitry designed in.  Exceed that setpoint current for a few milliseconds that the PSU will normally completely kill that buss' output!

What pulls from the 12v?  Is just the mobo pulling from 3.3v and 5v rails?  I am trying to figure out how big of a PSU to get or maybe just hitting the easy button and getting a 850 though the math I am working (which may be wrong) my drives should never be pulling 35amps/420 watts with just a CPU and five drives.  

 

I don't understand power supplies well, if it isnt obvious.  Just get the 850 and call it a day?  750?  Is it not possible for the CPU to be at fault here?  

The example was a single rail supply.  (Most PS today are single rail.  A few years ago, dual-rail  +12V busses were common.)  What pulls from it?  Generally the HD's and the GPU.   (The Motherboard may use a bit of +12V current.)  With hard drives, there is an issue with inrush current when the drives spin up.  With Unraid this can be a problem since all of the HD's will be spun-up whenever any parity type operation starts.  The peak inrush current is about 2-3 amperes per drive depending on a lot of factors.  (Finding the inrush current from the manufacturer's spec sheet is a hit-and-miss situation.)  If this current exceeds the max current rating for even a few milliseconds, the PS will probably go brokers!  Remember that your MB's  CPU, RAM and ancillary chips all require power as do the case fans.   You can get a feel of the baseline load on your PS by looking at the power draw on the UPs settings.  (One of my systems is idling at about 61W right now.)

 

You have 5 HD's on your system right now so the +12V current requirement is 15A max (75W  180W)!   Your CPU is rated at about 60W.  That 450W should be enough unless you have a powerful GPU card.  However, that is not to say that the PS might not have issues.   Modern PS are not just a transformer, rectifier, smoothing capacitors and a couple of voltage regulators any more.  They interact with the MB in many ways.  Crashes, random shutdown and random reboots have been caused by 'bad' PS.  They are one of the easier components to replace.  (RAM is always the first choice here.)

 

One more thing.  Have you cleaned the case of dirt and dust?  Blown out the CPU cooler.  Made sure that all the fans were running.

Edited December 13, 20241 yr by Frank1940
Can't muliply 3X5X12!!!!!

Update: I ended up doing another MEMTEST and it failed so the RAM must have gone bad from the first test, should have checked that first.  

 

Replaced the two sticks, they passed MEMTEST (so did the first set...) and its ran a couple days now and some parity checks without crashing.  Let's hope that's all there is to it, thanks to everyone for helping me fumble through this.  Time will tell if it was just RAM or RAM + something else like the PSU but right now were stable.

Unfortunately, the Memory testing program only can identify bad RAM if it fails during the test.  A few years ago, most Gurus recommended a minimum of 24 hours with no failures.  That recommendation seems to have disappeared.  With the amount of RAM in a lot of servers these days, perhaps even 24 hours is too short. 

 

However, One Guru often recommends removing most of the RAM and testing with that configuration.

Edited December 19, 20241 yr by Frank1940