Server hard crashing every day.

[ZanthumGum]

For the past month or so my server has not been online for more than a day or so. I know it crashes nearly everyday about when my parity check auto starts since some of the logs I could recover all stopped at about 00:30, when my parity starts. I always notice after work when my backup client (a windows machine under my desk running Duplicati) fails to run my preconfigured backup routines. I'll attach my most recent diagnostics from immediately after my last reboot, can anyone see what might be causing it? I've already removed several plugins that I'm not using, and docker containers that I stopped using too. 

 

The reason I'm not using the Docker version of Duplicati is bc it pegs my server CPU at 100% and also causes a hard crash, but that ended up being a HDD IO error at the target location, but now I have a discrete client handling it anyway.

 

Hardware:

i5-6400

Asus H110M-C Rev X.0x BIOS May 2016

16GB DDR4 Crucial Budget 2133 MHz

 

skysword-diagnostics-20220825-2155.zip

[trurl]

4 minutes ago, ZanthumGum said:

about 00:30, when my parity starts

This statement makes me wonder how often you run parity check. Monthly is a commonly used schedule. Are you running it more often? Unraid parity is maintained realtime, parity checks aren't required to keep parity in sync, they are just a check to confirm that parity is in sync.

 

Have you done memtest recently?

 

Setup syslog server

[ZanthumGum]

1 hour ago, trurl said:

Unraid parity is maintained realtime, parity checks aren't required to keep parity in sync, they are just a check to confirm that parity is in sync.

 

Have you done memtest?

I'm new-ish to unRAID, so I thought parity check is what protected the data, that makes a lot more sense. This just means I've been putting a lot more stress on my parity drives than needed so thanks for the info. I'll run memtest tomorrow. Do you think it's a memory issue?

[itimpi]

The fact that the server has been crashing when parity checks start makes me wonder if you have a Power issue as that would be a time of maximum stress on the power supply.   What PSU are you using?   Are you using splitters on the PSU lines?

[trurl]

5 hours ago, ZanthumGum said:

putting a lot more stress on my parity drives than needed

ALL drives are involved in parity checks

[ZanthumGum]

7 hours ago, itimpi said:

The fact that the server has been crashing when parity checks start makes me wonder if you have a Power issue as that would be a time of maximum stress on the power supply.   What PSU are you using?   Are you using splitters on the PSU lines?

Now that you mention it, I am running a slightly older 500W Gold Rosewill PSU. it only has 4 sata power on a single power rail, that I'm breaking up into 8 for my drives.

[itimpi]

45 minutes ago, ZanthumGum said:

Now that you mention it, I am running a slightly older 500W Gold Rosewill PSU. it only has 4 sata power on a single power rail, that I'm breaking up into 8 for my drives.

How are you splitting it?   You can split a single SATA connection into 2 but more than that is likely to cause problems. 

 

If you have (or can borrow) a better power supply for testing purposes to see if it help it will be worth trying.

[ZanthumGum]

On 8/26/2022 at 8:53 AM, itimpi said:

How are you splitting it?   You can split a single SATA connection into 2 but more than that is likely to cause problems. 

 

If you have (or can borrow) a better power supply for testing purposes to see if it help it will be worth trying.

It's split 1 -> 2, 1 -> 3, 1 -> 3 (one sata connector broke). I think you're right, that it's a power issue; I'll look into getting a higher wattage psu with more sata connections. Ever since I set my parity check to monthly, my server hasn't crashed. I also just moved last weej to an older apartment and my server is temporarily plugged into the wall instead of my UPS. There's a solid chance that the power out of wall is dirty bc of older wiring and previously my server was just on the edge of stability on good power out of my UPS; but now with dirty power from the wall it's now no longer stable at full load/during parity.

Edited August 28, 2022 by ZanthumGum

[trurl]

You can also use molex->sata connectors to get drives connected to the peripheral cable of your PSU. Really better to split molex than to split SATA.

[ZanthumGum]

31 minutes ago, trurl said:

You can also use molex->sata connectors to get drives connected to the peripheral cable of your PSU. Really better to split molex than to split SATA.

I was reading that bad molex cables could short out and cause issues. Is it just about getting quality adapters? 

[JonathanM]

17 hours ago, ZanthumGum said:

I was reading that bad molex cables could short out and cause issues. Is it just about getting quality adapters? 

The SATA part of the adapters, not the 4 pin ends, are the real issue there, lots of high current with wires very close together. Molding errors or just tolerances over time with the manufacturing equipment can cause shorting issues leading to excessive heat and possible fire.

 

Less risk with a 4 pin to sata than there is with a SATA - SATA splitter, SATA connectors are just poor designs all around.

 

But yes, getting connectors from a manufacturer with good QC is key, being able to see each wire at the SATA end is a bonus. IDC vs molded.

Molded on both ends of a SATA splitter is just asking for trouble.

I'm not saying all molded cables are bad, it's just much easier to ignore or hide bad manufacturing.

[ZanthumGum]

1 hour ago, JonathanM said:

But yes, getting connectors from a manufacturer with good QC is key, being able to see each wire at the SATA end is a bonus. IDC vs molded.

 

Molded on both ends of a SATA splitter is just asking for trouble.

I was aware of the potential fire issues with cheap molded power connectors so I made sure to buy high quality connectors that were post-mold clamp style so that shouldn't be an issue.