Two ideas ...

 

1 - Have you tested memory extensively. I was playing with a new server and experimented with overclocking out of curiosity more than anything. I tweaked the memory speed and was able to run stress tests and Windows at least a day and all appeared fine. I booted unRaid and within a few minutes I got a spontaneous reboot with the machine check flashing on the screen for a second that I happened to see. Removing the memory overclock fixed it. I doubt you are overclocking memory in a prod server, but could be your memory has fallen short in some way of spec and causing similar problem. You might also try removing some sticks and seeing if stability returns. Or running at a slower than stock setting.

 

2 - I had some spontaneous reboots once unRAID was all setup that I could not trace. Happened twice in 3 days. So turned on FCP troubleshooting and also addressed the FCP warnings/errors. Most notably the ones about making UD mappings "RW slave". I had several mappings that were just plan "Read Write". I have been up for over a week with no problem since making those changes. Maybe a coincidence, but I'd check it.

 

Good luck!

7 hours ago, FreeMan said:

Before I getting a 'loaner', are the PSU testers actually good for detecting these flaky types of issues?

No - a normal PSU tester just validates the voltages. You don't get any loading of the PSU so you never runt it hot.

8 hours ago, FreeMan said:

@pwm - I'm sure a parity check doesn't max out the CPU like mprime would, but I've made it through quite a number of parity checks over the last several weeks involving spinning all the drives. If memory serves, I've had at most one reboot during a parity check, possibly none. Would a parity check be reasonably comparable?

 

The parity check itself doesn't normally load the CPU very much - but the unRAID system regularly does other things too and the actual CPU load is bursty on top of the constant power consumption by the disks during the parity check.

 

The intention with loading both CPU and disks at the same time is to push the PSU by keeping a constantly high load on the PSU. First off - if the PSU gives unstable voltages then you have a much higher probability of hanging the machine when you push the PSU. More so if you push it for a longer time, making the PSU hotter.

 

But load tests involving read operations on the disks are best done with the array stopped so any system crashes doesn't require a parity scan.

 

5 hours ago, Frank1940 said:

I would imagine that the best ones will  cost more than a replacement PS.

I have a huge PSU tester at home - designed to be able to handle over 800W of load to be able to stress-test PSU with known loads and measure efficiency and difference between true and apparent power. But it probably weights 10 kg compared to the 50 gram testers that just checks the voltages.

Well, new power supply on loan from the local Fry's. If it stays up for 20 days, I'll order a replacement PSU and return the loaner.

 

If not, I'll get some memory on loan from them.

On 3/1/2018 at 11:32 PM, SSD said:

Two ideas ...

 

1 - Have you tested memory extensively. I was playing with a new server and experimented with overclocking out of curiosity more than anything. I tweaked the memory speed and was able to run stress tests and Windows at least a day and all appeared fine. I booted unRaid and within a few minutes I got a spontaneous reboot with the machine check flashing on the screen for a second that I happened to see. Removing the memory overclock fixed it. I doubt you are overclocking memory in a prod server, but could be your memory has fallen short in some way of spec and causing similar problem. You might also try removing some sticks and seeing if stability returns. Or running at a slower than stock setting.

 

Hey, thanks for the input... No, I'm not overclocking anything. It could well be memory, but I'm testing out the PSU theory first.

 

Quote

2 - I had some spontaneous reboots once unRAID was all setup that I could not trace. Happened twice in 3 days. So turned on FCP troubleshooting and also addressed the FCP warnings/errors. Most notably the ones about making UD mappings "RW slave". I had several mappings that were just plan "Read Write". I have been up for over a week with no problem since making those changes. Maybe a coincidence, but I'd check it.

 

Good luck!

 

So far, other than the MCEs, nobody's indicated there were any issues that they saw in the logs, including having FCP in troubleshooting mode, so I'm not sure what's going on. It looks like it's just one of those fun "throw money at it until the problem goes away" kind of situations.

 

Good for an upgrade or two, though, right?!?

On 3/1/2018 at 7:40 PM, Frank1940 said:

You can read the reviews.  But I suspect that the best use for these 'testers' is to make sure that the various voltages are close the the spec limits and that they power on and off.

That's all I'll use a PSU test for, to check the voltage and make sure everything reports with-in 10% of specification. Unfortunately the consumer units don't really have a way of testing current and whether it's able to push it's max ampre. Still it's better than nothing.  Similar to the testers I use at work.

And, despite the new PSU, it rebooted again. Here are the logs from just prior to the reboot.

 

I'm going to try putting some old memory in the machine. I'd had 2 4GB sticks in it and upgraded to 2 8GB sticks. I'll put the 4GBs back in and see if at least the reboots go away, though I may start running into OOM issues again.

 

If anyone (@limetech, hint, hint ) sees anything in the logs that causes something to look suspicious, please let me know.

FCPsyslog_tail.txt

nas-diagnostics-20180311-1448.zip

Possibly a coincidence, but I was also intermittently having some weird reboots under 6.5  I removed the file integrity plugin and it went away.  Beyond that, nothing particularly pops out in the logs

37 minutes ago, Squid said:

Possibly a coincidence, but I was also intermittently having some weird reboots under 6.5  I removed the file integrity plugin and it went away.  Beyond that, nothing particularly pops out in the logs

 

I have unRAID 6.5 and file integrity happily running together, have not encountered any specific issues when updating to 6.5 (all RCs included)

Just now, bonienl said:

 

I have unRAID 6.5 and file integrity happily running together, have not encountered any specific issues when updating to 6.5 (all RCs included)

Not blaming at all    May have been a coincidence with a previous RC.  

I would love to "blame" FI, but I was getting the reboots under 6.4/6.4.1 without it installed, so that's not the likely candidate.

 

With the MCEs in the log, it does sound like it's hardware related, the painful part is figuring out which piece of hardware. I'm now running with my older set of 2x4GB sticks to see if it's a memory issue (signature updated to reflect the current memory installed). If the reboots stop, I guess that's as good as an excuse as any to upgrade to 2x16 or 4x8GB sticks!

 

If it continues to reboot, I'll pull the trigger on the HBA I've been eyeing and a couple of SAS breakout cables and at least get rid of the old patchwork of controllers I'm running. Couldn't hurt, and it should give some faster disk speeds.

Sadly, after less that 48 hours on the original PSU and different RAM, I've had another spontaneous reboot.

 

I saw the notice this morning that 6.5 is available. Since these problems started with the installation of 6.4, is it possible (likely even) that they'll go away with 6.5?

 

My only other thought at the moment would be to use the other pair of memory slots, since I put the old, know to be working fine last time I used them, DIMMs into the same slots on the mobo.

 

Sent from Tapatalk

 

 

 

 

3 minutes ago, FreeMan said:

Sadly, after less that 48 hours on the original PSU and different RAM, I've had another spontaneous reboot.

 

I saw the notice this morning that 6.5 is available. Since these problems started with the installation of 6.4, is it possible (likely even) that they'll go away with 6.5?

 

My only other thought at the moment would be to use the other pair of memory slots, since I put the old, know to be working fine last time I used them, DIMMs into the same slots on the mobo.

 

Sent from Tapatalk

 

 

 

 

You should not be getting spontaneous reboot with any release unless there is an underlying hardware issue.

 

You can try updating to 6.5 but it is unlikely it will fix the issue if it is hardware related.   At least you would be on the current release which might help from a support perspective.

You should not be getting spontaneous reboot with any release unless there is an underlying hardware issue.
 
You can try updating to 6.5 but it is unlikely it will fix the issue if it is hardware related.   At least you would be on the current release which might help from a support perspective.

I agree wholeheartedly, however, up thread, mention was made that the issues may have started because of OS movement within memory when installing 6.4. I was thinking that there might be additional movement in 6.5 that might mitigate the issue.

Of course, if that's the case, that means there its still some sort of hardware (likely memory?) issue lurking and all I would have done is pushed it somewhere else...

Sent from Tapatalk

This is a first, two reboots in under 24 hours.

I'll getting a bit antsy about data integrity... I haven't had any parity issues, and I don't want to. Critical files are backed up offsite (CrashPlan ftw), but that's a pain to about if possible.

Anyone have any suggestions on what else to test? I can run another memtest overnight, but as@limetech indicated, that's not a 100% guarantee. I guess I also need to dig into the stress testing someone mentioned earlier.

Sent from Tapatalk

I've swapped the RAM into the other bank of slots, we'll see what happens.

Just rebooted again.

 

Diagnostics and syslog_tail.txt attached.

 

I'm kicking off mprime -t to run a stress test while it does a parity check. We'll see what happens.

 

The mersenne.org website indicates that this test will "draw a lot of power". Based on what I'm seeing reported on my dashboard, that doesn't seem to be the case:

164 watts doesn't seem to be that much of a stress on my TX650 PSU, am I misinterpreting this?

 

FCPsyslog_tail.txt

nas-diagnostics-20180317-2329.zip

Edited March 18, 20188 yr by FreeMan

6 hours ago, FreeMan said:

164 watts doesn't seem to be that much of a stress on my TX650 PSU, am I misinterpreting this?

Max power used by the stress test is mostly related to how much power your CPU uses at max load, together with the other components like board, ram, disks, etc idling, nothing to do with max PSU power, power used will likely be more at power up due to all disks spinning up.

2 hours ago, johnnie.black said:

Max power used by the stress test is mostly related to how much power your CPU uses at max load, together with the other components like board, ram, disks, etc idling, nothing to do with max PSU power, power used will likely be more at power up due to all disks spinning up.

Thanks. So far (about 9 hours in) the only issue I'm seeing is OOM errors, which I expected since I'm running with 1/2 the RAM I had before. I've shut down a docker & turned off cachedirs for now in an attempt to preempt anything critical getting shut down. I guess at this point, I could probably shut down all dockers, but I figure if they're running, it's just that much more stress on the system (even if it's not much more).

 

Parity check has just passed 25% completion, so the 1TB drives may start spinning down soon, though I don't get much spin down (that I notice). Maybe I'll kick off a cp * > /dev/null on everything, later this afternoon. (About to leave for most of the day.)

 

FCP is in troubleshooting mode, in case anything interesting is logged there.

24 minutes ago, FreeMan said:

the only issue I'm seeing is OOM errors,

 

I would suggest doing this and see what happens.  It has resolved a lot of problems over the past few months...

 

    https://lime-technology.com/forums/topic/58855-regular-out-of-memory-problems/#comment-577424

 

5 hours ago, Frank1940 said:

 

I would suggest doing this and see what happens.  It has resolved a lot of problems over the past few months...

 

    https://lime-technology.com/forums/topic/58855-regular-out-of-memory-problems/#comment-577424

 

 

Thanks, but I know exactly why I'm getting OOMs - I'm running on 8GB of RAM now, not my normal 16GB. Just trying to trouble shoot to figure out what's going on. Since the machine rebooted with the old 4GB RAM sticks, and with them in the other pair of slots on the mobo, I'm presuming the issue is not with the RAM, so I'll be putting the 16GB back in after a day or two of stress testing. Might just add the 16GB instead of replacing and run on 24GB, just for fun...

mprime ran for about 20 hours with no errors. It was finally killed when the server ran out of memory:

Quote

 

Mar 18 20:00:24 NAS kernel: Out of memory: Kill process 10522 (mprime) score 226 or sacrifice child
Mar 18 20:00:24 NAS kernel: Killed process 10522 (mprime) total-vm:1896188kB, anon-rss:1707480kB, file-rss:0kB, shmem-rss:3648kB
Mar 18 20:00:24 NAS kernel: oom_reaper: reaped process 10522 (mprime), now anon-rss:0kB, file-rss:0kB, shmem-rss:0kB

 

 

The CPU was pegged at 100% (as would be expected). CPU temp was pretty stable at 40°C and mainboard temp stable at about 36°C.

 

Anyone have any other ideas or suggestions? Should I update to 6.5 to see if it moves something somewhere else in memory that won't cause it to reboot?

4 hours ago, FreeMan said:

mprime ran for about 20 hours with no errors. It was finally killed when the server ran out of memory:

The CPU was pegged at 100% (as would be expected). CPU temp was pretty stable at 40°C and mainboard temp stable at about 36°C.

 

Anyone have any other ideas or suggestions? Should I update to 6.5 to see if it moves something somewhere else in memory that won't cause it to reboot?

You should have been running something that stressed the disks (and preferably - if possibly - any GPU) at the same time, since it isn't  unlikely that your PSU is responsible for your problems.

 

mprime is great at verifying the cooling of the processor. But when only loading the processor you will still not load the PSU.

You should have been running something that stressed the disks (and preferably - if possibly - any GPU) at the same time, since it isn't  unlikely that your PSU is responsible for your problems.
 
mprime is great at verifying the cooling of the processor. But when only loading the processor you will still not load the PSU.

I'm not sure how much stress a parity check puts on the disks, but that was running the whole time as well.

I've got an AMD A-series CPU with its built in GPU as my only video in the machine. I use it for a monitor to watch reboots and nothing else.

If you have any GPU tester suggestions, I'm happy to try it again, however, it rebooted itself after just a couple of days with a totally doesn't, brand new PSU in it.

Sent from Tapatalk