I am new to unRAID. I have an old server that I recently refreshed the drives in and decided that would be a good time to try out unRAID. I have had continual lockups and sor far 1 kernel panic with unRAID. Is anyone else seeing stability issues with unRAID 6.1.3 running several dockers?

 

When the lockup occurs, my CPU fan ramps up to high, there is no console output and I can't ping the server anymore, seems to be hard locked. I have to turn it off and back on to recover.

 

The Kernel panic referenced an interrupt at the end, but I couldn't find a way to capture more than what was on the screen, didn't see it logged to flash or anything after I rebooted. A screenshot of the console output is attached here.

 

The server is old, but I have been running ESXi on it with perfect stability for years before now. I also ran SmartOS on it for a short time and it was perfectly stable also. I am open to the fact that this could be my old hardware, but considering how solid it was on the other platforms, I am not sold on that yet. The only thing that changed between OS' was the data HDDs. Cache SSDs and boot flash are the same.

 

I want to run unRAID because of some of its home focused features, but so far it just hasn't been stable for me. Curious what others are seeing and if anyone has seen similar behavior?

 

Thanks in advance for any help,

-Landon

First thing you should do is restart and run memtest for awhile (ideally 24 hours, but at least a couple of passes) and see if any errors happen.

 

Secondly, you should post the complete diagnostics (Tools / Diagnostics)

Thanks, I will start that today and see where I end up. I will report back after a few passes have completed and let it run at least 12 hours.

Within 35 minutes it was throwing errors as seen in the attached. I assume that is bad. 

 

I re-seated all the memory an I am running it again. So far it is cleaned and has passed where it was before. I am hoping I just knocked one of the sticks loose or something when replacing the drives. I will let it get through at least a couple of passes then try unRAID again if it stays clean.

 

Thanks for the advice so far.

 

-Landon

Since re-seating the memory the tests ran clean for 2 and half passes, so I fired unRAID back up and will see if it survives the night while restoring files with Crashplan which seem to exercise the memory pretty heavily.

 

Thanks for the guidance. Fingers crossed.

 

-Landon

Well, it ran for hours and under some very heavy load, but it just froze up a few minutes ago. I guess I will run memtest again over night tonight and see if it discovers errors again. Not sure what else to do at this point. Sure is frustrating and I hate to throw money at hardware since it has been so stable historically.

Anytime you get Red lines in Memtest, SOMETHING is wrong.

 

How many modules do you have installed?    If the answer is 4, remove 2 of them and run with just 2 modules.    With unbuffered memory this will be FAR more reliable.

 

Since re-seating the memory and starting memtest last night, it has now run with 4 passes cleanly. I am going to let it run through the day today and see if I can get it to fail. I am running with 4 DIMMs. I can try 2, but might be running pretty high on utilization at that point. I suppose it would get me by until I upgrade though.

 

Thanks for the comments, if it produces errors today, I will just get new hardware, but so far it isn't erroring anymore.

There's a MAJOR difference in the address and data bus loading with 4 DIMMs installed rather than 2.  Anytime you're having issues, reducing the load to 2 DIMMs will make a BIG difference in reliability.

 

Depending on the type of memory you have, and what your system supports, you may be able to get just as much memory capacity by replacing the DIMMs with a pair of larger DIMMs.  [e.g. 2 2TB modules instead of 4 1TB modules]

 

Unfortunately, the platform is fairly old and I am already using the MAX DIMM size.

 

If memtest doesn't produce any errors through the day today, then I will pull a couple of DIMMs and see if unRAID is any more stable.

 

Thanks,

-Landon

If you remove a couple, be sure you pull one from each channel, so you'll still get dual channel operation.

Not a major thing, but may as well have the best performance.

 

Well, Memtest ran all night and through the day today and was clean with 8 passes.

 

I have done as you advised and pulled to memory sticks leaving DIMM0 for channels A and B present in the machine. unRAID is back up and running and we will see if it hangs on any longer this way. If it doesn't, I will swap the other sticks back in to the slots they were in before and pull these out and see how it goes again.

 

If this still produces issues, I suppose I will just pull the trigger on some new hardware, but I sure hope I don't see the same thing once I do that.

 

Thanks for the advice!

-Landon

If it's rock solid with just 2 modules, I'd just leave it like that.  No need to even bother with the other pair.

 

The issue with 4 modules is that you're putting 64 TTL loads on the address and data buses, which significantly degrades the signaling waveform.  With 2 modules, it's still degraded (with 32 loads), but FAR less than with 4.

 

That's why server systems use buffered RAM modules => they only produce ONE load/module instead of 16 !!

 

Anytime you get Red lines in Memtest, SOMETHING is wrong.

 

How many modules do you have installed?    If the answer is 4, remove 2 of them and run with just 2 modules.    With unbuffered memory this will be FAR more reliable.

 

Especially AMD systems have almost always issues with RAM slots fully populated.

 

This is an Intel system with high quality memory, but it has been running for a long time!

 

After pulling the first set the server hung within an hour. I pulled those sticks and put the others back in and also placed the into channel 1, rather than channel 0. The server has been up 12 hours and it is the first time it has lasted the night. I am cautiously optimistic. I will just keep letting my restores run, they have about another 1.5 days before they complete. Assuming it stays up and allows them to complete, I will then run a parity check and keep my fingers crossed.

 

Thanks for the pointers. I will report back in a day or two if it stays stable.

 

Thanks!

-Landon

Well, unfortunately, the server just hung again.

 

Not sure what to do at this point except replace the server. I just REALLY hope I don't dump that money and then have stability issues with unRAID on new gear. I don't imagine I will since no one else seems to be complaining. Just hard to get over the fact that the server was so stable until I introduced unRAID. In fairness though I realize hardware can fail/start to fail at any time and it could be a coincidence.

 

Here is what I am looking to build out for the new box if anyone cares to tell me I am making a bad selection with regards to unRAID support:

 

To be purchased:

iStarUSA 4u rack mount case - D-407P-DE6BL

Enermax 630W PSU - ERX630AWT

2x Enermax 80mm case fans - UC-8EB

AMS 4x2.5" backplane - DS-524SSBK (for cache SSDs)

ASRock LGA1150 MB - E3C224

Intel XEON E3-1241 v3 - BX80646E31241V3

Crucial 16GB DD3 ECC Unbuffered 1600Mhz memory (2x8GB kit) - CT2KIT102472BD160B

 

Already purchased and in current server, but new:

2x Western Digital RED 4TB HDDs - WD40EFRX

Western Digital RED 3TB HDDs - WD30EFRX

 

Being re-used from the old server and seem to be in good shape (no smart errors):

2x Mushkin Enhanced Chronos 60GB SSDs - MKNSSDCR60GB

 

Thanks,

-Landon

 

Looks good => a server grade motherboard with a Xeon and ECC should be VERY reliable

 

Just wanted to post back on this that I built the server as described and it has been running stable now for several days.

 

I appreciate the help in chasing down where the issue might have been. Excited to be on unRAID now!!

 

Thanks!