I keep running it to this weird ram issue, where my log kinda get over flooded with RAM error messages, and don't know why. I have tried switching some of the ram around, but non of it seemed to have worked. Have even tried leaving the system running without any dockers or VM's running. And it still comes up after some time. Sometimes it takes as little as 2 hours to show up. Other times it can take 2 - 4 days. But as soon as it does. I'm hit with a massiv VM performance performance hit. 

 

I have a little screenshot and the diagnostic files here:

 

tower-diagnostics-20181119-1555.zip

Well, the clue is in the error message. You have a faulty RAM module, or a bad DIMM socket. If your BIOS has an event log it will help you locate the fault.

Be sure to check your MB's manufacturer's website to see if they recommend specific RAM modules for use with your MB.  Some MB's are very picky.  Particularly, after you have more than two RAM modules in use.  You may have to stick to a single manufacturer and model type and not mix with cross brands/types.  Make sure you are not using any memory overclocking features. 

Will try and look at the event log in my BIOS later today. Really don't wanna send back my motherboard, as it order it on newegg.com and live in the EU. And it's a long times since i bought the RAM on ebay. And the errors came up like 1,5 months ago I think. I should have all the same ram from the same manufactor, bought it all at once, and they have same stickers, and heatsinks on. And within the listed specs from my MB manufactor. And i'm not totally sure about the memory overclocking tho. But I don't think I have any of that activated. Could I post the BIOS event log, and have someone look a bit at it with me or?

 

Best regards,

Mads

5 hours ago, Brydezen said:

Could I post the BIOS event log

Yes, do. I'm curious to see what it says.

Sorry for the slow response. My server went offline while I'm away from home. Not sure if the internet connection dropped, or it is really offline tbh. But will post a BIOS event log when I get home. Just need to find out how to dump it tho. 

 

EDIT: Can see their is also a new BIOS update from 26/6/18 - for a spectre & meltdown patches. And improve system performance. So will be flashing that too when I get home.

Edited November 22, 20187 yr by Brydezen

@John_M I have some event log from between the 22 and 24. But I had the same memory error last night on the 23 tho

i will post a screenshot of my memory section in the bios in a sec

 

- I have switch the ram around, and can see that the issue now is on socket 1 instead of socket 0. So I think it's some of the ram that are failing maybe. Just really hard for me to identifying what stick(s) it is....

Edited November 24, 20187 yr by Brydezen

It looks like two faulty DIMMs. The Unraid log suggests Channel 0 and Channel 2 of CPU Socket 1 in your most recent post but that might not coincide exactly with your motherboard's terminology. Does the BIOS itself have an event log? If it does it should make it easier for you to identify them

I think it is the RAM then. Because I switched all of them around like 4 days ago. And the DIMMs location reporter by the log seem to have moved around indeed. But the documentation for the motherboard is not really clear on what DIMM slot is what channel. So maybe I need to do some trail and error and look in the reported installed memory to locate it..

 

But just flashed the BIOS to the newest version today. They released a spectre and meltdown patch. So will leave it running, until I get the error again. And will post it. But before I flashed it, it didn’t even show anything about memory errors in the event log :-(

Hi @John_M I got the error once after the BIOS upgrade, but then it showed socket: 0 - and their is literally no sign of anything in the eventlog 😮 I have no clue what is going on.. But have noticed, that on some rare occasions my motherboard throws a error 60. The BIOS error codes says that it's a memory error, and a recommended option is to reinstall the cpus and clear the CMOS. So maybe I will do that in the weekend. Take out the CPU's and swap them over in the opposite and maybe move tha ram around again. :-S 

The BIOS and the OS won't necessarily agree on which socket is which. You have two bad DIMMs so all you have to do is find them. That shouldn't be too difficult with the help of MemTest.

Sorry for asking a noobish question maybe. But is the memtest build into unraid or do I need to run it from a usb stick with some software on it? Saw some other post where the recommended to run it for about 24 hours.

 

Best regards,

Mads Bryde

It is a Boot Option on the Unraid boot menu.  As I understand it, it will not work with ECC memory.  And, it will not boot if your are using a UEFI boot.  You can download a newer version from here that will boot under UEFI:

 

       https://www.memtest86.com/download.htm

 

Try the boot option first.  in either case, you will need a monitor and keyboard attached to your server.  

I am booting unraid from UEFI usb install. And do have ECC memory in my system

 

EDIT: Their website state that they do support UEFI booting and ECC memory from V5 - so will test that this weekend for sure

Edited November 27, 20187 yr by Brydezen

Hello @Frank1940 and @John_M I'm running memtest86, from the unraid boot menu. And I just hit the first pass, it did 13 tests to complete one pass. But I have some minor questions; in the Testing line (just above the Pattern) it says 2048M of 64G. I did not increase when it did the first pass. Do I need to change something or? Seems quite odd to "only" test 2048 of 64G

 

 

It has been so long since I ran this test (and I can't remember if I sat there watching its progress), I don't remember exactly what is happening.  But it seems to me, that it has to test the block of memory where the code resides also.  So it may be doing that in a separate pass before it does the rest of RAM.  By the way, it is recommended that you run it for at least 24 hours.  Hopefully, it will complete more than one pass in that time period.  (But the problem with the 24 hours recommendation was that was back when 8GB was the high end of the RAM installed in most Unraid servers.)

The blue background looks like legacy mode to me. I recommend the free downloadable version 7.5. Make a separate USB flash (don't use you Unraid one!) and boot it in UEFI mode. The default settings are fine - it will do four passes of 13 tests. I ran that on a new build only a couple of weeks ago and it found a bad DDR4 DIMM.

I have tried downloading the memtest86 from the website. But when I boot the usb i get to use a even older version (4,3) I don’t get a option to use 7.5 for some reason. Maybe the program think my MB is just to old (AsRock EP2C602) 😕

It has been running for 17 hours and 33 minutes now. The first pass took around 4 - 5 hours. And it says that it’s running at 666 MHz?! It’s only 68% complete on the second pass 

8 hours ago, Brydezen said:

I have tried downloading the memtest86 from the website. But when I boot the usb i get to use a even older version (4,3) I don’t get a option to use 7.5 for some reason.

You get the old version when you legacy boot. You have to UEFI boot to get version 7.5. It has a different look about it. The old version uses VGA resolution; the new version uses a higher resolution and has a splash screen.

 

If your RAM is DDR3-1333 then 666 MHz is correct. You get two data transfers per clock pulse.

I had to change a few BIOS settings. So I boot UEFI first and fallback to legacy. I’m running MemTest87 V7.5 on almost the 6th hour by now. Been through “almost” 3 passes now. And both of those said their was a ECC error on Channel 7. But I did also change a setting in the BIOS from auto frequenzy on the ram to lock in 1333 MHz. But I’m pretty sure that would not matter much. As I kept getting the errors in the unraid logs too. - so think I have to message the eBay seller, and ask for a replacement. The are only little under a year old. 😕

I'm not sure why you're carrying on testing for hours after a fault has been revealed. You know you have faulty RAM. Unraid has already told you so. You just need to find out which DIMMs are the bad ones. Does "Channel 7" give you any clue? If so, either remove that DIMM or swap it with another (having labelled them first) and test again. If not, then move them around (again, having labelled them) and test again until you can identify the faulty ones. Your diagnostics suggested two bad DIMMs. Usually when you RMA RAM the vendor needs all the modules in the set to be returned together. I suggest you contact the vendor.

Interesting, I've heard before that the new passmark memtest can detect ECC errors, though apparently only on some boards and I though it was only the paid version, but it's the first time I've seen it work.

Yeah it is because of the diagnostics, that I keep on running the test. As only one back module or DIMM has shown up by now. And hoping that it will find the other one before it finishes the test. Don't know if I can do any RMA on a ebay seller, but will contact him/her and hear about what we can work out. But I don't really know what Channel 7 is tbh. My motherboard manual is pretty lose when it comes to the RAM DIMMs. 

Well glad I could help some johnnie

It's pretty hard for me to find the right memory stick. I have no clue how to identity it. The manual only list the slots as DDR3_A1,B1 (...) can I assume A1 is channel 0 and so on? So G1 is channel 7?