Is it worth trying an upgrade to 6.5.0?

Edited March 19, 20188 yr by PeteB

Yes, it's always worth trying with a later version.

 

Over the years, there have been millions of hw incompatibilities that has been fixed by updated Linux drivers or changed ways how Linux performs the hardware detection etc. This isn't just for unRAID but happens for Debian, Ubuntu, Red Hat, ...

 

It's enough to have a single chip on the motherboard with a bit shaky Linux support to make life a misery until some Linux developer happens to buy hardware with the same chip and then invests time to try to figure out what goes wrong.

 

So in case of stability issues, the full set of tools available has to be tried while trying to bracket what might be wrong

- try newer Linux kernels

- try memtest tools

- try other PSU

- try different burn-in tools

- try different clock settings for memory etc.

- ...

Yes, it's always worth trying with a later version.

Thanks for the encouragement, pwm, I'm upgrade this evening when I get home.

I've been putting it off in the name of "one change at a time", I guess now is the time.

Sent from Tapatalk

10 hours ago, FreeMan said:

 

10 hours ago, pwm said:

Yes, it's always worth trying with a later version.

 

Thanks for the encouragement, pwm, I'm upgrade this evening when I get home.

I've been putting it off in the name of "one change at a time", I guess now is the time.

 

There's also been a "roll-back to previous version" feature added to the webui, either with 6.5, or 6.4.1 (can't remember), so if you need to you can always hop backwards.

Edited March 19, 20188 yr by Jcloud

1 hour ago, Jcloud said:

There's also been a "roll-back to previous version" feature added to the webui, either with 6.5, or 6.4.1 (can't remember), so if you need to you can always hop backwards.

There have been a huge number of useful updates in the recent months.

OS updated to 6.5.0. We'll see how this goes.

OS updated to 6.5.0. We'll see how this goes.

Awesome. Hope you have some stability return.

Sent from my SM-N920I using Tapatalk

How did it go? Do you still have the problems?

Sadly, it rebooted again late last night.

I didn't think to put FCP on troubleshooting mode, but will do so now to see if it picks up anything different.

FCP is still reporting the MCEs, so it does seem that there is a hardware issue somewhere, I just haven't been able to isolate it yet.

Sent from Tapatalk

Yet another reboot. I had FCP running, but all my tools and tricks aren't letting me get remote access right now to attach the logs. I'll do so as soon as I get home this evening.

Sent from Tapatalk

Here is the syslog_tail and diagnostics from prior to the crash.

 

Just for giggles, I'm going to add the other memory back into the machine. Who knows, maybe it'll run better with 24GB than either 8 or 16. Couldn't be any less stable than it is right now...

 

Unfortunately, I'm still seeing this:

Mar 20 22:37:15 NAS root: mcelog: ERROR: AMD Processor family 21: mcelog does not support this processor.  Please use the edac_mce_amd module instead.

@limetech, I was under the impression that edac_mce_amd had been (or was supposed to have been) baked in at some point in the not to distant past.

 

FCPsyslog_tail.txt

nas-diagnostics-20180322-1018.zip

 

Update @ 20:10 (GMT+5): put both sets of memory in, currently running on 24GB.

Edited March 23, 20188 yr by FreeMan

Sadly, another reboot overnight.

Sent from Tapatalk

And another reboot an hour ago.

Ok, I've done some Googling, and this is what I've found:

 

It seems that edac_mce_amd support has been baked into unRAID:  "lsmod |grep -i edac" indicates the edac_mce_amd Module has been loaded (as best I can tell):

Quote

root@NAS:/mnt/user0# lsmod |grep -i edac
edac_mce_amd           28672  0

 

"dmesg | grep mce" produces this output:

Quote

[    0.018890] mce: CPU supports 7 MCE banks
[    0.034063] mce: [Hardware Error]: Machine check events logged
[    0.034170] mce: [Hardware Error]: CPU 0: Machine Check: 0 Bank 4: f2000010000b0c0f
[    0.034286] mce: [Hardware Error]: TSC 0
[    0.034403] mce: [Hardware Error]: PROCESSOR 2:610f01 TIME 1522279719 SOCKET 0 APIC 0 microcode 6001119

 

Turning to this post as a likely looking candidate to interpret what those errors mean, there's a quote in there to the effect of "any time I see Bank x, I assume it's memory". The next post down, he mentions using dmidecode to identify memory banks. I've done so and attached the output. Since the memory devices are listed as Dimm0 - Dimm3, I would assume that Dimm3 refers to Bank 4 in the mce error above. The output for Dimm3 is:

Quote

Handle 0x002B, DMI type 17, 34 bytes
Memory Device
    Array Handle: 0x0023
    Error Information Handle: Not Provided
    Total Width: 64 bits
    Data Width: 64 bits
    Size: 8192 MB
    Form Factor: DIMM
    Set: None
    Locator: Node0_Dimm3
    Bank Locator: Node0_Bank0
    Type: DDR3
    Type Detail: Synchronous Unbuffered (Unregistered)
    Speed: 1333 MHz
    Manufacturer: Undefined         
    Serial Number: 00000000    
    Asset Tag: Dimm3_AssetTag
    Part Number: F3-2133C9-8GX
    Rank: 2
    Configured Clock Speed: 1333 MHz

 

I don't recall exactly which DIMM was in which physical slot on the mobo to begin with, but I have replaced the pair of 8GB sticks with a pair of 4GB sticks in the same slots. Then I moved the 4GB sticks to the other pair of slots, then I added the 8GB sticks back in to the original pair of slots (running 24GB at the moment). There's no telling if the 8GB sticks are back in their original slots or if I swapped them when I reinstalled them.

 

Since there have been at least 2 different sticks (1 @ 4GB and 1 @ 8GB) in whichever slot dimm3 represents, and, the machine was running with no stick in there at all at some point, is it logical to conclude that the error is in the DIMM slot itself, not in any of the memory I've been putting in the slot? If that's the case, does it mean that it's time to chuck this mobo?

 

Understanding the caveats about the abilities of memtest86 to fully and completely test memory, would it make sense to try testing each of the 4 DIMMs in each of the 4 slots to see if any combination produces any errors? (Keep in mind, of course, that I do actually need to be able to use the machine over the next week or two while all this testing happens...)

 

I understand that this is probably above and beyond the call of duty for software support now, if there is anyone who has some uber geek level of linux knowledge that they'd be willing to share, I'd be more than grateful.

dmidecode.txt

 

 

EDIT: I've just submitted a ticket at Gigabyte's website to see if they can interpret this for me. I'm still open for community input.

Edited March 29, 20188 yr by FreeMan

If you think you might have narrowed the problem down to a physical DIMM socket then try running with it empty for a while. Try an 8 GB DIMM in each channel, avoiding the suspect socket. Though you've probably tried that before.

 

There isn't something inside the problem socket that shouldn't be there, is there? A foreign object, like a sliver of paper or plastic? It would be worth using a can of compressed air on it or a small paint brush.

If you think you might have narrowed the problem down to a physical DIMM socket then try running with it empty for a while. Try an 8 GB DIMM in each channel, avoiding the suspect socket. Though you've probably tried that before.
 
There isn't something inside the problem socket that shouldn't be there, is there? A foreign object, like a sliver of paper or plastic? It would be worth using a can of compressed air on it or a small paint brush.

I did do some blowing out as I was recabling when I swapped the PSU for testing. Not to say that something didn't get in there before memory was put in. However, the errors started before anything was touched. That doesn't preclude some dusty or something having settled on an unused DIMM socket.

I'll give it a good dusting out again.

Sent from Tapatalk

My question to Gigabyte tech support:

 

Quote

I've been getting mce errors causing my server built on this motherboard to reboot.

The mce errors from dmesg are:
[ 0.018890] mce: CPU supports 7 MCE banks
[ 0.034063] mce: [Hardware Error]: Machine check events logged
[ 0.034170] mce: [Hardware Error]: CPU 0: Machine Check: 0 Bank 4: f2000010000b0c0f
[ 0.034286] mce: [Hardware Error]: TSC 0
[ 0.034403] mce: [Hardware Error]: PROCESSOR 2:610f01 TIME 1522279719 SOCKET 0 APIC 0 microcode 6001119

It appears that it might be related to the memory, but I'm not certain. I've used 2 different pairs of DIMMs and had the same issue with both pairs. I've had them in different pairs of slots on the board and continued to have the same mce's and reboots.

Any hints as to what the problem actually is would be greatly appreciated.

 

And their answer:

Quote

Unfortunately we do not validate Linux on our boards as it is open source.
When exactly are you getting these error and did you tried a Windows based OS as well to testing the results?

 

Sigh.

 

Is there anything in Windows that would log & report on these types of errors?

On 3/28/2018 at 10:39 PM, John_M said:

If you think you might have narrowed the problem down to a physical DIMM socket then try running with it empty for a while.

 

Realized I didn't specifically address this point.

 

The server ran for a number of years with just 2 of the 4 slots filled. Then, when the 6.4 update came out, it started the rebooting. After pulling the 8GB sticks and replacing them with 4GB sticks, I had another reboot or two. I then moved the 4GB to the other pair of slots and continued to get reboots. So yes, I've tried running with both pairs of slots empty (well, not at the same time...) and have had reboots in either case.

Just installed an LSI 9201-8i. Here's to hoping the issue was with one of the patchwork of low-end controllers I had installed.

Nope, no luck.

Kicked off a parity check earlier this morning for a speed test and it rebooted about an hour ago.

The good news is that parity checks do seem to be running faster so far.

Sent from Tapatalk

After repeatedly being told that they don't support linux and asking if I've changed the CPU, it looks like I'm not going to get much help from Gigabyte support. I've also noticed that when Emby is transcoding video to stream that my CPU is pegged at 100%. So, I guess that means it's time to upgrade the MoBo, CPU & RAM.

 

If anyone has any other ideas prior to spending cash, I'm all ears!

 

Thanks to those who have contributed suggestions.

 

Contacting AMD support as a last ditch effort before waving the white flag.

Edited April 5, 20188 yr by FreeMan

Well, it seems AMD is willing to take responsibility!

 

I got a very prompt response back with a link to their warranty returns page. The only problem is that I've no idea how long my server will be out of commission without a CPU.

 

Maybe I can upgrade anyway then warranty the CPU then have guts for a 2nd test server.

 

Thank you all for your input, suggestions and help!