Parity checks pass then fail without any rhyme or reason, would like advice.

[hansolo77]

Hey guys!

 

I've been playing with Unraid over a year now.  I really like it, and don't ever want to use something else.    I'm having trouble though, and I can't figure out what's wrong.

 

I've made, or replied to, a post some time ago about my monthly parity checks.  At first, my parity checks were coming back perfect with 0 errors.  Then I started noticing 200 or 300 errors on a check.  Not that big a deal, I was almost positive I knew what had happened.  Not knowing what I was doing, I used the Krusader docker to forcibly move data around on my drives; moving data off of one, putting it on another, etc, without allowing the parity process to take place.  Lesson learned.  A few months go by and my parity checks come back clean again every time.  Then, out of nowhere, I started hitting 127k worth of errors a few months ago.  I've not changed any of my hardware.  Somebody somewhere (it was either in forums or in Discord, or reddit, etc), suggested this type of thing could be a symptom of bad RAM.  So I ran memtest for like 4 days (something like 12 passes) and it came back 100% clean with no problems.  Nothing seemed to be missing as far as my data, so I allowed the check to go while fixing the errors.  Then another 2 months go by with no issues.  Now, August 1st, I run the monthly check, and it comes back with about 7k errors again.  Nothing has changed.  Knowing that I've not done anything new, I had it correct the errors.  But this time, after it was done, I ran a 2nd uncorrecting check and verified it was 100%.  So I'm good.

 

Fast forward to a couple days ago...

 

I bought some more RAM for the server.  It has 4 slots, capable of 128g max.  When I first built the server, I bought 2x32g.  I went back and bought 2 more of the same model.  Shut the server down, then added the new sticks, and ran a memtest on the entire set.  Originally set (by default) for 4 passes.  I let it go through 3.5, about 2 days, and it came back clean.  So I'm 75% sure the memory is not an issue.  Booted up Unraid, it said parity was good.  But just out of curiosity I started another parity check.  Since it's only been about 2 weeks since my last 100% clean check, I didn't expect any issues.  But, after about 4 hours, it had already detected 7k (7374 exactly) errors again.  I'm so confused.  I don't know why this keeps happening.

 

So, I'm running another correction pass now.  Prior to doing this, I did a lot of "spring cleaning".  I took the server apart.  Pulled out all the cards, memory, cpu.  Disconnected all the drives.  Then blew the thing out with compressed air (not that it needed it).  I then put some fresh thermal grease on the CPU, and reconnected everything.  Made sure everything was seated right, and booted up.  Everything checked out.  I did a quick test last night, doing an uncorrecting check, and it errored out again after a couple of hours.  So I'm doing that correction pass.

 

I'm not sure what could be causing my problems.  My brother, whom I gave my older server to, and bought a license for, etc, has had no issues at all, doing the same thing I am (Plex server, with nzbget docker, etc).  The one advantage he seems to have, is the RAM is ECC, whereas mine is not.  It makes me pause and think; is ECC RAM necessary for proper Parity?  I might have to go back and rebuild my system from scratch again if that's the case, because my motherboard doesn't support it.  Something else I've been thinking.. If I have dockers running when a parity check is running, could they be interfering with the process?  Like, I mentioned nzbget..if it's downloading while a check is occurring, could it be writing files that aren't getting added to parity?  If this is a possibility, is there a way to turn off dockers and vm's while a check is running?

 

I'm really just at a loss of ideas here.  With no hardware changes (other than the new ram 2 days ago), why would I suddenly go from 0 errors for months, to over 120k then back to 0 for months, then back up to around 7k?  Also... I have a UPS, and it does do it's job.  But I've not had any improper shutdowns due to power failures (I've got it timed so I get plenty of time to safely shut down).  I'm just looking for any possible reasons.

 

I know how to post a debug log, or whatever it is.  I'm hesitant to do it right now because the parity check is taking place.  If it would help, let me know and I'll include it.

 

Thanks in advance to whomever feels they want to try and help me!

[itimpi]

I would suggest you at least post your diagnostics zip file to your next post in this thread so we can see what is going on.

[hansolo77]

Yup.. I had a feeling that would be necessary.    I just didn't know if it was something I can do while the check is running.

 

One quickie... I logged into the terminal to see if I could catch the log of the parity check, and I'm seeing a whole lot of red:

Aug 15 11:21:44 Kyber nginx: 2022/08/15 11:21:44 [error] 9999#9999: nchan: Out of shared memory while allocating message of size 503. Increase nchan_max_reserved_memory.
Aug 15 11:21:44 Kyber nginx: 2022/08/15 11:21:44 [error] 9999#9999: *783811 nchan: error publishing message (HTTP status code 500), client: unix:, server: , request: "POST /pub/devices?buffer_length=1 HTTP/1.1", host: "localhost"
Aug 15 11:21:44 Kyber nginx: 2022/08/15 11:21:44 [error] 9999#9999: MEMSTORE:00: can't create shared message for channel /devices
Aug 15 11:21:44 Kyber nginx: 2022/08/15 11:21:44 [crit] 9999#9999: ngx_slab_alloc() failed: no memory
Aug 15 11:21:44 Kyber nginx: 2022/08/15 11:21:44 [error] 9999#9999: shpool alloc failed
Aug 15 11:21:44 Kyber nginx: 2022/08/15 11:21:44 [error] 9999#9999: nchan: Out of shared memory while allocating message of size 234. Increase nchan_max_reserved_memory.
Aug 15 11:21:44 Kyber nginx: 2022/08/15 11:21:44 [error] 9999#9999: *783812 nchan: error publishing message (HTTP status code 500), client: unix:, server: , request: "POST /pub/arraymonitor?buffer_length=1 HTTP/1.1", host: "localhost"
Aug 15 11:21:44 Kyber nginx: 2022/08/15 11:21:44 [error] 9999#9999: MEMSTORE:00: can't create shared message for channel /arraymonitor
Aug 15 11:21:44 Kyber nginx: 2022/08/15 11:21:44 [crit] 9999#9999: ngx_slab_alloc() failed: no memory
Aug 15 11:21:44 Kyber nginx: 2022/08/15 11:21:44 [error] 9999#9999: shpool alloc failed
Aug 15 11:21:44 Kyber nginx: 2022/08/15 11:21:44 [error] 9999#9999: nchan: Out of shared memory while allocating message of size 824. Increase nchan_max_reserved_memory.
Aug 15 11:21:44 Kyber nginx: 2022/08/15 11:21:44 [error] 9999#9999: *783813 nchan: error publishing message (HTTP status code 500), client: unix:, server: , request: "POST /pub/update3?buffer_length=1 HTTP/1.1", host: "localhost"
Aug 15 11:21:44 Kyber nginx: 2022/08/15 11:21:44 [error] 9999#9999: MEMSTORE:00: can't create shared message for channel /update3
Aug 15 11:21:44 Kyber nginx: 2022/08/15 11:21:44 [crit] 9999#9999: ngx_slab_alloc() failed: no memory
Aug 15 11:21:44 Kyber nginx: 2022/08/15 11:21:44 [error] 9999#9999: shpool alloc failed
Aug 15 11:21:44 Kyber nginx: 2022/08/15 11:21:44 [error] 9999#9999: nchan: Out of shared memory while allocating message of size 233. Increase nchan_max_reserved_memory.
Aug 15 11:21:44 Kyber nginx: 2022/08/15 11:21:44 [error] 9999#9999: *783815 nchan: error publishing message (HTTP status code 500), client: unix:, server: , request: "POST /pub/wireguard?buffer_length=1 HTTP/1.1", host: "localhost"
Aug 15 11:21:44 Kyber nginx: 2022/08/15 11:21:44 [error] 9999#9999: MEMSTORE:00: can't create shared message for channel /wireguard
Aug 15 11:21:44 Kyber nginx: 2022/08/15 11:21:44 [crit] 9999#9999: ngx_slab_alloc() failed: no memory
Aug 15 11:21:44 Kyber nginx: 2022/08/15 11:21:44 [error] 9999#9999: shpool alloc failed
Aug 15 11:21:44 Kyber nginx: 2022/08/15 11:21:44 [error] 9999#9999: nchan: Out of shared memory while allocating message of size 605. Increase nchan_max_reserved_memory.
Aug 15 11:21:44 Kyber nginx: 2022/08/15 11:21:44 [error] 9999#9999: *783816 nchan: error publishing message (HTTP status code 500), client: unix:, server: , request: "POST /pub/temperature?buffer_length=1 HTTP/1.1", host: "localhost"
Aug 15 11:21:44 Kyber nginx: 2022/08/15 11:21:44 [error] 9999#9999: MEMSTORE:00: can't create shared message for channel /temperature
Aug 15 11:21:44 Kyber nginx: 2022/08/15 11:21:44 [crit] 9999#9999: ngx_slab_alloc() failed: no memory
Aug 15 11:21:44 Kyber nginx: 2022/08/15 11:21:44 [error] 9999#9999: shpool alloc failed
Aug 15 11:21:44 Kyber nginx: 2022/08/15 11:21:44 [error] 9999#9999: nchan: Out of shared memory while allocating message of size 800. Increase nchan_max_reserved_memory.
Aug 15 11:21:44 Kyber nginx: 2022/08/15 11:21:44 [error] 9999#9999: *783817 nchan: error publishing message (HTTP status code 500), client: unix:, server: , request: "POST /pub/cpuload?buffer_length=1 HTTP/1.1", host: "localhost"
Aug 15 11:21:44 Kyber nginx: 2022/08/15 11:21:44 [error] 9999#9999: MEMSTORE:00: can't create shared message for channel /cpuload
Aug 15 11:21:45 Kyber nginx: 2022/08/15 11:21:45 [crit] 9999#9999: ngx_slab_alloc() failed: no memory
Aug 15 11:21:45 Kyber nginx: 2022/08/15 11:21:45 [error] 9999#9999: shpool alloc failed
Aug 15 11:21:45 Kyber nginx: 2022/08/15 11:21:45 [error] 9999#9999: nchan: Out of shared memory while allocating message of size 3620. Increase nchan_max_reserved_memory.
Aug 15 11:21:45 Kyber nginx: 2022/08/15 11:21:45 [error] 9999#9999: *783818 nchan: error publishing message (HTTP status code 500), client: unix:, server: , request: "POST /pub/var?buffer_length=1 HTTP/1.1", host: "localhost"
Aug 15 11:21:45 Kyber nginx: 2022/08/15 11:21:45 [error] 9999#9999: MEMSTORE:00: can't create shared message for channel /var
Aug 15 11:21:45 Kyber nginx: 2022/08/15 11:21:45 [crit] 9999#9999: ngx_slab_alloc() failed: no memory
Aug 15 11:21:45 Kyber nginx: 2022/08/15 11:21:45 [error] 9999#9999: shpool alloc failed
Aug 15 11:21:45 Kyber nginx: 2022/08/15 11:21:45 [error] 9999#9999: nchan: Out of shared memory while allocating message of size 503. Increase nchan_max_reserved_memory.
Aug 15 11:21:45 Kyber nginx: 2022/08/15 11:21:45 [error] 9999#9999: *783819 nchan: error publishing message (HTTP status code 500), client: unix:, server: , request: "POST /pub/disks?buffer_length=1 HTTP/1.1", host: "localhost"
Aug 15 11:21:45 Kyber nginx: 2022/08/15 11:21:45 [error] 9999#9999: MEMSTORE:00: can't create shared message for channel /disks
Aug 15 11:21:45 Kyber nginx: 2022/08/15 11:21:45 [crit] 9999#9999: ngx_slab_alloc() failed: no memory
Aug 15 11:21:45 Kyber nginx: 2022/08/15 11:21:45 [error] 9999#9999: shpool alloc failed
Aug 15 11:21:45 Kyber nginx: 2022/08/15 11:21:45 [error] 9999#9999: nchan: Out of shared memory while allocating message of size 314. Increase nchan_max_reserved_memory.
Aug 15 11:21:45 Kyber nginx: 2022/08/15 11:21:45 [error] 9999#9999: *783820 nchan: error publishing message (HTTP status code 500), client: unix:, server: , request: "POST /pub/parity?buffer_length=1 HTTP/1.1", host: "localhost"
Aug 15 11:21:45 Kyber nginx: 2022/08/15 11:21:45 [error] 9999#9999: MEMSTORE:00: can't create shared message for channel /parity
Aug 15 11:21:45 Kyber nginx: 2022/08/15 11:21:45 [crit] 9999#9999: ngx_slab_alloc() failed: no memory
Aug 15 11:21:45 Kyber nginx: 2022/08/15 11:21:45 [error] 9999#9999: shpool alloc failed
Aug 15 11:21:45 Kyber nginx: 2022/08/15 11:21:45 [error] 9999#9999: nchan: Out of shared memory while allocating message of size 503. Increase nchan_max_reserved_memory.
Aug 15 11:21:45 Kyber nginx: 2022/08/15 11:21:45 [error] 9999#9999: *783821 nchan: error publishing message (HTTP status code 500), client: unix:, server: , request: "POST /pub/devices?buffer_length=1 HTTP/1.1", host: "localhost"
Aug 15 11:21:45 Kyber nginx: 2022/08/15 11:21:45 [error] 9999#9999: MEMSTORE:00: can't create shared message for channel /devices
Aug 15 11:21:45 Kyber nginx: 2022/08/15 11:21:45 [crit] 9999#9999: ngx_slab_alloc() failed: no memory
Aug 15 11:21:45 Kyber nginx: 2022/08/15 11:21:45 [error] 9999#9999: shpool alloc failed
Aug 15 11:21:45 Kyber nginx: 2022/08/15 11:21:45 [error] 9999#9999: nchan: Out of shared memory while allocating message of size 234. Increase nchan_max_reserved_memory.
Aug 15 11:21:45 Kyber nginx: 2022/08/15 11:21:45 [error] 9999#9999: *783822 nchan: error publishing message (HTTP status code 500), client: unix:, server: , request: "POST /pub/arraymonitor?buffer_length=1 HTTP/1.1", host: "localhost"
Aug 15 11:21:45 Kyber nginx: 2022/08/15 11:21:45 [error] 9999#9999: MEMSTORE:00: can't create shared message for channel /arraymonitor
Aug 15 11:21:45 Kyber nginx: 2022/08/15 11:21:45 [crit] 9999#9999: ngx_slab_alloc() failed: no memory
Aug 15 11:21:45 Kyber nginx: 2022/08/15 11:21:45 [error] 9999#9999: shpool alloc failed
Aug 15 11:21:45 Kyber nginx: 2022/08/15 11:21:45 [error] 9999#9999: nchan: Out of shared memory while allocating message of size 810. Increase nchan_max_reserved_memory.
Aug 15 11:21:45 Kyber nginx: 2022/08/15 11:21:45 [error] 9999#9999: *783823 nchan: error publishing message (HTTP status code 500), client: unix:, server: , request: "POST /pub/update3?buffer_length=1 HTTP/1.1", host: "localhost"
Aug 15 11:21:45 Kyber nginx: 2022/08/15 11:21:45 [error] 9999#9999: MEMSTORE:00: can't create shared message for channel /update3
Aug 15 11:21:45 Kyber nginx: 2022/08/15 11:21:45 [crit] 9999#9999: ngx_slab_alloc() failed: no memory
Aug 15 11:21:45 Kyber nginx: 2022/08/15 11:21:45 [error] 9999#9999: shpool alloc failed
Aug 15 11:21:45 Kyber nginx: 2022/08/15 11:21:45 [error] 9999#9999: nchan: Out of shared memory while allocating message of size 233. Increase nchan_max_reserved_memory.
Aug 15 11:21:45 Kyber nginx: 2022/08/15 11:21:45 [error] 9999#9999: *783824 nchan: error publishing message (HTTP status code 500), client: unix:, server: , request: "POST /pub/wireguard?buffer_length=1 HTTP/1.1", host: "localhost"
Aug 15 11:21:45 Kyber nginx: 2022/08/15 11:21:45 [error] 9999#9999: MEMSTORE:00: can't create shared message for channel /wireguard
Aug 15 11:21:45 Kyber nginx: 2022/08/15 11:21:45 [crit] 9999#9999: ngx_slab_alloc() failed: no memory
Aug 15 11:21:45 Kyber nginx: 2022/08/15 11:21:45 [error] 9999#9999: shpool alloc failed
Aug 15 11:21:45 Kyber nginx: 2022/08/15 11:21:45 [error] 9999#9999: nchan: Out of shared memory while allocating message of size 802. Increase nchan_max_reserved_memory.
Aug 15 11:21:45 Kyber nginx: 2022/08/15 11:21:45 [error] 9999#9999: *783825 nchan: error publishing message (HTTP status code 500), client: unix:, server: , request: "POST /pub/cpuload?buffer_length=1 HTTP/1.1", host: "localhost"
Aug 15 11:21:45 Kyber nginx: 2022/08/15 11:21:45 [error] 9999#9999: MEMSTORE:00: can't create shared message for channel /cpuload
Aug 15 11:21:46 Kyber nginx: 2022/08/15 11:21:46 [crit] 9999#9999: ngx_slab_alloc() failed: no memory
Aug 15 11:21:46 Kyber nginx: 2022/08/15 11:21:46 [error] 9999#9999: shpool alloc failed
Aug 15 11:21:46 Kyber nginx: 2022/08/15 11:21:46 [error] 9999#9999: nchan: Out of shared memory while allocating message of size 235. Increase nchan_max_reserved_memory.
Aug 15 11:21:46 Kyber nginx: 2022/08/15 11:21:46 [error] 9999#9999: *783826 nchan: error publishing message (HTTP status code 500), client: unix:, server: , request: "POST /pub/notify?buffer_length=1 HTTP/1.1", host: "localhost"
Aug 15 11:21:46 Kyber nginx: 2022/08/15 11:21:46 [error] 9999#9999: MEMSTORE:00: can't create shared message for channel /notify
Aug 15 11:21:46 Kyber nginx: 2022/08/15 11:21:46 [crit] 9999#9999: ngx_slab_alloc() failed: no memory
Aug 15 11:21:46 Kyber nginx: 2022/08/15 11:21:46 [error] 9999#9999: shpool alloc failed
Aug 15 11:21:46 Kyber nginx: 2022/08/15 11:21:46 [error] 9999#9999: nchan: Out of shared memory while allocating message of size 3620. Increase nchan_max_reserved_memory.
Aug 15 11:21:46 Kyber nginx: 2022/08/15 11:21:46 [error] 9999#9999: *783827 nchan: error publishing message (HTTP status code 500), client: unix:, server: , request: "POST /pub/var?buffer_length=1 HTTP/1.1", host: "localhost"
Aug 15 11:21:46 Kyber nginx: 2022/08/15 11:21:46 [error] 9999#9999: MEMSTORE:00: can't create shared message for channel /var
Aug 15 11:21:46 Kyber nginx: 2022/08/15 11:21:46 [crit] 9999#9999: ngx_slab_alloc() failed: no memory
Aug 15 11:21:46 Kyber nginx: 2022/08/15 11:21:46 [error] 9999#9999: shpool alloc failed
Aug 15 11:21:46 Kyber nginx: 2022/08/15 11:21:46 [error] 9999#9999: nchan: Out of shared memory while allocating message of size 503. Increase nchan_max_reserved_memory.
Aug 15 11:21:46 Kyber nginx: 2022/08/15 11:21:46 [error] 9999#9999: *783828 nchan: error publishing message (HTTP status code 500), client: unix:, server: , request: "POST /pub/disks?buffer_length=1 HTTP/1.1", host: "localhost"
Aug 15 11:21:46 Kyber nginx: 2022/08/15 11:21:46 [error] 9999#9999: MEMSTORE:00: can't create shared message for channel /disks

 

Why would I be getting "no memory" errors?  Unless it's not really memory but storage, and the storage for my logs is full?  In either case, here's my diagnostics:

kyber-diagnostics-20220815-1450.zip

[JorgeB]

12 hours ago, hansolo77 said:

used the Krusader docker to forcibly move data around on my drives; moving data off of one, putting it on another, etc, without allowing the parity process to take place. 

FYI this won't cause sync errors.

 

Running a Ryzen server with overclocked RAM like you have is known to corrupt data, respect max officially supported speed for your config and see it it helps.

 

https://forums.unraid.net/topic/46802-faq-for-unraid-v6/?do=findComment&comment=819173

 

[hansolo77]

Yeah, we went through this before.  I forgot about it honestly.  When I installed the additional RAM, the BIOS reset.  So I went in and set it all back up the way I thought it should be.  I found my old thread here and I'm going back into the BIOS now to adjust things.  Hopefully that's all I need.

[codefaux]

Glad you solved it, something to always keep in mind when building a system.

 

 

To the general public who winds up here looking for similar solutions;

(Posting it here because they won't go back and experience it like you did. Seriously not trying to rub it in. This needs to be seen *anywhere* the conclusion is "overclocked RAM" and is *NOT* JUST FOR RYZEN SYSTEMS but they do it worst. We've had this talk, I know you know.)

 

Never overclock RAM, at all, regardless of what it says on the box, INCLUDING XMP, without testing it. Gaming system, Facebook system, ESPECIALLY A SERVER. XMP IS OVERCLOCKING.

 

I'll say it again because people argue it so much; before you argue, Google it. XMP IS OVERCLOCKING. It's a "factory supported overclock" but it IS OVERCLOCKING, and you HAVE TO TEST FOR STABILITY when overclocking. Read Intel's documentation on the XMP spec. I DO NOT CARE if your BIOS defaults it on, VIGILANTLY turn it back *OFF* unless you're going to run an extremely extended *(DOZENS OF HOURS) RAM test when you're building a server. RAM overclocking leads to SILENT DATA CORRUPTION, which is *RIDICULOUS* on a SERVER, which is explicitly present to HANDLE DATA. I should also note that I personally have never seen a literal server-intended motherboard which supports *any* means of overclocking, and I feel like that's due to the causal link between overclocking and data corruption. Overclocking server hardware is NOT a good decision, unless you also test for stability.

 

Overclocking RAM without testing it is literally identical to knowingly using a stick of failing RAM. You're running a piece of engineered electronics faster than the engineers who built it said to, often at higher voltages than it's designed for, to get it to go fractionally faster. Does that move a bottleneck? Not even a little bit, not since the numbered Pentium era. It HELPS on a high end system, IF YOU TEST IT THOROUGHLY, but I would NEVER overclock RAM on a server. I feel like NOT having silently corrupted data on a randomly unstable system is better overall than "the benchmark number is a little higher and I SWEAR programs load faster!" Stop thinking overclocking RAM is safe. Stop using XMP on "server" systems. Any potential speed gain is not worth it. Be safer with your systems, and your data.

[hansolo77]

I haven't successfully solved the problem "YET".  I've got it running another parity check now.  But it's not come up with any sync errors since I started it 8 hours ago, and it was just a couple hours in before when it started throwing up errors.  The thing about this that bugged me, I completely forgot about leaving the RAM speed at default.  I had it set, and it was working.  When I installed the additional memory, the BIOS detected the change and reset it's defaults.  Only then did I go in and change it back from 2666mhz (default/auto) to the RAM spec of 3600mhz.  It just slipped my mind about not changing that.  It was only after I remembered a year ago having crazy amounts of parity errors that had me go back and find my old thread where the RAM speed was the issue.  

 

It's really confusing actually.  The spec sheet for the motherboard says it can support all these crazy speeds, even up to like 4k as long as the CPU is a 3rd gen.  Then I look at the spec sheet for the CPU and it too says i can support up to the 4k mhz speeds.  So in my mind, if the RAM I buy is built with the label indicating it's 3600mhz, I just automatically start tweaking BIOS settings to get that speed.  It's what everything else should support.  But it wasn't until I saw the post/screenshot that indicated the Ryzen processors are only designed to run RAM at 3200mhz MAX for stability, that I realized I was overclocking.  Strange thing is, I don't remember if I underclocked it to 3200mhz before or not.  By default, when the BIOS detects the RAM it sets it to 2666mhz.  So that's what I have it set to now.

 

For added assurance, I did go ahead and disable the C-States.  There is another setting for controlling the CPU idle power, which I set to something like "standard" versus "low" or "disabled" (I don't recall what it was exactly).  I think by default this motherboard sets it to disabled, which allows it to have the full range of power.  I read that having that capability can cause corruption if the CPU idles harshly, or something like that.  Another setting somebody suggested checking was the IOMMU.  The options are AUTO, ENABLED, and DISABLED.  I left it set to auto for now.  I feel disabling may prevent the 10G NIC from being usable in my VM's, which I need.  But I didn't want to completely ENABLE it, if IOMMU was in fact givng me trouble and the AUTO just enables what it needs.

 

Anyway, so far so good.  I hope this works.  I really miss having my data.  I don't want to risk accessing any part of the array until the check is completed 100% clean. 

[JorgeB]

Note that even if the problem is fixed the 1st check might find errors, from previous runs, after those are fixed no more should be found on future runs.