Parity check three times slower than sync or rebuild. Is it normal?


Recommended Posts

There's one case on this thread that seems to be AMD board or CPU related, others appear to be sata controller related, check your CPU usage during a parity check, if it’s very high maybe you have the same AMD issue.

 

Also please post your complete hardware, may help diagnose your issue.

 

Sure.  I just started a parity check to test.  Initially, everything appeared what I would expect from v5.  ~100+ MB/sec rates.  CPU utilization was ranging from 20% - 25%.  Average time to complete was around 5 hours.  Again, this all looks good.

 

I sat there watching it very close.  At exactly 2.1% completed it hit the wall and dropped down to exactly 35 Mb/sec.  However, CPU Utilization actually dropped.  It went down to ~10% and stays there.  The average time to complete spiked up to over 15 hours, exactly what others are reporting here with much slower parity check times when compared to v5 and almost exactly 1/3.  All the drives look great with temps ranging from 26 - 28 degrees.  The 35 Mb/sec parity check will stay there and 15 hours is a reality compared to 5 hours with v5.  What in the world could be causing the sudden and sustained change?

 

The equipment is as follows:

 

CPU:  AMD A87650K

Motherboard:  MSI A78M-E35

RAM:  4 Gigs Corsair DDR3 1600 (single channel, one bank)

SATA Controller:  All onboard the motherboard

Cache Drive:  SSD Samsung

Parity Drive:  WD 2 TB 7200 RPM

Data Drives:  1 1TB WD, 1 2TB Samsung, 1 2TB Hitachi, 1 2TB WD

 

 

 

 

Link to comment
  • Replies 79
  • Created
  • Last Reply

Top Posters In This Topic

Top Posters In This Topic

Posted Images

Sorry to take this in a different direction ... but can't resist. "Three times slower"? What does that mean. Do you mean it takes "3x as long"?

 

I would think that if you had a disk that ran a preclear in 10 hours, and another than ran it in 11 hours, you could compare a third drive that took 12 hours as "two times slower"?

 

Gives me a headache. ;)

Link to comment

Sorry to take this in a different direction ... but can't resist. "Three times slower"? What does that mean. Do you mean it takes "3x as long"?

 

You have it right.  My parity checks took 5 hours with v5.  Using v6 with the same drives and upgraded hardware now take 15 hours.

Link to comment

Sorry to take this in a different direction ... but can't resist. "Three times slower"? What does that mean. Do you mean it takes "3x as long"?

 

You have it right.  My parity checks took 5 hours with v5.  Using v6 with the same drives and upgraded hardware now take 15 hours.

 

"... and upgraded hardware  ..." ==> i.e. DIFFERENT hardware.    While it certainly SHOULD be faster, it IS different.    Did you by any chance try this same system using v6 on the original hardware?    Just curious if that also has this same slowdown issue.

 

There have been enough reports of slow parity check speeds with v6 that it seems there must be something that's different between v5 & v6 drivers for some of the hardware; but not enough reports for any consistent hardware configuration to "jump out" as the culprit.    The more data the better in terms of isolating this, so it would definitely be useful to know if y our original system has the same slowdown with v6.

 

Link to comment

@Stripe, You appear to have the same problem as the thread starter, high CPU usage during parity check causing a slowdown, he also had an older AMD CPU, Opteron 2346 HE 1.8Mhz.

 

@TUMS, you have the same issue I and many other have with the SAS2LP, there’s no apparent solution, you can try to minimize it by having the card on a PCIe 2.0 8x slot (some boards have 16x slots that are only 4x electrical, this issue gets a lot worse on a 4x slot) and by running tunables tester.

 

@JP, I have no clue what you’re issue is but if you only notice the slowdown at 21% parity check it seems unrelated, did you complete a parity check and does the speed remain low until the end?

 

Link to comment

"... and upgraded hardware  ..." ==> i.e. DIFFERENT hardware.    While it certainly SHOULD be faster, it IS different.    Did you by any chance try this same system using v6 on the original hardware?    Just curious if that also has this same slowdown issue.

 

There have been enough reports of slow parity check speeds with v6 that it seems there must be something that's different between v5 & v6 drivers for some of the hardware; but not enough reports for any consistent hardware configuration to "jump out" as the culprit.    The more data the better in terms of isolating this, so it would definitely be useful to know if y our original system has the same slowdown with v6.

 

Trust me Gary.  I understand the challenge this presents to people at Lime Tech.  I did NOT use v6 on the original hardware because it was broken and is what led me to upgrading in the first place.  The old hardware used an AMD Sempron Processor and Biostar Motherboard.  It was a pretty popular choice amongst those here about 5 years ago.  5 years went by and I had no issues, but then about 4 weeks ago the Server would just freeze.  The Server would remain powered on, but the monitor went black, you couldn't Telnet in to it, and the log files showed no issues.  Memtest passed after 5 days of testing.  There was exhaustive troubleshooting advice provided by other forum members, but ultimately, all anyone could point to was a possibility that there was an issue with the motherboard, processor, ram, or power supply OR a combination of these.     

 

Since my time was worth something I simply elected to replace all this equipment and the only thing I brought over was the drives.  It only made sense to upgrade to v6 at this time as well.  Since the replacement of this equipment I haven't had any "freezes."  But I have seen what might be two potential issues with v6:

 

1.  Web GUI Issues - There are other posts dedicated to this where I've provided feedback, but v6's GUI will lock up when using Internet Explorer, Firefox, or Safari from a number of different clients.  This takes about 5 - 15 commands for it to happen.  It is only the GUI that locks up.  Everything else (NAS, plugins, dockers, etc.) is all operational when the GUI locks up.  You have to powerdown restart to get it back.  I have no idea why, but it never happens using Chrome and Chrome will work from any client.  It is very odd and I can't even imagine how difficult this might be to sort out from a developer's perspective.  Ultimately, it is pretty much a non-issue for me now.  I just use Chrome for any needed unRAID commands.

 

2.  Parity Checks take 3x longer in v6 compared to v5 - This is the issue we are discussing in this thread.

 

As part of this journey I did upgrade the flash drive as well.  Again, all new hardware except for the drives.  There was an outside chance the old flash, which had v5 on it was part of the problem when I was having the freezes with the original hardware.  So I wrote LimeTech to get a new flash with v6 on it registered.  So I DO have the old flash with v5 on it (not used) as well as the new flash with v6.  If it helps I might switch all the hardware out with the old system and see what the parity checks are with v5 versus v6 on it.  However, here is the rub.  Again, the old hardware will eventually freeze up and I'm certain it will do so during a parity check.  Moving up to v6 was one of the first things I tried when the freezes started so I know I at least tried, but never got so far as the parity check.  Once the freeze happened in v6 on the old hardware I knew it needed to be upgrade. 

 

I apologize for the long post, but I thought I would explain some of the logic behind all this.  Again, I'm certain these bugs are very challenging for people at Lime Tech, but it would never keep me from being a die-hard Lime Tech fan.  Even with these two issues my unRAID server is running great otherwise.  PLEX transcoding is smooth and reads from the NAS are great.  The GUI and Parity Check issues are merely an inconvenience and not the end of the world.  Thanks again.             

Link to comment

@JP, I have no clue what you’re issue is but if you only notice the slowdown at 21% parity check it seems unrelated, did you complete a parity check and does the speed remain low until the end?

 

Not 21%, but 2.1%, in case it matters.  Yes, I did complete a parity check.  It did take a little over 15 hours compared to 5 hours with v5. 

Link to comment

Did you try reverting to v5 on the new hardware to see if the parity check speed bumps back up to what you were experiencing before?    This would confirm that the issue is v6 and not your new hardware.  I think that's VERY likely the case, but it's always nice to confirm it.

 

Link to comment

A few more tests that I hope can help Limetech fix or at least minimize the issue with the SAS2LP.

 

All parity checks done on same server with yet a different board (Supermicro X7SBE / Celeron E1200 1.6 / 4GB DDR2)

Controllers used: Onboard Intel ICH9R, Adaptec AAR-1430SA PCIe 1.0 4x, AOC-SAT2-MV8 PCI-X, AOC-SASLP-MV8 PCIe 1.0 4x, AOC-SAS2LP-MV8 PCIe 2.0 8x (on a PCIe 1.0 8x slot)

 

3 x Intel SSD 40Gb

 

 

V5b12

Intel Onboard - time=204sec - 196.2 Mb/s

AAR-1430SA - time=204sec - 196.2 Mb/s

SAT2-MV8 - time=205sec - 195.2 Mb/s

SASLP-MV8 - time=204sec - 196.2 Mb/s

SAS2LP-MV8 - time=204sec - 196.2 Mb/s

 

V5.0.6

Intel Onboard - time=204sec - 196.2 Mb/s

AAR-1430SA - time=205sec - 195.2 Mb/s

SAT2-MV8 - time=205sec - 195.2 Mb/s

SASLP-MV8 - time=211sec - 189.7 Mb/s

SAS2LP-MV8 - time=259sec - 154.5 Mb/s

 

v6.0.1

Intel Onboard- time=217sec - 184.4 Mb/s

AAR-1430SA - time=218sec - 183.6 Mb/s

SAT2-MV8 - time=226sec - 177.1 Mb/s

SASLP-MV8 - time=243sec - 164.7 Mb/s

SAS2LP-MV8 - time=520sec - 77 Mb/s

 

Whatever the issue is in V6 appears to also be present in v5.0.6 comparing with v5beta12, although in a much smaller scale.

All controllers lose some speed in v6, but the SAS2LP is much more penalized.

 

A few more observations:

This is the 5th different board I use, 4 different manufactures and chipsets, all are affected.

The amount of RAM doesn’t make any difference, tried with 4GB and 16GB

The CPU used also doesn’t make any difference, from a Celeron 1.6Mhz to a 3rd gen i5 overclocked to 4.5Ghz

 

In fact, the only thing that for me consistently made a difference was the type of PCIe slot used, but always with a big slowdown, using only 3 disks:

 

SAS2LP on PCIe 2.0 8x: 95 Mb/s

SAS2LP on PCIe 2.0 4x: 38 Mb/s

SAS2LP on PCIe 1.0 8x: 77 Mb/s

 

 

(Edited to add Adaptec 1430SA results)

Link to comment

Did you try reverting to v5 on the new hardware to see if the parity check speed bumps back up to what you were experiencing before?    This would confirm that the issue is v6 and not your new hardware.  I think that's VERY likely the case, but it's always nice to confirm it.

 

I started this last night.  Under v5 the parity check took right at 7 hours.  This was more than I thought it would be, but I still think v6 will be considerably longer.  I won't know by how much until it completes and I just started it this morning.  One item to note is that under v5 my speeds were sustained at 100+ MB/sec for quite some time, maybe around 25% complete, but it did ramp down more than I thought.  I saw it at ~75 MB/sec at around 50% complete and then it did go down to about ~55 MB/sec at one point.  However, I still think this probably deviates quite a bit from what v6 is doing.  With v6 it doesn't appear to take much time at all (around 2% complete) for it to fall off a cliff from 100+ MB/sec to 35 Mb/sec and then stay there for the rest of the duration.  Again, once this parity check completes under v6 I'll have something more tangible to pass on.

Link to comment

It appears I owe those looking in to this issue an apology.  The parity check in v6 completed in 7 hours and 17 minutes with an average speed of 76.2 MB/sec.  The parity check in v5 I think was just under 7 hours, but clearly not a significant difference as I initially thought.  It doesn't appear I have the same issue that others might be reporting in this thread.

 

It still seems odd to me though because I know I wasn't dreaming with how long the initial parity check took under v6 a few weeks ago.  I specifically remember starting it in the morning and I was shocked to see it was only a little over half-way done by that evening.  Again, I want to say it took about 15 hours.  The only difference I did between those parity checks was this time I had all the dockers turned off.  I only had 3 installed at that time (now 4) and they weren't the type of dockers I would think might impact a parity check. 

 

Ultimately, I think this is all good news because an average speed of 76.2 MB/sec seems more reasonable than 35 MB/sec.  Thanks again for the help. 

Link to comment

Just a FYI, the parity check speed drop which I incurred when LT switched to a preemptible kernel in 6.0b15 has now returned back to its faster speed with 6.1rc-1

 

Interesting, just tried it with a SAS2LP on my test server and results are the same as 6.0.1

 

V5beta12 - time=204sec - 196.2 Mb/s

V5.0.6 - time=259sec - 154.5 Mb/s

v6.0.1 - time=520sec - 77 Mb/s

V6.1rc1 - time=521sec - 76.8 MB/sec

 

 

Link to comment

I don't think that my parity slowness issue is like what you guys have posted so far.  I originally posted in this thread

 

http://lime-technology.com/forum/index.php?topic=41719.0

 

But it was suggested to me that I should start looking at this thread. 

 

I haven't heard anyone mention the kworker process yet.  The Kworker process seems to be eating my cpu alive when a parity check is running. 

 

Screenshot Attached.  What does this process do?  Anyone have any ideas as to why it would go wild during a parity check?  When I stop the check it goes back down to 0%.

kworker.jpg.12f15070559e62ca2d9e83bcbeda11e5.jpg

Link to comment

I decided to downgrade to 6.0 b14, and lo and behold parity check speeds are back to normal.  And a screenshot of Top is attached.  And as one would expect with unraid, the CPU is not a limiting factor in the parity check, even though it's a 7 year old single core celeron. 

 

I found out that the kworker process is directly related to things the linux kernel is doing.  So I was very hopeful that upgrading to 6.1 rc1 as alluded to in Squid's post would help me, but it did not.  I saw no noticeable difference between parity check speeds in 6.0.1 and 6.1rc1. 

 

I noted that earlier in this thread several posts referred to the SASLP and SAS2LP.  And it seemed to be that most people thought that the SAS2LP might be a problem.  I have an SASLP with 7 drives connected to it in my system, with no open ports to experiment with elsewhere in my system. 

6b14paritychecktop.jpg.647613488e2bd10e0681416735365bb7.jpg

Link to comment

I just upgraded to 6.0.1 from V5 and discovered a huge drop in parity check speed.

I also have the SAS2LP card but with an Intel celeron processor.

 

I can barely sustain 11MB/s during parity check and my CPU is pinned.

My estimated time to finish is around 3 days.

 

I must have missed this potential problem when I was researching the upgrade.

Hopefully this can be solved since it appears to have been introduced during the Beta phase.

 

 

Link to comment

I also have the SAS2LP card but with an Intel celeron processor.

 

I can barely sustain 11MB/s during parity check and my CPU is pinned.

My estimated time to finish is around 3 days.

 

I used an older single core celeron [email protected] on my test server and parity check was about 14Mb/s, CPU pinned by kworker process, same as user TSM above.

 

Only 3 SSDs connected to the motherboard controller, so it appears v6.0.1 requires a better CPU, my Core2Duo based dual core celeron [email protected] doesn't have this issue, so it should only affect P4 based celerons and probably single core P4 CPUs.

 

Link to comment

johnnie.black,

 

I thought this issue is directly related to the SAS2LP controller?

TSM confirmed that his parity checks were fast using 6.0 b14 and slow on later versions.

 

Also the quote from Squid:

 

"Just a FYI, the parity check speed drop which I incurred when LT switched to a preemptible kernel in 6.0b15 has now returned back to its faster speed with 6.1rc-1"

 

would indicate there is a known issue.

 

I have a spare Core2Duo E7500 that I was planning on swapping in anyway so I will see if the CPU is really the bottleneck.

 

 

Link to comment

I and some more people have slow parity check with a SAS2LP in v6.0.1 (still present in 6.1RCs) but it's not the same issue you're seeing, speed is not that slow and the CPU utilization during the check is normal, about 40%.

 

The extremely slow speed you're getting should be related to the CPU and resolved if you switch to the E7500, you may your may not then have the SAS2LP issue, with a fully loaded card I get around 40Mb/s, should get around 110M/s.

Link to comment

I and some more people have slow parity check with a SAS2LP in v6.0.1 (still present in 6.1RCs) but it's not the same issue you're seeing, speed is not that slow and the CPU utilization during the check is normal, about 40%.

 

The extremely slow speed you're getting should be related to the CPU and resolved if you switch to the E7500, you may your may not then have the SAS2LP issue, with a fully loaded card I get around 40Mb/s, should get around 110M/s.

For my clarity, are you suggesting he use the different cpu because of a compatibility issue with his old cpu, or just simply because it's faster and therefore a cpu bound process should work faster on a faster cpu? 

Link to comment

I don't know if the issue is due to the poor performance of the P4 based single core celerons ore some compatibility issue with them, but I could reproduce it on my test server, so I believe changing to a core2duo based CPU should solve it.

 

My test server with 3 SSDs connected to the onboard sata, parity check speed:

 

Celeron [email protected] - 14Mb/s

Celeron E1200 [email protected] - 184Mb/s

 

 

Link to comment

I don't know if the issue is due to the poor performance of the P4 based single core celerons ore some compatibility issue with them, but I could reproduce it on my test server, so I believe changing to a core2duo based CPU should solve it.

 

My test server with 3 SSDs connected to the onboard sata, parity check speed:

 

Celeron [email protected] - 14Mb/s

Celeron E1200 [email protected] - 184Mb/s

I hear ya.  I'm wondering if there are people out there who are using this type of processor and are having no problems?  Admittedly I've been contemplating swapping out my motherboard and cpu for a while now.  According to Newegg I bought my mild mannered cpu in February of 2008. 

 

http://www.newegg.com/Product/Product.aspx?Item=N82E16819116038

 

But at this point it's the principle of the thing.  I want to upgrade my cpu because I want to.  Not because I have to.  And frankly the reason I haven't upgraded yet is because until the latest updates my 7 year old hardware has been performing more than adequately for my needs. 

Link to comment

Here is an update on my situation.

 

I swapped out the processor for the E7500 plus 4GB of Ram and my Parity check speed did improve to around 25-29 Mb/s

CPU utilization is sitting at 60%.

 

Definitely an improvement but still much slower than it was on V5.

 

I had a failed disk yesterday and the rebuild ran at close to 100 Mb/s.

 

Looks like there is an issue with 6.0.1 and the SAS2LP controller.

 

Link to comment

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.

Guest
Reply to this topic...

×   Pasted as rich text.   Restore formatting

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.