Parity check three times slower than sync or rebuild. Is it normal?


Recommended Posts

Noticed something I don't understand.

 

On the same system, same conditions, parity sync (writing to parity, reading others) and data disk rebuild (writing to data disk, reading others) both go with average speed ~140 MB/sec (start at ~195 MB/s, end at ~90 MB/s). Parity check (reading only) goes with average speed of ~55 MB/s (varies during check randomly between ~45 and ~70 MB/s).

 

Is it the way it should be in unRAID Trial 6.0-beta14b?

 

Setup: Supermicro H8DME-2, Opteron 2346 HE, 4GB RAM, unRAID Trial 6.0-beta14b.

Parity: 4+4TB=8TB RAID-0 pool via Areca ARC-1110 in PCI-X slot

Disk1: 8TB Seagate Archive in motherboard SATA port

Disk2: 8TB Seagate Archive in motherboard SATA port

 

Link to comment
  • Replies 79
  • Created
  • Last Reply

Top Posters In This Topic

Top Posters In This Topic

Posted Images

Definitely not ==> I'd certainly expect parity checks to run with a similar speed ... with the same downward ramp as the check moves from the outer cylinders to the inner cylinders.

 

This likely has something to do with the use of the RAID array for parity, but I can't think of a reason why it would be so much slower, since you're using the same configuration for syncs and rebuilds.

 

Do you have a 3rd Seagate 8TB unit?    If so ... and if you're inclined to experiment with it, it'd be interesting to see if you get similar behavior if you replace your parity array with a 3rd Archive drive.

 

One other thought => Are you starting this check immediately after doing one of the other activities (sync or rebuild) ??    It's possible that your Seagate drives are "catching up" with band rewrites to empty their persistent cache areas ... this would slow them down a good bit.

 

 

 

Link to comment

This likely has something to do with the use of the RAID array for parity, but I can't think of a reason why it would be so much slower, since you're using the same configuration for syncs and rebuilds.

Yeah... I can't imagine same RAID pool reads fast for data rebuild and reads slow for parity check.

 

Do you have a 3rd Seagate 8TB unit?    If so ... and if you're inclined to experiment with it, it'd be interesting to see if you get similar behavior if you replace your parity array with a 3rd Archive drive.

Will do.

 

One other thought => Are you starting this check immediately after doing one of the other activities (sync or rebuild) ??    It's possible that your Seagate drives are "catching up" with band rewrites to empty their persistent cache areas ... this would slow them down a good bit.

No, at one occasion there was 24+ hours since last activity.

Link to comment

I'm looking at "top"...

 

mdrecoveryd which, I suppose, is parity check thread, has priority 20 and often waits for disk I/O (status=D).

 

Bu there are three processes called migration/1, migration/2 and migration/3, all with priority RT, and migration/3 constantly taking from 50% to 300% of CPU and having status=R.

 

What are those? Googled it out. "The migration kernel process distributes workload across CPU cores."

 

Hmmm.... kernel issue? But then why it does not affect parity sync and data rebuild?

Link to comment

Okay... I mean, not OK.  >:(

 

Next tests: server "Testt", AMD-based. Downgrading unRAID from 6.0-beta14b to 6.0-beta2 did not change anything. Configuration 8TB Archive parity and two 8TB Archives for data disks showed even worse performance for parity check - around 30 MB/s. Process mdrecoveryd again waits for migration/3 a lot. Parity sync performance is as with all previous tests -  ~140 MB/s.

 

My other server (Intel-based, not AMD) "Tower" is overpopulated, underpowered, runs unRAID-5.0,  and generally is far form optimum. However, data disk rebuild it does with average speed ~66 MB/s, and parity check (check, not sync) with ~64 MB/sec. During parity check processes unraidd and mdrecoveryd take ~35% CPU each, migration/0 and migration/1 both take 0%.

 

I'm starting to suspect unRAID kernel problem with this particular AMD-based motherboard/chipset/CPU configuration.

 

Thinking about other tests - unRAID 5.0 on AMD-based platform, and how to test on Intel-based platform without risking my real data on it...

Link to comment

Next test: server "Testt", AMD-based. Downgraded unRAID to 5.0 - no changes. Process migration/3 eats all CPU when parity check, parity sync flies no problem.

 

After consideration I decided that I'm not idiot enough desperate enough brave enough to do this testing on my real data Intel-based server, so I'll wait until another X7SBE (Intel-based) motherboard arrives, should be few days.

Link to comment

Next: tested it on X7SBE (Intel-based) motherboard - no "migration process eating CPU" problem. Parity check speed is ~120 MB/s at start, versus ~50 MB/s at start on AMD-based H8DME-2 motherboard - all else being equal.

 

This is still considerably slower than parity sync (~195 MB/s at start) but that I understand - in parity check there are two processes, unraidd and mdrecoveryd competing for CPU cycles. In parity sync there is only mdrecoveryd.

 

Well, looks like my long living unlove to AMD just got another reason to thrive.

 

On the other hand, if this kernel bug is known since 2011 why is it still here...

Link to comment

Very interesting ... but at least it's trivial to avoid => just stay with Intel  :)

Yeah... but... I could probably live with ridiculously slow parity checks, but I was going to run Dockers on my AMD-based H8DME-2 server, and with AMDs inability to balance load Dockers are out of question. So I'll need a new Intel-based motherboard - can't use X7SBE since it only supports 8GB of RAM. Which also means new expansion cards for PCI-e since almost all of mine are PCI-X. And I was hoping I'm done with hardware purchases for at least a year... another upgrade  :-\ ...upgrade?  ::) ...upgrade!  ;D

Link to comment

...  can't use X7SBE since it only supports 8GB of RAM ...

 

Have you actually confirmed that you need more than 8GB of RAM for what you want to do?    I'd try it before upgrading.

 

One other thought:  SuperMicro boards often work fine with larger RAM modules than they were originally specified for.  I do NOT know if that's true for an X7SBE, but I know for a fact, for example, that an X7SPA-HF-D525 board works perfectly with a pair of 4GB modules (8GB total) even though the specifications say the board only supports 4GB of RAM [One of my servers has been happily running on this board with 8GB of memory for over 2 years].    Only way to know for sure is to try it -- if you happen to have access to a 4GB DDR2 ECC modules you could try it (or you could buy a pair as long as they're returnable ... worst case is it'd cost you a restocking fee).

... If you happen to try that, be sure and post the results -- I'm sure a lot of folks who are using this board would be interested in knowing that. 

 

Note for X7SBE Users:  If you're already tested this and know the answer, it'd be nice if you'd post that here  :)

Link to comment

...  can't use X7SBE since it only supports 8GB of RAM ...

Have you actually confirmed that you need more than 8GB of RAM for what you want to do?    I'd try it before upgrading.

No. I'm just reading the latest  Dockers threads and it's more like "Wow, I want this Docker, and that one too, no way they both (all) will fit in 8GB memory".

 

Thank you for this question, really. It triggered my serious reconsideration of what I really need - versus what I'd like to play with.

 

And you know... first result of this reconsideration is that I realized that I've just recently bought 'bout 32 TB of storage volume which I enjoy playing with, but not really needed... not sure which smilie put here... this  ::) or that  :-[ or maybe this  ???

 

One other thought:  SuperMicro boards often work fine with larger RAM modules than they were originally specified for. 

...

Oh my... you have just sent me to another round of hardware purchases and testing!

Link to comment

but I was going to run Dockers on my AMD-based H8DME-2 server, and with AMDs inability to balance load Dockers are out of question.

 

What is about Docker and AMD that you don't like??

I've just found that on my H8DME-2 (AMD-based) motherboard parity check (not sync) is going ridiculously slow.

 

All thing equal motherboard X7SBE  (Intel-based) provides 2-2.5 faster parity check.

 

I blame this on kernel-level inability to properly (re)distribute load between the CPUs/CPU cores.

 

This is my personal opinion and it does not reflect superiority or inferiority or unrelatediority or whateverriority of the aforementioned fully respected manufacturers.

 

It's just that if the current kernel can't, on  H8DME-2 (AMD-based) motherboard, balance load between the CPUs/CPU cores during simple task of parity check, than I don't even want tor try if it can do it with multiple Dockers running.

 

 

 

 

Link to comment

but I was going to run Dockers on my AMD-based H8DME-2 server, and with AMDs inability to balance load Dockers are out of question.

What is about Docker and AMD that you don't like??

I've just found that on my H8DME-2 (AMD-based) motherboard parity check (not sync) is going ridiculously slow.

 

All thing equal motherboard X7SBE  (Intel-based) provides 2-2.5 faster parity check.

 

I blame this on kernel-level inability to properly (re)distribute load between the CPUs/CPU cores.

 

I've just started a non-correcting parity sync on my AMD server running a few dockers and it is running at between 110 and 120mb/sec.  This is a bit slower than the initial parity generation, but not the kind of numbers you are referring to. 

 

Shutting off all the dockers boosts it about 10 mb/sec., but depending on how hard the dockers are working, the parity will slow down.  Intel will too.  The only thing I notice as difference between my Sandy Bridge Intel Xeon server and this one is that the AMD sucks about 30 watts more power. 

 

Dockers are Transmission, Plex, Owncloud, Tonido, MariaDB,

 

It has an Areca 1280 with 2x2 tb parity and other 9 other 3 and 4 tb drives passed through. Nothing is connected to the motherboard sata ports at present.  Other specs:

 

Gigabyte Technology Co., Ltd. - GA-880GA-UD3H

CPU: AMD Phenom II X6 1055T @ 2800

Memory: 16384 MB (max. installable capacity 16 GB)

Network: eth0: 1000Mb/s - Full Duplex

Link to comment

but I was going to run Dockers on my AMD-based H8DME-2 server, and with AMDs inability to balance load Dockers are out of question.

What is about Docker and AMD that you don't like??

I've just found that on my H8DME-2 (AMD-based) motherboard parity check (not sync) is going ridiculously slow.

 

All thing equal motherboard X7SBE  (Intel-based) provides 2-2.5 faster parity check.

 

I blame this on kernel-level inability to properly (re)distribute load between the CPUs/CPU cores.

 

I've just started a non-correcting parity sync on my AMD server running a few dockers and it is running at between 110 and 120mb/sec.  This is a bit slower than the initial parity generation, but not the kind of numbers you are referring to. 

 

Shutting off all the dockers boosts it about 10 mb/sec., but depending on how hard the dockers are working, the parity will slow down.  Intel will too.  The only thing I notice as difference between my Sandy Bridge Intel Xeon server and this one is that the AMD sucks about 30 watts more power. 

 

Dockers are Transmission, Plex, Owncloud, Tonido, MariaDB,

 

It has an Areca 1280 with 2x2 tb parity and other 9 other 3 and 4 tb drives passed through. Nothing is connected to the motherboard sata ports at present.  Other specs:

 

Gigabyte Technology Co., Ltd. - GA-880GA-UD3H

CPU: AMD Phenom II X6 1055T @ 2800

Memory: 16384 MB (max. installable capacity 16 GB)

Network: eth0: 1000Mb/s - Full Duplex

Thanks for sharing. Well, that tells me that the problem is with Supermicro H8DME-2 motherboard. At least, I don't  know where else it would be. I've tested two different H8DME-2 motherboards, with single CPU and with two CPUs, with different versions of unRAID, with different amounts of RAM from 4GB to 16GB... no changes. And, to recap, the Supermicro X7SBE motherboard, in exactly same setup, does not have this problem. Too bad 'cause I was hoping to use the H8DME-2.  :(

Link to comment

Hate to burst the bubble on Intel and AMD.  It is a linux driver issue of sorts.  My two systems are similar except for the controller cards.  Server 1 uses the onboard SATA and a AOC-SAS2LP-MV8.  System 2 uses only the 2 AOC-SASLP-MV8.

 

Anyway System 1 rebuilds at 130MB/s and parity checks at 40MB/s.  System 2 rebuilds at 70MB/s and parity checks at 70MB/s. System 2 controller cards are max out hence the constant 70MB/s.

 

I have not upgraded system 1 to v6 yet so I don't know if this situation has been fixed.  But every v5 had the same results on the two systems.  I tried bear bone installs of v5 on system 1 and I would always get the same 40MB/s parity checks.  I borrowed a AOC-SASLP-MV8 and put it back in system 1.  The parity checks went back to 70MB/s again.  Made no sense.  I'm just waiting for v6 final:) before I put it on System 1.

Link to comment

Hate to burst the bubble on Intel and AMD.  It is a linux driver issue of sorts.  My two systems are similar except for the controller cards.  Server 1 uses the onboard SATA and a AOC-SAS2LP-MV8.  System 2 uses only the 2 AOC-SASLP-MV8.

 

Anyway System 1 rebuilds at 130MB/s and parity checks at 40MB/s.  System 2 rebuilds at 70MB/s and parity checks at 70MB/s. System 2 controller cards are max out hence the constant 70MB/s.

 

I have not upgraded system 1 to v6 yet so I don't know if this situation has been fixed.  But every v5 had the same results on the two systems.  I tried bear bone installs of v5 on system 1 and I would always get the same 40MB/s parity checks.  I borrowed a AOC-SASLP-MV8 and put it back in system 1.  The parity checks went back to 70MB/s again.  Made no sense.  I'm just waiting for v6 final:) before I put it on System 1.

Haven't really paid much attention to the conversation, but have you tried playing with the disk tunables?  http://lime-technology.com/forum/index.php?topic=29009.0

Link to comment

... Anyway System 1 rebuilds at 130MB/s and parity checks at 40MB/s. ...

Sounds suspiciously similar to mine.

 

Could you please run "top" during parity check for a minute or two? You don't need to run full parity check, it can be canceled at any time.

 

In "good" system, during parity check, there are two processes, unraidd and mdrecoveryd, consuming 20-30% of CPU each, all others consume 0-2%.

 

If, in addition to unraidd and mdrecoveryd, your often see process migration or migration/x in the topmost line, consuming a lot of CPU, then it's probably  problem similar to mine.

Link to comment
  • 2 months later...

Anyone found a solution for this? I'm having the exact same problem.

 

Two of my servers have almost identical hardware:

 

Asrock B75 Pro3-M

Celeron G550 and G540

4GB DDR3 1333

1 x SAS2LP-MV8

15 x 2TB HDDs, 6 on intel onboard, 1 on asmedia onboard, 8 on SAS2LP

 

When I first built parity got around 105Mb/s, limited by some of the older disks which are only 500Gb/platter.

Running a parity check or a disk rebuild I only get 40Mb/s.

 

Either server get the same results, so it can’t be a bad board or controller, board has two pci-e slots, tried the SAS2LP on both.

 

I also did the following tests:

 

Added one SASLP-MV8, connected 4 HDDs from the SAS2LP, still same 40Mb/s parity check.

Removed all disks from SAS2LP and connected all 8 to the SASLP, got about 75MB/s.

Tested on different Unraid 6 beta versions, including beta15 and RC6.

 

 

So, even though SASLP is limited by the slower PCI-e connection it's almost twice as fast as the SAS2LP that should not be bus limited, but this only happens on parity check, parity sync works as expected.

 

Link to comment

Just to add that tried running disk tunables and there was no significative difference, best result was about 45Mb/s.

 

I suspect a compatibility issue with the SAS2LP but it seams to be a card used by many here, is anyone getting a good parity check speed (>80Mb/s) with this card?

 

Thanks

Link to comment

Anyone found a solution for this? I'm having the exact same problem.

 

Two of my servers have almost identical hardware:

 

Asrock B75 Pro3-M

Celeron G550 and G540

...

So it's an Intel-based motherboard... could you please run "top" (via telnet, Putty, ssh, console, etc.) during parity check for a few minutes to verify?

 

In "good" system, during parity check, there are two processes, unraidd and mdrecoveryd, consuming 20-30% of CPU each, all others consume 0-2%.

 

If, in addition to unraidd and mdrecoveryd, your often see process migration or migration/x in the topmost line, consuming a lot of CPU, then it's probably  problem similar to mine.

 

Link to comment

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.

Guest
Reply to this topic...

×   Pasted as rich text.   Restore formatting

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.