Will upgrading controller help my parity check speeds


Recommended Posts

It seems ever since i fully loaded my SAS card, my parity checks slowed down quite a bit.  I've often wondered if this is just the overloaded slot when through putting data for 8 drives.

 

Would upgrading to the newer version of the card help at all? 

 

I currently have a http://www.newegg.com/Product/Product.aspx?Item=9SIA5EM2JS7185&cm_re=AOC-SASLP-MV8-_-16-101-358-_-Product

 

and looking at a http://www.newegg.com/Product/Product.aspx?Item=9SIA5EM2JS7184

 

So the question is would the newer X8 card actually improve the speed of a parity check?    My last speed averaged 78, but have seen it in the lower 70's too.  It used to be in the 90s if not 100, but never gets above 80 anymore.

 

Current hardware / software in my signature is accurate.

 

Thanks all for any help.

Link to comment
  • Replies 80
  • Created
  • Last Reply

Top Posters In This Topic

Top Posters In This Topic

Posted Images

The short answer is yes, the card is overloaded.

 

Your current card is PCIe v1, and so limited to 250MB/s per PCIe lane. It is an x4 card so has a theoretical bandwidth limit of 1GB/s. If the card has 8 drives connected, that means a maximum throughput of 125MB/s per drive. Most current gen spinners are capable of faster than 125MB/s, so on the faster outer sectors of the drives (early in the parity check) the card is not capable of transferring the data to the PCIe bus as fast as 8 drives can simultaneously read.

 

The new card you link to is both x8 and PCIe 2.0. PCIe 2.0 is capable of 500MB/s per lane, so x8 gives you 4GB/s throughput, which is plenty enough for 8 spinners. In fact, it's almost enough for 8 SSD's. So assuming your motherboard slots are PCIe 2.0, then yes, the new card would remove the bottleneck.

Link to comment

Just looked up the mobo and it is indeed PCIe 2.0. Upgrading the card should get you back to the parity check speeds you were seeing before you fully populated the existing card.

 

But also as tr0910 says, a parity check can only move as fast as the slowest drive. The 500GB drive in your array is most certainly pulling down your average speed on parity checks. If you were to replace the card and remove the 500GB drive from the array, you average parity check speeds would likely increase to around 125MB/s or so. As an example my array is 5 of the Seagate ST3000DM001 drives and my average parity check speed is roughly 155MB/s.

 

EDIT: in fact you could probably get your speeds up to around 125ish without even replacing the card. If you removed the 500GB drive, put your fastest 6 drives on the mobo SATA ports, and your slowest 6 drives on the card, that may do it. It may not get you to 125 but it should certainly be faster than it is now.

Link to comment

r.e. the basic question ... "... Will upgrading controller help my parity check speeds? "  ==>  PROBABLY.

 

As already noted, the SAS2LP version of this card is a PCIe v2 x8 card, vs. a PCIe v1 x4 card that you have now.  This will double the lanes, and each of the lanes will have twice the bandwidth => so the new card can handle 4 times the data rate as your current card.

 

This will easily provide full throughput on any 8 spinning drives you might connect to it.

 

As also noted, however, the speed can never exceed the speed of the slowest "currently involved" drive.  Once your parity check gets past the 500GB point (thus no longer using your old 500GB drive), what kind of speed does the check hit at that point?

 

 

 

Link to comment

Took a look at your drive mix, and the 500GB drive isn't actually as much of a bottleneck as I initially thought.

 

Yes, it will slow down quicker than the other drives as it reaches the inner cylinders (as all drives do), but the platter density on this drive is the SAME as your WD20EARS units, so it's not going to be any slower at the initial phases of your parity check, since the sustained rate will be the same as the EARS drives.

 

FWIW, your 500GB Hitachi and 2TB WD20EARS units all use 500GB platters.

The HD204UI and ST2000DL003 drive have 667GB platters.

Your DT01ACA300 and ST3000DM001 drives have 1TB platters.

 

I'd put your ST3000DM001 drives and your SSD on the motherboard ports, and your current card should then have plenty of bandwidth for the other drives, even with the x4 interface, since their sustained data rates are limited by their lower areal density.

 

In other words, with your current mix of drives, if you put the units with 1TB platters on the motherboard ports, then until you update to higher density drives, the answer to your original question is actually No, an updated controller card isn't likely to make any appreciable difference [it may help a bit, as most cards don't actually achieve their theoretical maximum bandwidth].

 

You may still want to move to the newer card, however, if you're planning to upgrade your drives to higher capacity units -- just be sure your buy drives with 1TB platters as you do so.  [And if you do this, replace the 500GB drive first; then the EARS units]

 

 

 

 

Link to comment

It seems ever since i fully loaded my SAS card, my parity checks slowed down quite a bit.  I've often wondered if this is just the overloaded slot when through putting data for 8 drives.

 

Would upgrading to the newer version of the card help at all? 

 

I currently have a http://www.newegg.com/Product/Product.aspx?Item=9SIA5EM2JS7185&cm_re=AOC-SASLP-MV8-_-16-101-358-_-Product

 

and looking at a http://www.newegg.com/Product/Product.aspx?Item=9SIA5EM2JS7184

 

So the question is would the newer X8 card actually improve the speed of a parity check?    My last speed averaged 78, but have seen it in the lower 70's too.  It used to be in the 90s if not 100, but never gets above 80 anymore.

 

Current hardware / software in my signature is accurate.

 

Thanks all for any help.

 

Parity checks are gated by the slowest drive OR bandwidth restrictions. If the bandwidth restriction is 125 MB/sec, I would not say that is a serious limitation. Although a parity check can run faster than that initially, with 500G and even 2T drives in the array, they are more likely going to be the limiting factor the vast majority of time.

 

While I might agree than upgrading the controller may improve your speeds marginally, I would be very surprised if it was very much. And if that is the only reason for the upgrade, would likely spend my money elsewhere - like on drive cages (if I didn't have them), or a new drive. Besides bragging rights, a little faster parity check once a month is not so valuable.

Link to comment

The ST3000DM001 are pretty crappy drives, I'd be getting shot of those long before the Greens.

 

Crappy or not, they're 1TB/platter drives, so from the standpoint of parity check speed, they're the fastest drives in the system and all of the OTHERs are what are keeping the speeds relatively low.

 

As I noted above, the answer to the question (which I'd originally said was "Probably" until I looked in more detail at the specific drives and their areal densities)  is No, an updated controller isn't going to help with the mix of drives involved.    From that perspective, the ST3000DM001's are the "best: drives in the system  (where "best" isn't a measure of quality -- just of sustained data rate).

 

Link to comment

My parity checks on a 3TB array took a hair over 10 hours at an average speed of ~80MB/s (same as yours).

 

I recently replaced a WD20EADS (the last 2TB drive in the array) with a WD 3TB Red and parity checks now take a bit over 8 hours at an average speed of over 100MB/s

 

Last checked on Wed 01 Jul 2015 08:12:24 AM EDT (21 days ago), finding 0 errors. 
? Duration: 8 hours, 12 minutes, 22 seconds. Average speed: 101.6 MB/sec

 

@Garycase, some "EARS" drives were 667 platters, depending on time of purchase.

Link to comment

@Garycase, some "EARS" drives were 667 platters, depending on time of purchase.

 

True.  The 2TB EARS units used 500GB platters for drives ending in 00S8B1, 22S8B1, 00J2GB0, 42S0XB0, and 00J99B0;  and 667GB platters for drives ending in 00MVWB0, 07MVWB0, 22MVWB0, and 60MVWB0

 

Link to comment

Thanks to all for the input and assistance....

 

My parity checks on a 3TB array took a hair over 10 hours at an average speed of ~80MB/s (same as yours).

 

I recently replaced a WD20EADS (the last 2TB drive in the array) with a WD 3TB Red and parity checks now take a bit over 8 hours at an average speed of over 100MB/s

 

Last checked on Wed 01 Jul 2015 08:12:24 AM EDT (21 days ago), finding 0 errors. 
? Duration: 8 hours, 12 minutes, 22 seconds. Average speed: 101.6 MB/sec

 

@Garycase, some "EARS" drives were 667 platters, depending on time of purchase.

 

what type of controller are you running?

 

 

The ST3000DM001 are pretty crappy drives, I'd be getting shot of those long before the Greens.

 

I have since heard they don't last.  I am not buying more of this type, despite they are fast.

 

 

While I might agree than upgrading the controller may improve your speeds marginally, I would be very surprised if it was very much. And if that is the only reason for the upgrade, would likely spend my money elsewhere - like on drive cages (if I didn't have them), or a new drive. Besides bragging rights, a little faster parity check once a month is not so valuable.

 

I still run the original drive cages that came with the 1200 (plus one more factory cage) so I can hold 12 drives and that covers enough space for the foreseeable future

 

 

Link to comment

Thanks to all for the input and assistance....

 

My parity checks on a 3TB array took a hair over 10 hours at an average speed of ~80MB/s (same as yours).

 

I recently replaced a WD20EADS (the last 2TB drive in the array) with a WD 3TB Red and parity checks now take a bit over 8 hours at an average speed of over 100MB/s

 

Last checked on Wed 01 Jul 2015 08:12:24 AM EDT (21 days ago), finding 0 errors. 
? Duration: 8 hours, 12 minutes, 22 seconds. Average speed: 101.6 MB/sec

 

@Garycase, some "EARS" drives were 667 platters, depending on time of purchase.

 

what type of controller are you running?

 

none. all on mb ports.

Link to comment

 

 

True.  The 2TB EARS units used 500GB platters for drives ending in 00S8B1, 22S8B1, 00J2GB0, 42S0XB0, and 00J99B0;  and 667GB platters for drives ending in 00MVWB0, 07MVWB0, 22MVWB0, and 60MVWB0

 

EARS numbers

WDC_WD20EARS-00M_WD-WMAZA1640390

WDC_WD20EARS-00J_WD-WCAYY0217419

 

What am i missing?

Link to comment

You have to actually look at the drives to see the designators I noted above.

 

The info you posted shows the serial numbers and the first 3 characters of the designators -- from those you can tell that you have one with 667GB platters (the -00M unit) and one with 500GB platters (the -00J unit).

 

Link to comment

You have to actually look at the drives to see the designators I noted above.

 

The info you posted shows the serial numbers and the first 3 characters of the designators -- from those you can tell that you have one with 667GB platters (the -00M unit) and one with 500GB platters (the -00J unit).

 

In looking a parity check speeds, I would remark that comparing the densities of the platters and rotational speeds is not the end all be all.

 

Say I had a 500G, 1T, 2T, and 3T drive in my array. All identical densities and rotational speeds.

 

And I compare that to having all 3T drives in my array, with the same density and rotational speed as above.

 

The all 3T array is going to perform much faster on a parity check. In fact, the densities / rotational speed of the diverse drives could be somewhat higher, and I'd I'd still bet that the average parity check speed would still be quite a bit faster with the homogeneous drive size array.

 

Why? The lower number sectors are on the outer track of the disks. In one revolution, the amount of data read on an outer track is much larger than on the much shorter inner track. (Remember 2*pi*r?, if r is 3.5, C is 22, if r is 1.75, C is 11. So a drive is only 1/2 as fast on the innermost track as the outermost on a parity check). With diverse sizes, you are going to be on the inner tracks on each drive size at different times, and parity check speeds are gated by the slower disks at the time. With the homongeneous sizes, the inner tracks on ALL of the drives are going to be hit at the same time!

 

So in the first example, the 500G drive will start to slow down quickly, and be quite slow for the last 10-20% of the drive. After the 500G capacity is passed, if you were paying attention, you'd see a significant boost in speed. But your 1T drive is more that 1/2 gone, and it will be slowing down soon. By the time you get to the 800-900G mark, the 1T drive will be markedly slower. After 1T - turbochargers kick in again, and parity check speed will shoot up. But then the 2T drive will start to run out of gas. After the 2T capacity is reached, the 3T drives will remain, with will also slow down near the end.

 

With only 3T drives, the parity check will sail along until you get to maybe 2.5T mark or so, before starting to slow down significantly. It will have gone past the 500G, 1T, and 2T marks at a much faster speeds.

 

But the situation is worse for the diverse drive sizes. Over time, technology advances have allowed the platter densities to increase substantially. Older smaller drives tend to therefore be much slower in general, and their innermost sectors on those drives slow down the parity speed even more.

 

But faster parity check speeds have little to do with faster write speeds. Say, in the diverse array scenario with all 7200RPM drives, all of the smaller drives are full and you are using a brand new 7200RPM 3T drives for new files, and compare it to a homogeneous array doing I/O on a brand new 5400RPM 3T drive, the diverse array writes will be faster.

 

So if you have drives of various sizes in your array, your parity check speeds are impacted. If you have say 1 500G and 1 1T drive, and rest are larger, eliminating those two drive would have a significant speed increase. Every time you can eliminate all drives of a certain size, you are going to speed up your parity checks.

 

Again - parity check speed is not the end all be all stat in my mind. Gradually growing an array has a lot of advantages. Having all drives of the same size and vintage means that they all get old and risk failing at about the same time. And if you need more space and are out of slots, it is better to have a few small drives to upgrade to large ones. (I'd much rather swap out a 1T drive for a 6T one, than a 4T drive for a 6T one!) By having drives at a variety of ages and sizes, long term array management works better. But it works against parity check speed. Do I care? No. I'll live with a couple hours longer on my parity checks once a month!

Link to comment

... In looking a parity check speeds, I would remark that comparing the densities of the platters and rotational speeds is not the end all be all.

 

Of course not.  As I noted earlier, ALL drives slow down as they reach the inner cylinders.  [in commenting on the 500GB drive I noted "... Yes, it will slow down quicker than the other drives as it reaches the inner cylinders (as all drives do) ..."

 

From a parity check perspective, it's clearly better to not have mixed size drives, as you then only encounter this "inner cylinder penalty" one time.    But regardless of whether that's the case, it's also true that if you have multiple drives of the same size with different areal densities, the drives with smaller platters clearly limit the speed for all of the drives of that size ... so knowing the platter density is useful in determining which drives you might want to replace first.  Note that a 2TB drive with 1TB platters will be as fast on the inner cylinders as a 2TB drive with 500GB platters is on the outermost platters.    Basically, if you had an array with all 2TB drives and all-but-one had 1TB platters, but the outlier had 500GB platters, you could cut your parity check time in half by replacing that one drive.

 

In this specific case, the major limiting factor is the 500GB drive.    It's no slower from a sustained data rate perspective than the 2TB drives with the same platter densities; but it clearly reaches the inner cylinders quicker, and this slows everything down until the 500GB point is passed.    That's why I asked what speeds were being seen just after the 500GB point -- to see what a more "normal" speed for this system is after that drive was out of the picture.

 

I agree, however, that the parity check speed isn't really a big deal -- it's a once/month event, and it's not like you're "waiting" for it to finish.    Nevertheless, getting rid of drives with low areal density is a good idea from an overall system performance perspective, as your read and write speeds are impacted by the transfer rate ... which is directly related to the areal density.

 

But to focus on the question that was asked here -- upgrading the controller is not likely to make any appreciable difference in the parity check speeds for this array.    By far the best single improvement that could be made is to replace the 500TB drive with a new, larger drive -- and if that's going to be done, I'd definitely use a drive with 1TB platters (preferably a 3TB drive, since that's the size of the parity drive).  [The next drive I'd replace is the EARS unit with 500GB platters.]

 

 

 

Link to comment

Of course not.  As I noted earlier, ALL drives slow down as they reach the inner cylinders.  [in commenting on the 500GB drive I noted "... Yes, it will slow down quicker than the other drives as it reaches the inner cylinders (as all drives do) ..."

 

From a parity check perspective, it's clearly better to not have mixed size drives, as you then only encounter this "inner cylinder penalty" one time.

 

I think what may not have been clear is how profoundly a singleton sized disk in the array can slow things down, even if that disk is quite fast and dense.

 

Some observations ...

 

1. If you think you have a controller that is gating parity check performance, here is a not so scientific test. Start a parity check, wait a couple minutes, and check the speed. That is basically the fastest speed with all of the drives participating. (You can stop the check if you like). If that speed is well below your expected (e.g., below 75 MB/sec), a faster controller may very well help. But if it is at or close to the expected speed (e.g., 120 MB/sec), the controller is not a significant bottleneck.

 

2. If you only have one drive at a certain size, removing that drive can have a significant impact on overall parity check speed. Doesn't matter the size (500G, 1T, 2T, 3T, etc.) - a single drive of a certain size is going to slow down the parity check significantly.

 

3. If buying a new drive, it is best to pick a size equal to an existing drive in your array, and with specs at least as fast as the slowest disk of that size in your array.

 

Caveating again - high parity check speeds are not high on my list.

Link to comment

Caveating again - high parity check speeds are not high on my list.

 

I agree that parity check speed per se is not very important, but usually parity check = disk rebuild speed, and those I like to complete as soon as possible.

 

Sorry - can't really agree. A couple hours on a disk rebuild does not add any significant risk. If there is an error on a different disk while rebuilding - there will be an issue with the rebuild - and it is going to interfere whether the rebuild whether it takes 12 hours or 15.

Link to comment

Caveating again - high parity check speeds are not high on my list.

 

I agree that parity check speed per se is not very important, but usually parity check = disk rebuild speed, and those I like to complete as soon as possible.

 

Sorry - can't really agree. A couple hours on a disk rebuild does not add any significant risk. If there is an error on a different disk while rebuilding - there will be an issue with the rebuild - and it is going to interfere whether the rebuild whether it takes 12 hours or 15.

 

While I would agree that data rebuild does not need to be super-fast, it does need to be done timely. This is key for a system like unRAID based on single parity protection. The large disk drive report from google circa 2007, only found one strong predictor for disk failure and that was another disk's failure. Meaning if one disk dies, that is an indicator that other disks will fail. This is know as cohort failure. Defining the cohort can be complex. But in many unRAID arrays, it's pretty simple. The rest of the disks you installed. If you built an unRAID array of X disks, they are the cohort.  Drive failure in the cohort is your best predictor that failure is more likely in the rest of the cohort. BackBlaze's reports also indicate this.

 

Because of this, you'll want rebuilding to be timely. With unRAID and single parity, miss that window and you enter the world data loss. The SCO paper of 2014 shows that longer rebuild times increase the probability of data loss. Hence the need for erasure coding.

 

With a very large disk population, and various cohort definitions, I have learned that disk drive life is shorter in Denver than Omaha and many other oddities. Monitoring disk drive failures in real time can show geological and climatic events.

 

So, rebuild speed is important and longer rebuild time does increase risk. But I am not even sure that parity speed correlates to rebuild speed.

Link to comment

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.

Guest
Reply to this topic...

×   Pasted as rich text.   Restore formatting

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.