Will upgrading controller help my parity check speeds

[garycase]

... But I am not even sure that parity speed correlates to rebuild speed.

 

It will correlate pretty closely, since the basic I/O requirements are very close to the same ... the fundamental difference is that instead of reading all of the disks, you're reading all but one and writing that one.    One difference is that if the disk being rebuilt was a relatively slow drive (e.g. the 500GB drive in this specific system); and the new disk has an appreciably higher areal density (e.g. a 1TB/platter 3TB unit), then the rebuild will be notably faster than the parity checks were. 

 

... but I agree that doing the rebuild QUICKLY after the failure is far more important than the actual speed of the rebuild.

 

[SSD]

... but I agree that doing the rebuild QUICKLY after the failure is far more important than the actual speed of the rebuild.

 

This is getting off-topic of the thread. If a disk fails than unRaid becomes an unprotected set of disks, with the disk that failed being tenuously maintained by a combination of parity and every other disk in the array. Replacement should be prompt.

 

When buying new disks for your array, it is smart to preclear such disks immediately but not add them to your array until you actually need the space. If a failure does occur, you have a disk that can be used for reconstruction. If a disk is added to an array, even if empty, it can't be used in this way.

 

If you have no disk for reconstruction, get one quickly. I recommend a preclear cycle on any new disk, even one being used for reconstruction. Depending on the level of risk (e.g., other disks with SMART issues), I sometimes suggest powering down a server until the replacement disk is acquired and precleared.

 

So called dual parity will really shine in these types of cases as even if one disk has failed, you will remain protected from another failure.

[TODDLT]

To all who have been part of this discussion, it has been educational so thank you for the time.  A few notes and questions as i decide what, if, and in what order I will do things now.

 

1.  Just mentioning I do keep a precleared warm spare (3TB) in the box.  Multiple drive failures worry me so I'd rater not insert UPS into the timeline for a repair.  Having said that, I really hope we get to two drive parity protection one of these days.  Of course I'd probably just keep a cold spare on the shelf instead of in the box...  But moving along from here.

 

2.  My 11 drive array is 21.5 TB and I have 9 free overall.  I will not need more capacity any time soon. 

2a.  Of the 11 data discs 3 have specific purposes that don't mingle with the others  IE each has their own user share.

      * One of them is a 32 GB SSD.  I got this because I wanted very fast access that never went to sleep for a relatively small fixed amount of data. 

      * One of them is the 500GB drive.  This is not just a case of a small drive lieing around that got thrown in for some more space.  Most of my server houses media, large files.  This one drive houses boatloads of small personal use files that I didn't want scattered across the array.  For one reason I keep it backed up routinely in more than one location so when the backups run only one drive has to spin up.  Having said all of that, this is a newer drive and not even 50% full.  Newer is relative, probably at least a couple years old now but it still feels a little wasteful to replace it just to speed up a parity check.  If I was to replace it, I think I'd serious consider a SSD for this one use since it doesn't need to be a larger drive.  It will be a long time before it needs to be more than 500 GB (Thoughts?)

2b.  2 of the 8 remaining have one use and the other 6 another use.  So that is two groups of user shares that segregate the 8 drives into 2 and 6.

    * One of the 6 drive group is completely empty and after this conversation I think I'll go with the gut feeling I've had for a while and just remove it until the capacity is needed.  This was a case of a drive that came free and just got stuck in the array before the space was needed.  Actually it is one of the EARS, but unfortunately it is the 00M (667) unit that is empty.  So the next question is if it's worth removing this from the array and then replacing the 500 platter drive with the 667.  (any thoughts? worth the effort multi stage effort?)

 

Generally I agree with you on the parity speed not exactly being a high priority.  I have watched my initial parity speed of 110 ish drop to 70 and below as the array ha s grown and would like to see it get back up above 100.  It would have been worth a single $100 card, but not worth the cost of replacing 3 or 4 drives for speed alone.  This has been extremely helpful because as drives do eventually get replaced, I can plan better.  The parity speeds do seem to have slowed down as the card has filled up and so that is why I was leaning that way, but it seems there is a lot more to consider.  I will also note based on the above comments, my speeds are slow right from the start and don't waiver up or down much throughout the check (though I don't exactly sit here for 12 hours).  I saw a logical argument that this could point to the card being an issue.

 

Here is my thoughts on what to do:

1. Remove the empty EARS drive from the array.  I think I have to do a new parity calc when this is done anyway so I'll find out if it helps right away.  (is this process any easier in version 6?)  This will take one drive off the controller, so I wont really know if its the card bus capacity or the individual drive, assuming things actually improve.

2. Replace the 003 EARS with the 00M, then check out the parity speed.

3. Depending on what is said above about my parity checks being fairly consistent in speed, the card may be next.

4. If this doesn't appreciably help and is still worth the investment, would the next step to be replacing the 500 GB drive with a SSD?  an SSD should never be the bottleneck right?

 

Anyway, thanks for the continued discussion.

[garycase]

... I really hope we get to two drive parity protection one of these days.  Of course I'd probably just keep a cold spare on the shelf instead of in the box...  But moving along from here.

 

Coming soon => Tom just announced v6.1 RC2 and in that announcement noted that after v6.1 stable is released, the key upgrade that will be in the next stable release (v6.2) is dual fault tolerance 

 

 

As for your other thoughts ...

 

=> Yes, replacing the 500GB drive with a 500GB SSD would give you a NICE improvement in your parity check speeds.  Clearly an SSD does not have an "inner cylinder slowdown" issue 

 

=>  There's really no advantage to removing the EARS unit from your array ... it's very unlikely to make any difference in your parity check speeds.  You have other 2TB drives that are slower, so they're still going to be the key speed limiters.

 

=>  If you DO remove the EARS drive (as a bit of a test to see if it helps to have one less drive sharing the controller's bandwidth), I wouldn't make any other changes at the same until you see what it does to the parity check speeds.  If parity check speed does improve, then that does indeed imply that your controller was a bit of a bottleneck.    After you've done that, if you want to swap the 667GB/platter unit for the 500GB/platter unit you could do that, although that's a marginal improvement so I'm not sure I'd bother.

 

=>  Note that a MUCH easier way to test whether or not your controller's bandwidth is a potential bottleneck here is to simply move a few drives to your motherboard controllers.    You only have 9 drives in the array ... and the motherboard has 6 SATA ports => so use them.    You don't need to make ANY configuration changes in the array ... just move the SATA connections to the motherboard ports for a few drives.

 

[TODDLT]

 

Coming soon => Tom just announced v6.1 RC2 and in that announcement noted that after v6.1 stable is released, the key upgrade that will be in the next stable release (v6.2) is dual fault tolerance 

 

 

As for your other thoughts ...

 

=> Yes, replacing the 500GB drive with a 500GB SSD would give you a NICE improvement in your parity check speeds.  Clearly an SSD does not have an "inner cylinder slowdown" issue 

 

=>  Note that a MUCH easier way to test whether or not your controller's bandwidth is a potential bottleneck here is to simply move a few drives to your motherboard controllers.    You only have 9 drives in the array ... and the motherboard has 6 SATA ports => so use them.    You don't need to make ANY configuration changes in the array ... just move the SATA connections to the motherboard ports for a few drives.

 

I actually have 13 drives connected in my case.  1 parity, 11 data drives, and 1 connected warm spare.  4 are on the MB, 8 on the SAS, and 1 is on a separate PCI controller card. 

Then 2 of the MB ports are connected to ESATA ports and really only used for pre-clearing a drive before putting it in the case.  Acknowledged that this is not a good use of the MB ports but the ports have a female SATA connected and I can't plug the female SAS cable to that so this was the only way to make it all work.  Having said that, Maybe i'll get an inexpensive PCI controller card for the external ports. 

 

For the moment I disconnected them and took 2 drives off the SAS card and put them on the MB for tonight.

 

So I started a parity check:

- It started in the upper 50's to low 60's until somewhere around 35 GB It jumped into the 90's to 100 MB/sec  (My single 32 GB SSD is connected to that seperate PCI controller card.  This is the only connection to that I can find to the change in speed). 

- Around 100 GB it slowed back down to upper 70's to low 80's.  and has hung there through 200... running now. 

- At 499 GB it was at 65 MB's

- At 514 it was at 85 and kept bouncing in the upper 70's low 80's

 

My parity checks have been averaging in the 70's (78 last time), so I'm not seeing much change.  The short lived run up at 35 seemed peculiar.  I guess it doesn't seem to have changed anything.

 

[garycase]

Agree it doesn't seem to have made any difference -- which confirms that your 8-port controller card isn't a bottleneck.

 

The early jump is interesting.  You indicated you have a PCI controller card ... do you mean PCI or PCIe ??  If it's truly a PCI card, what are the specs?  i.e. which version of the PCI spec does it support; what SATA version does it support; etc.    Is the 32GB SSD the only thing connected to that card?    What's the make/model of the SSD?

 

 

[TODDLT]

Agree it doesn't seem to have made any difference -- which confirms that your 8-port controller card isn't a bottleneck.

 

The early jump is interesting.  You indicated you have a PCI controller card ... do you mean PCI or PCIe ??  If it's truly a PCI card, what are the specs?  i.e. which version of the PCI spec does it support; what SATA version does it support; etc.    Is the 32GB SSD the only thing connected to that card?    What's the make/model of the SSD?

 

The card is a PCI standard card, yes that old.  it only has one SATA internal and one eSATA external but only the one drive plugged into it.  I'm pretty sure 95% it's SATA 2 compliant.

The drive is 1 - 32 GB Patriot Torqx 2 SSD.  http://www.newegg.com/Product/Product.aspx?Item=N82E16820220580&cm_re=Patriot_Torqx_2-_-20-220-580-_-Product

 

Do you (or anyone else) know of a Male / Male SATA adapter?  it would allow me to connect my two eSATA ports to the SAS cables inside the case.  I've been digging around for one and haven't come up with it.  Newegg or Monoprice. 

[TODDLT]

one other thought.  Is there a way to speed test each drive and see where the real bottlenecks are?

[garycase]

Not clear why you need an adapter to connect to the eSATA ports => can't you simply unplug the connections from the eSATA backplate from the motherboard ports and then use those?

 

[garycase]

r.e. the PCI card and 32GB SSD.    How old is this SSD?  A PCI card would indeed be a bottleneck, but shouldn't slow things down as much as you're seeing.  But if it's an old SSD that's been written to a lot it could easily be beyond it's reasonable life, which would slow it own a lot.    You've got quite a few variables involved here, so it's difficult to definitely say what's causing the relatively slow speeds.    Outside of the possibility of the 32GB SSD, it's fairly clear you primary limiting factor (from a parity check speed perspective) is the 500GB drive; but the even slow speed of the SSD is a bit perplexing.  If possible, I'd at least plug it in to a motherboard port if you can use one of the ports currently used for the eSATA connections.

 

[JorgeB]

In my experience a fully loaded SASLP will limit you parity check speed to around 70Mb/s, but you do appear to have other bottlenecks on your system.

 

one other thought.  Is there a way to speed test each drive and see where the real bottlenecks are?

 

You can run diskspeed to test each disk speed.

 

You can also run tunables tester although it gives better results on a non bus limited system.

[mr-hexen]

SATA to eSATA bracket. http://www.monoprice.com/Product?c_id=104&cp_id=10407&cs_id=1040707&p_id=7638&seq=1&format=2

 

Some of them even have provisions for power.

 

@Garycase, i think his PCI Card has a built in eSATA port?

[garycase]

@Garycase, i think his PCI Card has a built in eSATA port?

 

Possible, but I don't think so.    The specs for the motherboard show 6 SATA ports ... yet he refers to 4 on-board ports and 2 eSATA ports.    I suspect there's simply a rear panel bracket that's connected to 2 of the motherboard ports.

 

[TODDLT]

SATA to eSATA bracket. http://www.monoprice.com/Product?c_id=104&cp_id=10407&cs_id=1040707&p_id=7638&seq=1&format=2

 

Some of them even have provisions for power.

 

@Garycase, i think his PCI Card has a built in eSATA port?

 

Yes, I have a bracket with 2 esata ports and 1 - 4 pin IDE power port.  As you can see, the cables are meant to plug into the MB so they don't meet up with the SAS cables.  They make brackets you can just plug the cables from the card into, but with the 4 pin power port, its a convenient setup to pre-clear drives before bothering to open the case.

 

http://www.newegg.com/Product/Product.aspx?Item=9SIA39V2UW6172&cm_re=esata_bracket_power-_-9SIA39V2UW6172-_-Product

 

 

[garycase]

I have that same bracket on most of my systems.  But for testing purposes, you could simply unplug the connections from the motherboard and then use those 2 motherboard ports to connect a couple drives to (notably your 32GB SSD).

 

[TODDLT]

I have that same bracket on most of my systems.  But for testing purposes, you could simply unplug the connections from the motherboard and then use those 2 motherboard ports to connect a couple drives to (notably your 32GB SSD).

 

I did this last night ,but not with the 32 GB. 

[TODDLT]

 

You can run diskspeed to test each disk speed.

 

 

Thanks, great utility

 

/dev/sdb (Parity): 152 MB/sec avg

/dev/sdc (Disk 2): 108 MB/sec avg  Disk 2 Hitachi_HDS721050CLA362_JP1572JE0VNS3K (sdc)

/dev/sdd (Disk 3): 157 MB/sec avg  Disk 3 ST3000DM001-1CH166_W1F29KYY (sdd)

/dev/sde (Disk 4): 81 MB/sec avg  Disk 4 WDC_WD20EARS-00J2GB0_WD-WCAYY0217419 (sde)

/dev/sdf (Disk 7): 104 MB/sec avg  Disk 7 SAMSUNG_HD204UI_S2H7J1PB500857 (sdf)

/dev/sdg (Disk : 107 MB/sec avg  Disk 8 ST2000DL003-9VT166_6YD0EZXA (sdg)

/dev/sdh (Disk 1): 96 MB/sec avg  Disk 1 Patriot_Torqx_2_32GB_SSD_BA140721090800019963 (sdh)

/dev/sdi: 149 MB/sec avg                  sdi TOSHIBA_DT01ACA300_X3G68PJKS (sdi) 32 C     (THIS IS MY WARM SPARE)

/dev/sdj (Disk 11): 90 MB/sec avg  Disk 11 WDC_WD20EARS-00MVWB0_WD-WMAZA1640390 (sdj)

/dev/sdk (Disk 10): 100 MB/sec avg  Disk 10 Hitachi_HDS722020ALA330_JK1170YAGS9G8T (sdk)

/dev/sdl (Disk 9): 157 MB/sec avg  Disk 9 ST3000DM001-1ER166_Z500MVPC (sdl)

/dev/sdm (Disk 6): 103 MB/sec avg  Disk 6 SAMSUNG_HD204UI_S2H7J1PB500855 (sdm)

/dev/sdn (Disk 5): 163 MB/sec avg  Disk 5 ST3000DM001-9YN166_S1F0PM8K (sdn)

 

Certainly a lot there.  I think this gives me some direction over time but too many drives at play to bother replacing just to increase parity.  I will get the SSD on the right card and do something about the low EARS now.  The rest will wait.

 

thanks again.

 

[mr-hexen]

as garycase and I (possibly others) predicted, the EARS are the dogs in this setup causing the slower parity times.

[TODDLT]

a tangent but related.  can you preclear a drive through a USB3 SATA dock?  Would the speed of the preclear be diminished?  That could eliminate my need for the esata ports in back. 

[TODDLT]

If I do decide to dump the EARS drives which are getting up in age anyway, which of these would you go with?  the Seagate looks surprisingly inexpensive for what they are calling an Enterprise drive.  It does however, only have a 1yr warranty.

 

The Toshiba is a pretty good deal for a 3TB

http://www.newegg.com/Product/Product.aspx?Item=N82E16822149396

 

the Seagate looks surprisingly inexpensive for what they are calling an Enterprise drive.  I'm confused on the warranty though.  It says 3 years on the main page and 1 year in the pecs.

http://www.newegg.com/Product/Product.aspx?Item=9SIA5AD2NC0924

 

Hitachi, it's been a long while since I bought any.

http://www.newegg.com/Product/Product.aspx?Item=N82E16822145911

 

[TODDLT]

Any help on these two questions (above)?

[garycase]

Of the 3 drives you listed, I'd go with the HGST NAS unit.    They're very well thought of and have an excellent reliability record.

 

As for pre-clearing through a USB3 dock => I've not tried that, but it should work okay with one BIG caveat ... SMART data generally isn't accessible through USB ... and this is a key thing you want to look at in the pre-clear report, so I wouldn't do it.

 

 

 

 

[TODDLT]

Of the 3 drives you listed, I'd go with the HGST NAS unit.    They're very well thought of and have an excellent reliability record.

 

As for pre-clearing through a USB3 dock => I've not tried that, but it should work okay with one BIG caveat ... SMART data generally isn't accessible through USB ... and this is a key thing you want to look at in the pre-clear report, so I wouldn't do it.

 

thanks

[SSD]

If I do decide to dump the EARS drives which are getting up in age anyway, which of these would you go with?  the Seagate looks surprisingly inexpensive for what they are calling an Enterprise drive.  It does however, only have a 1yr warranty.

 

The Toshiba is a pretty good deal for a 3TB

http://www.newegg.com/Product/Product.aspx?Item=N82E16822149396

 

the Seagate looks surprisingly inexpensive for what they are calling an Enterprise drive.  I'm confused on the warranty though.  It says 3 years on the main page and 1 year in the pecs.

http://www.newegg.com/Product/Product.aspx?Item=9SIA5AD2NC0924

 

Hitachi, it's been a long while since I bought any.

http://www.newegg.com/Product/Product.aspx?Item=N82E16822145911

 

Although I love the HGST drives and think they are great, I would have a very hard time paying $135 for a 3T. That is $45 per terabyte! Ouch.

 

Although I don't have any direct experience, I think I'd have to look seriously at that Toshiba 3T drive. They get relatively good ratings.

 

But I think 3T is a little small these days. I'd look at 4T and 5T drives.

 

You might also look at THESE TOSHIBA 5T DRIVES. At $29/T they are a good value. It's what I would buy today if I needed a drive.

[garycase]

If price/TB is your primary consideration the Toshiba's are indeed good values ... $30/TB for the 3TB, $29/TB for a 5TB.

 

Note, of course, that if you buy a 5TB, you'll have to use it as parity (i.e. replace the parity drive) and then use the old parity drive to replace the EARS unit.    [Or simply buy 2 5TB drives and repurpose the old parity drive.]

 

Personally, $/TB isn't a significant consideration when I buy drives.