i'm not home right now but will post diagnostics as soon as i get home if needed.

 

the parity check goes only as fast as the slowest drive right? i replaced one of my devices which was reporting ~69 MB/sec in one of the diskspeed scripts posted on here (which was also the average speed of my last few parity checks since upgrading to a 4TB parity) since i thought that was the reason for the slow checks. now all my drives (that are part of the array anyway) are reporting 90+ MB/sec avg reads, is it safe to assume that my parity check should also run at the same or similar speeds? specs are in sig.

 

#diskspeed.sh

diskspeed.sh for UNRAID, version 2.6.1
By John Bartlett. Support board @ limetech: http://goo.gl/ysJeYV

Warning: Files in the array are open. Please refer to /tmp/lsof.txt for a list
/dev/sdb (Disk 3): 96 MB/sec avg
/dev/sdc (Disk : 98 MB/sec avg
/dev/sdd (Disk 13): 110 MB/sec avg
/dev/sde (Disk 6): 120 MB/sec avg
/dev/sdf (Disk 1): 112 MB/sec avg
/dev/sdg (Disk 11): 122 MB/sec avg
/dev/sdh (Disk 10): 116 MB/sec avg
/dev/sdi (Disk 7): 92 MB/sec avg
/dev/sdj (Cache): 134 MB/sec avg
/dev/sdk (Disk 4): 120 MB/sec avg
/dev/sdl (Disk 9): 123 MB/sec avg
/dev/sdm (Disk 2): 124 MB/sec avg
/dev/sdn: 64 MB/sec avg
/dev/sdo (Disk 12): 123 MB/sec avg
/dev/sdp (Disk 5): 113 MB/sec avg
/dev/sdq: 69 MB/sec avg
/dev/sdr (Parity): 151 MB/sec avg

controller?

controller?

 

I'd have to double check with drives are connected to what, but I have these cards installed.

 

Silicon Image SIL3132

ROSEWILL|RC-201 2SATA RTL

SUPERMICRO AOC-SASLP-MV8

Mixed size drives?

Yessir. Mostly 2TB drives but i have a couple 1TB and also 2x4Tb (one is parity, other is part of the array)

I'd have to double check with drives are connected to what, but I have these cards installed.

 

Silicon Image SIL3132

ROSEWILL|RC-201 2SATA RTL

SUPERMICRO AOC-SASLP-MV8

 

That's very important, eg, the SASLP fully loaded maxes out at 80MB/s (or 70MB/s without optimal tunables)

I'd have to double check with drives are connected to what, but I have these cards installed.

 

Silicon Image SIL3132

ROSEWILL|RC-201 2SATA RTL

SUPERMICRO AOC-SASLP-MV8

 

That's very important, eg, the SASLP fully loaded maxes out at 80MB/s (or 70MB/s without optimal tunables)

 

are you saying rearranging my drives would help? or basically if all those drives are spinning then they're basically only running at 80 MB/sec no matter what?

 

if 80MB/sec is the max, i'm not too far off at 69MB/sec.. might as well keep my old drive in for now if that's the case.

You can try to connecting the disks in the best possible way to take advantage of each controller bandwitdh, in your case with 14 disks it would be:

 

6 x Onboard SATA

1 x RC-201

1 x Sil3132 (assuming this is another controller)

4 x SASLP

 

This should permit a parity check speed of 100MB/s+.

You can try to connecting the disks in the best possible way to take advantage of each controller bandwitdh, in your case with 14 disks it would be:

 

6 x Onboard SATA

1 x RC-201

1 x Sil3132 (assuming this is another controller)

4 x SASLP

 

This should permit a parity check speed of 100MB/s+.

 

i have 17 disks total (14 array+1 parity+2 that are not in use, just precleared and waiting.. for, not sure actually, they're too small for anything good.. 320G and 500G). so the SASLP is not yet 100% full, i think i'm currently only using 6 of 8.

I'd assume your couple of 1TB drives are slowing you down.

 

Monitor the speed during a parity check and you'll find that the parity check speed is low towards the end of the first 1TB of the check and then picks up quite a bit right after the 1TB mark.

actually checking the history of my parity checks, even with 2TB parity i was averaging like 48 MB/sec so i guess it was always horrible and i just never noticed.

Without more specifics on the drives and the controllers, it's hard to say for sure ... but in addition to the slowdowns caused by mixed drives which are reaching their slowest inner cylinders at varying times during your checks, it seems you also have controller-based bottlenecks -- not only with the SASLP, but with the other two controllers as well -- I assume one of those is a PCI card and the other is PCIe x1 ... is that correct? 

 

The areal density of your disks is also significant -- your smaller disks may be older and have 500MB (or even smaller) platters.

 

As you grow your array and replace the slower drives with new 4TB drives you'll eventually see a nice improvement (note that these jumps in speed won't necessarily happen with every drive replacement -- they're more likely to happen when you've just eliminated a drive size ... e.g. after you're replaced your last 1TB drive).

 

Without more specifics on the drives and the controllers, it's hard to say for sure ... but in addition to the slowdowns caused by mixed drives which are reaching their slowest inner cylinders at varying times during your checks, it seems you also have controller-based bottlenecks -- not only with the SASLP, but with the other two controllers as well -- I assume one of those is a PCI card and the other is PCIe x1 ... is that correct?

 

 

yes this is correct.

 

for now i'll keep an eye on this as i remove and replace drives down the road. currently it's about 16 hours to complete a parity check which is long but really at once a month it doesn't bother me too much as long as it works. can i expect this time to stay around the same after adding a few more drives? or can we be talking day+ long parity checks? if that's the case, maybe it'd be worth it to replace my 2TB's with all 4's before using up all the empty slots.

 

i don't currently have any plans to replace my current MB/cpu/controllers for a few more years unless something breaks.

Didn't even noticed that the ROSEWILL|RC-201 is PCI  , you'll definitely don't want more than one disk there, with 2 disks it will limit it you to ~50MB/s max.

... can i expect this time to stay around the same after adding a few more drives?

 

Yes, as long as you don't add yet-another size drive to the mix.  I'd recommend that whenever you add another drive you use another 4TB drive ... eventually moving towards an all-4TB array.   

 

There are basically two things that are impacting your parity check times now:

 

(1)  The mixed drive sizes and probably mixed areal densities.  As long as you only use 4TB drives when adding to the array, you'll eventually move away from that issue.  Your parity check times based on this factor won't change by adding more 4TB drives -- and will in fact improve if/when you eliminate one of the smaller sizes (e.g. when you replace the last 1TB drive)

 

(2)  Controller bandwidth restrictions.  That's why I asked for confirmation on the PCI controller.  As Johnnie noted, if you use 2 drives on this controller, that alone is a bandwidth limiter.  If you MUST attach two drives to it; use your smallest drives, so when your parity check gets past that size this controller is no longer restricting the speed.  Your other controllers are also limiting your speeds somewhat, but definitely not as much as the PCI unit.

 

 

well you guys nailed it right on the head as usual. after i hit the 2TB mark, i'm seeing much improved speeds of about 100+ MB/sec. I guess no matter what when i fill up all slots i'll eventually have the same issue no matter if all drives are the same size because of the SATA controllers/mobo I'm using. it sounds like i'm possibly better of leaving those 2TB drives on whatever controller they're on so at least the back half of the parity check will be fast instead of the whole 4TB going at a snail's pace.

That's basically correct.  If you had all 4TB drives you'd get well over 100MB/s IF there were no controller bottlenecks -- but that's not the case with your system.  So basically the optimal setup would be to consider the bandwidth of each of your controllers; then connect disks to them in a way that as the parity check proceeds past your smaller drives the bandwidth available to the 4TB drives is as much as possible (given the # of 4TB drives you have).

 

For example, the SASLP is good for about 640MB/s total bandwidth, so if you have more than 4 modern 1TB/platter drives connected to it, their bandwidth begins to be limited ... to as low as about 80MB/s each if you have 8 drives connected.

 

The motherboard ports are SATA-3, so they aren't limiting any spinners (they would slightly restrict SSD speeds).

 

The PCIe x2 card can sustain up to 250MB/s, so while it will limit the bandwidth on the outer cylinders of a modern 1TB/platter drive, it's still fast enough that it's not something to be concerned with.

 

The PCI card is your biggest bottleneck => I would only connect one drive to it if possible; and if you need to use it for 2 drives, be sure one of them is the smallest drive in the array, so during parity checks it will drop out as soon as possible (until that happens, this will be the limiting factor for your parity check speed).

 

Basically, with up to 10 4TB drives, you can at least get very good speeds beyond the 2TB point by connecting 6 drives to the motherboard and 4 to the SASLP.    I'd probably connect the next 2 to the PCIe card;  the next 2 to the SASLP; and if you get beyond 14 4TB drives you won't have any choice but to use the last 2 SASLP ports and the PCI card.

 

But with only 2 4TB drives at the moment, it'll likely be a while before you get past 10 of them -- so I'd just be sure you use motherboard ports, followed by SASLP ports for those drives.

 

 

This may sound like a stupid question but have you got any dockers running when you do a parity check? Because some dockers will have a rather bad effect on parity.

 

 

Sent from my iPhone using Tapatalk

Not just Dockers => ANY read or write activity on the array during a parity check will slow things down appreciably ... both the parity check and the read/write will be MUCH slower than normal during the period when both are happening simultaneously.

 

... with respect to Dockers; note that most Dockers use the cache drive for their data -- and read/writes to the cache have NO impact on a parity check.