Staggering HDDs across different controllers?

[jason4207]

Another piece of advice (which I originally got from someone here who really knows his stuff, but whose name I can't recall just now) is to stagger the disks across the different controllers, i.e., to not have consecutively-numbered disks on the same controller. I have examples where that has side-stepped some internal bottleneck, for a substantial improvement in parity performance.

 

The worst bottleneck from the perspective of unRAID is the PCI bus, however. It becomes a factor already with 3 or 4 disks, if I remember the calculations correctly, and there's no overcoming that.

 

 

I found this quote from an older thread. 

 

Is there any truth to this?

 

If so, once I get my Promise Sata150 TX4 PCI I want to do this:

 

Intel = ICH10R 6-port

JM = JMicron on-board 2-port

SI = Silicon Image PCIex1 2-port

TX4 = Promise TX4 PCI 4-port (2 used)

 

Intel-1: 2TB Samsung Parity

JM-1: 2TB Samsung disk 1

Intel-2: 1TB WD disk 2

SI-1: 1TB WD disk 3

Intel-3: 1TB WD disk 4

TX4-1: 1TB WD disk 5

Intel-4: 1TB WD disk 6

JM-2: 1TB WD disk 7

Intel-5: 640GB WD disk 8

SI-2: 640GB WD disk 9

Intel-6: 640GB WD cache

TX4-2: 640GB WD disk 11

 

I will have to look into how to swap my disk order, but iirc it can be done.

 

What do you guys think?  Does anyone have any experience that this will improve parity performance?

 

 

[Rajahal]

Yes, there is truth to this, however, it may not affect your situation much since you have only 2 drives on the PCI bus.  It is still worth trying, though, since it certainly can't hurt performance.

 

The order of your drives as you detailed it above is probably fine, though you may want to swap your 1 TB WD (disk 5) with one of the 640 GB drives (disks 8 or 9), assuming that the 640 GB drives won't be accessed as often.

 

Basically, you have 10 'fast' ports (all your onboard ports, plus the SI PCIe card ports) and 4 'slow' ports (the PCI-based TX4).

 

The key is to keep your parity, cache, and most-used drives off the PCI bus.  In your case, that means placing only your least used drives on your TX4 card.  Generally speaking, your least used drives will be your smallest drives, but this depends on your organizational scheme.  If you do nightly backups to your smallest drive, then this obviously doesn't apply to you.

 

You may not notice the difference at first or at all since you use a cache drive.  The main benefit comes from accessing multiple drives simultaneously.  Since all of your writes to the server first go to your cache drive, you may not notice the difference at all.  However, by placing your smallest disks on the TX4 card you can expect your parity checks to speed up a bit, since the 'slow' parity check times involving the saturated PCI bus will be minimized.

 

The procedure is discussed and detailed here:

Move Largest and Fastest Data Drives Off PCI Bus

[Rajahal]

Oh, and another tip is to use the user share's 'included/excluded' disks options to pick and choose where you want your data stored.  For example, if you access your 'movies' share a lot, set that share to include your fastest disks and exclude your slowest disks.  On the other hand, if you don't access your 'pictures' share very often, you could limit that share to include only your slowest disks and restrict the rest.

 

You will want to do this after physically rearranging your disks.  You will also need to move data manually to match your 'includes/excludes' above, since unRAID will apply those rules only to new incoming data, not to data already stored on your server.

[jason4207]

Oh, and another tip is to use the user share's 'included/excluded' disks options to pick and choose where you want your data stored.  For example, if you access your 'movies' share a lot, set that share to include your fastest disks and exclude your slowest disks.  On the other hand, if you don't access your 'pictures' share very often, you could limit that share to include only your slowest disks and restrict the rest.

 

You will want to do this after physically rearranging your disks.  You will also need to move data manually to match your 'includes/excludes' above, since unRAID will apply those rules only to new incoming data, not to data already stored on your server.

 

You read my mind on reorganizing my data, and answered a question I was going to ask in advance! 

 

I was hoping unRAID would automatically reorganize my data if I setup include/exclude how I want it, but was pretty sure I'd have to do it manually.

 

Intel-1: 2TB Samsung Parity

JM-1: 2TB Samsung disk 1

Intel-2: 1TB WD disk 2

SI-1: 1TB WD disk 3

Intel-3: 1TB WD disk 4

TX4-1: 640GB WD disk 5

Intel-4: 1TB WD disk 6

JM-2: 1TB WD disk 7

Intel-5: 1TB WD WD disk 8

SI-2: 640GB WD disk 9

Intel-6: 640GB WD cache

TX4-2: 640GB WD disk 11

 

This look good?

 

Thanks!

[WeeboTech]

I might even consider putting the cache drive on the TX4.

 

The Promise TX4 is no slouch of a controller. If it's on a PCI Bus that is PC2.3 and 66MHZ, it can be a really good performer.  I was amazed at a recent test I did with a Samsung Spinpoint F3 1TB I did today.

 

Considering how fast the WD640 could possibly read & write. Having the cache on that controller alone may still provide adequate performance.

 

If it were me I might even consider a purchase that allowed me to merge the 640's to one spindle, then sell them off on eBay.

[jason4207]

 

 

If it were me I might even consider a purchase that allowed me to merge the 640's to one spindle, then sell them off on eBay.

 

In due time.  I'll get 2TB drives as i need them.  The 2TB drives are only getting cheaper, but the 640GB AAKS are still worth $40-$50.  I'm plan to max out at 12 drives and just upgrade drives as needed.

[terrastrife]

the TX4 _should_ be running at 66MHz, given it is on an unsahred PCI slot (the ICH10 is PCI2.3 compliant). this is plenty of bus bandwidth for 2 HDDs

[Kaygee]

My JM Micron PCI-e JMB363 isnt stunningly fast (60MB/s), a 66Mhz TX4 will certainly keep up with it, with just two drives hooked up the TX4 might well be faster.

 

Worth experimenting with. Post up how you get on I'm curious to see if controller stripping has any effect.                     

[Rajahal]

You read my mind on reorganizing my data, and answered a question I was going to ask in advance!  

 

Welcome to Psychic Burger, where we make your burger before you order it!  You left your wallet in your car, but you have a 5 dollar bill in your left pocket.  And don't worry about that milkshake you are about to spill...

 

Intel-1: 2TB Samsung Parity

JM-1: 2TB Samsung disk 1

Intel-2: 1TB WD disk 2

SI-1: 1TB WD disk 3

Intel-3: 1TB WD disk 4

TX4-1: 640GB WD disk 5

Intel-4: 1TB WD disk 6

JM-2: 1TB WD disk 7

Intel-5: 1TB WD WD disk 8

SI-2: 640GB WD disk 9

Intel-6: 640GB WD cache

TX4-2: 640GB WD disk 11

 

This look good?

 

Thanks!

Yep, looks good.  I agree with WeeboTech that you could put the cache drive on the TX4, but I don't see it as necessary, since you've got spare slots anyway.  Will you be adding disks or upgrading disks in the future?  If you are upgrading, just upgrade the non-TX4 disks first.  If you are adding, then one-by-one migrate the 640 GB (or other smallest) disks onto the TX4 so that your largest disks can all reside on the non-TX4 slots.  I would keep your cache drive off the TX4 as long as you have other non-TX4 slots open, but if you run out, you can move it.  Your nightly data transfers from the cache drive to the protected array will be slow if the data disk being used also happens to be on the TX4, but this should be occurring while you sleep anyway, so it shouldn't matter much.  Your writes to the cache drive might be slightly slower, depending on the speed of your cache drive.

[WeeboTech]

I would keep your cache drive off the TX4 as long as you have other non-TX4 slots open, but if you run out, you can move it.

 

Agree, I must have missed something. In any case the TX4 with the right hard drive is capable of reaching 80MB/s on a PCI Bus. I think that's fast enough for a cache drive. No?

 

On a PCI-66MHZ bus, it's capable of hitting 150Mb/s. In fact it's capable of writing to two drives at 80MB/s simultaneously.

 

The TX4 has a FIFO on the controller so it's quite a capable performer.

 

See how fast you can do a dd write to a 640gb drive on the motherboard, then test it on the TX4 (alone) and compare results.

[jason4207]

Finally finished getting her setup the way I want...well close...I still have a 1TB and a 640GB on the TX4 instead of 2 640GB drives on there.  And I ended up losing all the data on a 640Gb drive during all the swapping around, but fortunately it's not anything personal and it's easily replaceable.

 

I ran a parity-check today, and this is what I'm getting:

 

It starts out at ~34MB/s, and climbs to ~44MB/s fairly quickly.  I missed the 640GB and 1TB transistions, but at 85% complete it was doing 67MB/s.  At the end syslog said it took 9.74hrs at an average speed of 55.7MB/s.  Without the TX4 I think I started out around 55-58MB/s and maxed out around 89MB/s, but I didn't take good notes to do a decent before & after comparison.  Parity checks are definitely faster w/o it...I wonder if it's running at 66MHz?  Anyone know how to check?  It's still plenty fast considering everything.

 

I'm in good shape now as far as ports and speed, and I've maxed out this case with 12 drives.  From here on out I'll just be upgrading drives as needed.