When I do my parity sync I get 60MB/s until the 2TB mark, then it goes to 95MB/s for the remaining 1TB. I assume this is because of bus bandwidth being limited since i'm using 21 drives. Is there any way for me to wire this so it won't happen? I'm using SATA->SAS so I can't just move my parity to onboard to my knowledge (and i'm not even sure if this mobo's onboard sata supports 3TB).

 

Pretty sure the top 2 PCI-X share bus, and that's probably where the issue is. I personally see no way to fix this, but figured I'd post just to make sure i'm not missing something. Would be nice to get the full 95MB/s for the entire sync.

 

http://img94.imageshack.us/img94/7775/cablesl.jpg

http://img833.imageshack.us/img833/2968/sasends.jpg

http://img64.imageshack.us/img64/6123/sataends.jpg

 

CPU: Intel E5200 (2.50GHz)

Motherboard: Supermicro MBD-X7SBE

Bios: 1.2a

Memory: 4GB G.Skill (DDR2-800 5-5-5-15)

Hard Drive: 14x WD20EARS, 7x WD30EZRX

SATA Cards: 3x SUPERMICRO AOC-SAT2-MV8

Power Supply: 850W Corsair HX850W

Case: Norco 4224

Operating System: unRAID 5.0 b12a

Total 'Advertised' Space: 49TB

 

 

The difference is due to the number of drives being accessed:

At <2TB all 21 drives are being accessed giving you 60MB/s

At >2TB only 7 drives are being accessed giving you 95MB/s

 

21 drives are using significantly more bandwidth than 7...

Actually all 4 of your PCI-X slots share the same x8 connection from the 3210 MCH.  No "rewiring" can help that.  You can get a PCI-E card (AOC-SASLP-MV8 or AOC-SAS2LP-MV8) and put that in the x4 or x8 slot (respectively) and move some drives to improve your throughput.

 

Regards,  Peter

You could try moving 6 of the 2T data drives to the motherboard. It would be fairly simple to unplug from the SATA cards and move down to the motherboard ports. Or, since you are getting very decent speeds, you could just not worry about the parity check speed so much and enjoy watching the media on the server.

 

Peter

I have the same motherboard and two of the SAT2 cards. I did some testing and found that placing two controller cards on the 133 bus did indeed limit bandwidth. The card has less bandwidth than the bus so two cards sharing the same bus does not half the bandwidth of the card. I also did some tests using a SASLP in place of one of the SAT2s. Two SAT2 cards had more bandwidth than the combo of SAT2 and SASLP.

 

lionelhutz is right, to maximize throughput you need to offload some of those ports from the 133 bus to the motherboard. I know it is painful to "waste" a SAT2, but that will get you maximum throughput. I also tested the motherboard ports and you will get the best throughput moving six drives to the motherboard. The other thing that is coming into play is that the 2T drives are slower when at the inner tracks than your 3T drives when they are 2/3rds of the way through the parity check. There is no solution for that other than to buy 15 more 3T. drives. Just leaving one "smaller" drive in the mix will limit parity check speeds until it is finished.