January 26, 200917 yr I built a fully loaded unraid system. Initially I started out with 7 1.5TB drives and was getting 90-100MB/s on my parity check. I then added 9 more 1.5TB drives and had to buy a PCI SATA card. I picked up the cheap Syba 4 port and updated the bios to the non-raid version. Works fine but I realize the performance hit is not acceptable. So here is my quandary. I have the "official" Supermicro S2CEE board with 6 on-board SATA along with 2 Adaptec PCI Express cards. I need a setup that will support 17 drives (16 array drives + 1 cache drive). Does anyone have any suggestions? My thinking is to try a different motherboard with 8 or more SATA ports - or get a board that has more PCI Express slots so I can use more Adaptec cards. Any tips would be greatly appreciated. Thanks!
January 26, 200917 yr Correct me if I'm wrong but it sounds like your PCI-e 1x slot is free on that motherboard unless you have aren't using the onboard NIC. The following may work: 6x onboard SATA 4x Adaptec PCI-E #1 (curious which model? I'm guessing 1430SA) 4x Adaptec PCI-E #2 2x PCI-e 1x SATA (Perhaps SIL 3132 although there may be something quicker) 1x Syba 4 port PCI (Cache drive most likely) ------ 17 I would wait for confirmation from another forum member before hitting the "buy" button
January 26, 200917 yr I think we've talked about this before in another thread but the Intel D975XBX2 (BadAxe2) has 8 SATA and 3 PCI-E slots that support x4 or greater giving you the potential for 20 drives. I am using 2 of the Adaptec 1430SA adapters and get about 45MBs parity on 4.4.2 using 12 WD Greenpower drives that run at 5400rpm. The Seagate 1.5TB drives are noticeably faster so you might see better performance there. Newegg has the bulk version in stock as well as some open box units... http://www.newegg.com/Product/Product.aspx?Item=N82E16813121060 http://www.newegg.com/Product/Product.aspx?Item=N82E16813121059R Regards, Peter
January 26, 200917 yr Author I have: 6x SATA on-board 4x Adaptec PCI-E 1430SA 4x Adaptec PCI-E 1430SA 4x SYBA PCI (Silicon Image SIL3124) - this is what I believe is slowing my whole system down. ---- 18 It does not help that the on-board NIC port on my board was dead and I had to use a Netgear gigabit PCI card. So I guess the questions is - what is the best way to go - I am inclined to replace the board since mine already has a problem. I could get a board with 8x or 10x onboard and with 3+ PCI Express slots. This would give me the ability to expand once unraid supports more drives. (My case supports 21 drives) Just want to be sure I get something that is supported and works.
January 26, 200917 yr Author I think we've talked about this before in another thread but the Intel D975XBX2 (BadAxe2) has 8 SATA and 3 PCI-E slots that support x4 or greater giving you the potential for 20 drives. I am using 2 of the Adaptec 1430SA adapters and get about 45MBs parity on 4.4.2 using 12 WD Greenpower drives that run at 5400rpm. The Seagate 1.5TB drives are noticeably faster so you might see better performance there. Newegg has the bulk version in stock as well as some open box units... http://www.newegg.com/Product/Product.aspx?Item=N82E16813121060 http://www.newegg.com/Product/Product.aspx?Item=N82E16813121059R Regards, Peter Wow - Thanks! I actually use this same board in my main home system. Thanks for the advice!
January 26, 200917 yr Author One more thing - when making a change like this - is there any trick / tips on how to handle all the drives. Or do you just take note of which drive number has what serial number and then re-assign them accordingly?
January 26, 200917 yr One more thing - when making a change like this - is there any trick / tips on how to handle all the drives. Or do you just take note of which drive number has what serial number and then re-assign them accordingly? Exactly! WHen you first start the new system you will get all kinds of errors regarding missing drives, etc. Just reassign them on the devices page and everything will be okay. I did a screen dump of the old devices page and printed that out to make sure I did not mess it up. Good luck! Regards, Peter
January 26, 200917 yr BTW, if you do get the Bad Axe 2 the setup is pretty easy. I disabled sound, floppy, etc., basically everything that I did not need. The two important changes were making sure the drives were set to AHCI mode and that the USB boot configuration was set to ALL FIXED DRIVES (or something like that since I am writing this from memory). If you need help, just post or PM. Regards, Peter I also use the Bad Axe 2 for my main Hackintosh....
January 26, 200917 yr You could try an 8x card. I have an LSI card http://lime-technology.com/forum/index.php?topic=3109.0 and I'm just waiting on 4.5b2 to do the final testing. Essentially you keep one of your 4x 4 port cards and use this 8x card and you'll have 18 ports with 12 on individual PCIe lanes.
January 26, 200917 yr Just to provide a bit of perspective though, the parity check is not something you run every day. On average, it is run for the normal user about once a month, during off hours so who cares. So trying to optimize the system just for that does not seem to be a very important priority. The PCI bus does create a bottleneck for simultaneous access through it, but most unRAID server access is from a single drive at any one point in time, and the PCI bus won't limit that. I believe the only time you would perceive a slowdown in normal operations would be trying to stream to or from 3 or more drives on the PCI bus, or 2 or more drives plus a PCI gigabit card.
January 26, 200917 yr Author Just to provide a bit of perspective though, the parity check is not something you run every day. On average, it is run for the normal user about once a month, during off hours so who cares. So trying to optimize the system just for that does not seem to be a very important priority. The PCI bus does create a bottleneck for simultaneous access through it, but most unRAID server access is from a single drive at any one point in time, and the PCI bus won't limit that. I believe the only time you would perceive a slowdown in normal operations would be trying to stream to or from 3 or more drives on the PCI bus, or 2 or more drives plus a PCI gigabit card. If my Supermicro board had a working NIC port - I would probably just leave it as is and deal with it. But with 4 drives on the PCI bus and my nic card on the PCI bus - I have seen some big performance drops. So since I want to get the NIC off the PCI bus and back on the onboard - I need to replace my board either way. When transfering large files to the array I went from 16-17mb/s to 8mb/s. The decrease on the parity check is totally dismal - going from 90-100mb/s to 27mb/sec. I did not run my parity check all the way through but it would have taken a LONG time to finish.
January 27, 200917 yr Are the 4 drives on the PCI bus assigned sequentially across the same controller. If so, I might stagger the assignments so the next consecutive drive was accessed on a different bus. In addition I would double check if the PCI bus supported a 32Bit/66MHZ PCI card. I know the promise cards support a 66MHZ pci bus. On a mini ITX system I had with a 66MHZ PCI bus and a promise TX4 I was getting 59MB/s with 4X1TB WD Green drives.
January 27, 200917 yr Here is another option to consider. This board; http://www.newegg.com/Product/Product.aspx?Item=N82E16813182142 and then throwing the extra drives onto a PCI-X to 8-SATA card is a good way to add more drives. http://www.newegg.com/Product/Product.aspx?Item=N82E16815121009&Tpk=AOC-SAT2-MV8 From what I understand, you could support 8 drives on each PCI-X buss (th 100Mhz and the 133Mhz) without much of a performance penalty because each of those busses is independent from the other and the PCI-X buss is decent for supporting 8 SATA drives at the present SATA drive speeds. This way, you can connect 30 drives some day if you so desire. 6 onboard 4 PCIe #1 4 PCIe #2 8 PCI-X #1 8 PCI-X #2 It's also got onboard VGA video so you don't have to bother with a video card for the times you want/need to connect to it. WeeboTech has a very good point to not overlook. Stagger the drives on the PCI device. Disk1 - PCI, Disk2 - Motherboard, Disk3 - PCI etc Peter
January 28, 200917 yr Here is another option to consider. This board; http://www.newegg.com/Product/Product.aspx?Item=N82E16813182142 and then throwing the extra drives onto a PCI-X to 8-SATA card is a good way to add more drives. http://www.newegg.com/Product/Product.aspx?Item=N82E16815121009&Tpk=AOC-SAT2-MV8 From what I understand, you could support 8 drives on each PCI-X buss (th 100Mhz and the 133Mhz) without much of a performance penalty because each of those busses is independent from the other and the PCI-X buss is decent for supporting 8 SATA drives at the present SATA drive speeds. This way, you can connect 30 drives some day if you so desire. 6 onboard 4 PCIe #1 4 PCIe #2 8 PCI-X #1 8 PCI-X #2 It's also got onboard VGA video so you don't have to bother with a video card for the times you want/need to connect to it. WeeboTech has a very good point to not overlook. Stagger the drives on the PCI device. Disk1 - PCI, Disk2 - Motherboard, Disk3 - PCI etc Peter I think you could connect 20 more drives than that. There are 4 PCI-X slots, 2 on each PCI-x bus. That would get you 16 more. THe PCIe 8x slot could also handle 8 drives, albeit with a more expensive controller, getting you 4 more. This means you could get 50 connected to this motherboard. (We won't talk about the IDE port.) With the new 2T drives, that could mean 100T in one server. Wow! The only concern I'd have about going this route is that PCI-X was not very popular outside of the server space, and the ongoing availabliity of motherboards that have PCI-X slots is definitely in question. PCIe seems to pretty much be taking over. Even today, you pay a large premium for a PCI-X motherboard. But the fact that the 8 drive PCI-X card is pretty cheap per port helps offset that if you are going for an uber drive setup. With the PCI/PCI-X architecture, there is a pool of shared bandwidth available. At first I thought that all 4 PCI-X slots would shared one chunk of bandwidth, but if I'm reading the block diagram right, it appears that the board does, in fact, have 2 PCI-X busses. At 133MHz, you would get about 1GB/sec (1064 MB/sec) throughput. At 100MHz, you would get about 800MB/sec throughput. Loading these up with 8 drives each would not be a problem. With 2 cards on each bus, the worst things could get is about 50MB/sec. Still pretty respectable for a 50 drive system. The PCIe bus is, by contrast, serial. Each "lane" provides a dedicated 255MB/sec of bandwidth. So a 4x card would provide 4 times that bandwidth = ~1GHz, about the same as the PCI-X 133MHz bus. But since it is serial, it cannot be combined (unless the controller gets fancy). In most x4 implementations, each drive gets one dedicated lane. For most 1x cards, one lane is shared between two drives. With PCI-X, I think it would be possible to have a solid state drive that could pump close to 1GB/sec and the bus could keep up with it. In a PCIe setup, the drive would be limited to one 255MB/sec lane.
January 29, 200917 yr Author Here is another option to consider. This board; http://www.newegg.com/Product/Product.aspx?Item=N82E16813182142 and then throwing the extra drives onto a PCI-X to 8-SATA card is a good way to add more drives. http://www.newegg.com/Product/Product.aspx?Item=N82E16815121009&Tpk=AOC-SAT2-MV8 From what I understand, you could support 8 drives on each PCI-X buss (th 100Mhz and the 133Mhz) without much of a performance penalty because each of those busses is independent from the other and the PCI-X buss is decent for supporting 8 SATA drives at the present SATA drive speeds. This way, you can connect 30 drives some day if you so desire. 6 onboard 4 PCIe #1 4 PCIe #2 8 PCI-X #1 8 PCI-X #2 It's also got onboard VGA video so you don't have to bother with a video card for the times you want/need to connect to it. WeeboTech has a very good point to not overlook. Stagger the drives on the PCI device. Disk1 - PCI, Disk2 - Motherboard, Disk3 - PCI etc Peter I think you could connect 20 more drives than that. There are 4 PCI-X slots, 2 on each PCI-x bus. That would get you 16 more. THe PCIe 8x slot could also handle 8 drives, albeit with a more expensive controller, getting you 4 more. This means you could get 50 connected to this motherboard. (We won't talk about the IDE port.) With the new 2T drives, that could mean 100T in one server. Wow! The only concern I'd have about going this route is that PCI-X was not very popular outside of the server space, and the ongoing availabliity of motherboards that have PCI-X slots is definitely in question. PCIe seems to pretty much be taking over. Even today, you pay a large premium for a PCI-X motherboard. But the fact that the 8 drive PCI-X card is pretty cheap per port helps offset that if you are going for an uber drive setup. With the PCI/PCI-X architecture, there is a pool of shared bandwidth available. At first I thought that all 4 PCI-X slots would shared one chunk of bandwidth, but if I'm reading the block diagram right, it appears that the board does, in fact, have 2 PCI-X busses. At 133MHz, you would get about 1GB/sec (1064 MB/sec) throughput. At 100MHz, you would get about 800MB/sec throughput. Loading these up with 8 drives each would not be a problem. With 2 cards on each bus, the worst things could get is about 50MB/sec. Still pretty respectable for a 50 drive system. The PCIe bus is, by contrast, serial. Each "lane" provides a dedicated 255MB/sec of bandwidth. So a 4x card would provide 4 times that bandwidth = ~1GHz, about the same as the PCI-X 133MHz bus. But since it is serial, it cannot be combined (unless the controller gets fancy). In most x4 implementations, each drive gets one dedicated lane. For most 1x cards, one lane is shared between two drives. With PCI-X, I think it would be possible to have a solid state drive that could pump close to 1GB/sec and the bus could keep up with it. In a PCIe setup, the drive would be limited to one 255MB/sec lane. Don't get me started (I just invested in 20 1.5TB drives and WD released their 2TB drives today!) I just gotta settle in on what I have started or this will be the project that never ends. I do want to apologize about starting a thread that probably should have been in the hardware forums. I decided to get a 3rd Adaptec 1430SA controller and change my board to the Intel D975XBX2KR (Bad Axe2 as some call it). 8 on board SATA with 3 PCI-E slots. Board has 8x SATA ports. 4 Using Intel Controller and 4 more using a Matrix controller? 3 x Adaptec PCI-Express 4 port cards Total 20 ports. Since I have a Norco 4020 Case with 20 drive support (4 columns x 5 drives down) - I decided to alternate connections. Each column of 5 drives are connected to a different controller set. (3 PCI-E Adaptecs, 1x On-board Intel, 1x On-board Matrix) Hopefully this helps to achieve the best performance. Right now Un-raid only supports 17 total drives (including cache drive) so I have 3 open drive slots ready to go.....
January 29, 200917 yr Would you mind posting a "mini review" and perhaps a couple photos of that Norco case? What do you think of the overall quality, especially of the case fans?
January 29, 200917 yr Don't get me started (I just invested in 20 1.5TB drives and WD released their 2TB drives today!) I just gotta settle in on what I have started or this will be the project that never ends. Do you have a full 30TB of data? IMO, it's better to spread large amounts of drive purchases over a period of time. I've found that if a drive does fail, if you have other drives from the same batch, chances of a second failure are much higher. in comparison, if you were to purchase a new drive once a month over the course of time, chances of multiple failures are reduced. I suppose if you are getting a discount on a large amount of drives, it makes sense. However, I would highly recommend you keep one drive idle as a warm spare just in case.
January 29, 200917 yr Author Don't get me started (I just invested in 20 1.5TB drives and WD released their 2TB drives today!) I just gotta settle in on what I have started or this will be the project that never ends. I suppose if you are getting a discount on a large amount of drives, it makes sense. However, I would highly recommend you keep one drive idle as a warm spare just in case. I did spread my purchases of the 1.5TB drives around. I started out with 4 of them about 3 months ago. Then bought 5 more when I started my unraid project (from Newegg) - then using a 5% cash back from live.com bought 9 more from Tiger. I did have one hot spare drive ready to go but decided to put it in my Directv DVR instead. So far I did have one fail. When I am all done I will do a full review of what I did. From motherboard upgrades to case modifications to speed and test results. I can tell you that my norco case is currently running quieter than my home computer and nearly undetectable unless you are 3-4 feet away from it. During parity checks the highest any drives reach is 47C. But I have a few more modifications up my sleeve - goal is to bring that number down to low 40's during parity checks.
January 29, 200917 yr Would you mind posting a "mini review" and perhaps a couple photos of that Norco case? What do you think of the overall quality, especially of the case fans? Here's a thread with some info... http://lime-technology.com/forum/index.php?topic=2935.0 Regards, Peter
Archived
This topic is now archived and is closed to further replies.