June 6, 201313 yr The image shown above is from a parity rebuld that I'm currently running but the speed is what I used to get for my parity check for as long as I can remember. Right around the time I upgraded to 5.x, which is also about the time I maxed out my server with 24 drives, my parity check speeds went to approx 37mb/s Hardware: Supermicro C2SEA celeron 450 4gb ram 2x SUPERMICRO AOC-SASLP-MV8 sil3132 While running this rebuild I have 20 data disks. (I normally have 23 data disks but 3 were empty and I needed to rebuild parity anyway so I removed them to speed things up.) This is the same rebuild after getting past 2tb (all that remains are four 4tb drives). Speed is much better but decreases as it gets closer to finishing. I've been reading through some of the later pages in this thread Ultra Low Power 24-Bay Server - Thoughts on Build? which seem to indicate that the results I'm seeing should be expected using a PCI-e 1.0 x4 card with 8 drives. I'm not 100% sure I'm following it all though. Should't the MV8 get 1GB/s giving each of the 8 drives connected 125MB/s. Or is the 1GB/s for all PCI-e cards combined? If that's the case would the sata ports on the MB also be included in that? Are the speeds I'm seeing correct for my equipment? Is it possible to speed it up and if so what would i need to do/replace? Thank you
June 6, 201313 yr The image shown above is from a parity rebuld that I'm currently running but the speed is what I used to get for my parity check for as long as I can remember. Right around the time I upgraded to 5.x, which is also about the time I maxed out my server with 24 drives, my parity check speeds went to approx 37mb/s Hardware: Supermicro C2SEA celeron 450 4gb ram 2x SUPERMICRO AOC-SASLP-MV8 sil3132 While running this rebuild I have 20 data disks. (I normally have 23 data disks but 3 were empty and I needed to rebuild parity anyway so I removed them to speed things up.) This is the same rebuild after getting past 2tb (all that remains are four 4tb drives). Speed is much better but decreases as it gets closer to finishing. I've been reading through some of the later pages in this thread Ultra Low Power 24-Bay Server - Thoughts on Build? which seem to indicate that the results I'm seeing should be expected using a PCI-e 1.0 x4 card with 8 drives. I'm not 100% sure I'm following it all though. Should't the MV8 get 1GB/s giving each of the 8 drives connected 125MB/s. Or is the 1GB/s for all PCI-e cards combined? If that's the case would the sata ports on the MB also be included in that? Are the speeds I'm seeing correct for my equipment? Is it possible to speed it up and if so what would i need to do/replace? Thank you Where is your parity drive? I tend to isolate that drive as much as I can. i.e. it's own controller or the fastest controller. Also, your parity generate/check speeds can only go as fast as your slowest drive. 37MB/s seems a bit slow. Have you tried the tunings suggested? As far as testing maximim controller speed and drive speed you could try the old DD read test. dd of=/dev/null bs=1024 count=10240000 if=/dev/sd? where ? = your drive letter. you can run these in parallel on each drive to see how they impact one another. That's how I discovered the issues with port multiplier speed. I would run a few of these in parallel on various drives on various controllers. This will be the absolute fastest your drive can read in the current hardware. It bypasses filesystem housekeeping. Example of my PM 840 256GB SSD on SATA III under ESX dd of=/dev/null bs=1024 count=10240000 if=/dev/sda 10240000+0 records in 10240000+0 records out 10485760000 bytes (10 GB) copied, 17.1762 s, 610 MB/s
June 7, 201313 yr Author My parity drive is on the MB. I'll give that DD command a try as soon as the parity check is done.
June 7, 201313 yr I'm not 100% sure I'm following it all though. Should't the MV8 get 1GB/s giving each of the 8 drives connected 125MB/s. Or is the 1GB/s for all PCI-e cards combined? If that's the case would the sata ports on the MB also be included in that? The PCIe 1.0 x4 connection is for that controller card only. So the 8 drives connected to that card have 1GB/s to share, 125 MB/s each. Two notes on that: Some/all of your drives are faster than 125MB/s (especially your 4TB drive), so they are being throttled. The 1GB/s and 125MB/s numbers don't take into account overhead, so those are theoretical figures you would never be able to reach in a real system. So yeah, your drives are being throttled. Here's a tip that helped me: I found that the mixture of drives on each controller dramatically affected the performance. If you can, put all of the same model/size/speed drives on the same controller together. I saw 75% improvement from this tweak alone.
June 7, 201313 yr As far as testing maximim controller speed and drive speed you could try the old DD read test. dd of=/dev/null bs=1024 count=10240000 if=/dev/sd? where ? = your drive letter. you can run these in parallel on each drive to see how they impact one another. That's how I discovered the issues with port multiplier speed. I would run a few of these in parallel on various drives on various controllers. This will be the absolute fastest your drive can read in the current hardware. It bypasses filesystem housekeeping. Example of my PM 840 256GB SSD on SATA III under ESX dd of=/dev/null bs=1024 count=10240000 if=/dev/sda 10240000+0 records in 10240000+0 records out 10485760000 bytes (10 GB) copied, 17.1762 s, 610 MB/s Is they a thread on how to run this on all the drives at the same time? Thanks.
June 12, 201313 yr As far as testing maximim controller speed and drive speed you could try the old DD read test. dd of=/dev/null bs=1024 count=10240000 if=/dev/sd? where ? = your drive letter. you can run these in parallel on each drive to see how they impact one another. That's how I discovered the issues with port multiplier speed. I would run a few of these in parallel on various drives on various controllers. This will be the absolute fastest your drive can read in the current hardware. It bypasses filesystem housekeeping. Example of my PM 840 256GB SSD on SATA III under ESX dd of=/dev/null bs=1024 count=10240000 if=/dev/sda 10240000+0 records in 10240000+0 records out 10485760000 bytes (10 GB) copied, 17.1762 s, 610 MB/s Is they a thread on how to run this on all the drives at the same time? Thanks. No thread on it, I basically start up multiple sessions and trigger them all off at the same time. I suppose I could write a script that does this. Taking the other hdparm speed test as a basic shell I could use that as skeleton to do a raw dd read test of each drive (rather then the hdparm test). In the meantime. I've modified the diskspeed.sh and will upload it to my googlecode page in the next day or so. I've been waiting for a preclear on 4 drives to finish before I start to test it again and upload it.
Archived
This topic is now archived and is closed to further replies.