I have an unRAID server with 18 data disks I built 5 years ago. It's running unRAID 5 and has 20 total disks, one parity and one cache on top of the data disks.

 

I've just recently started migrating the older, smaller 2TB drives to 6TB drives. In the process I've had the pleasure of syncing my parity and rebuilding data drives a few times. I also had some issues with loose cables and bad seating on one of the Norco drive cages that caused some red balls and I had to do rebuilds a few times more.

 

The current drive inventory on the array is 10 x 2TB, 5 x 4TB and 3 x 6 TB. One of those 6TB drives is currently rebuilding after it replaced a 2TB drive.

 

My motherboard is a BIOSTAR H55A+ LGA 1156 motherboard which has 2xPCIex16 2.0 expansion slots and 6x3Gbps SATA ports on the motherboard. I have two AOC-SASLP-MV8 SATA controllers on those PCIe slots connecting to 14 of the 20 drives on the server. 6 drives are connected to the SATA ports on the motherboard.

 

The data rebuild just passed the 2TB mark, and the speed drastically improved. I was doing roughly 15MB/s up until 2TB and the speed just picked up to 94MB/s after passing the 2TB mark.

 

After reading about this sort of stuff this morning I'm under the impression that the bottleneck is almost certainly those AOC-SASLP-MV8 controllers. They can't push enough data through the bus when it has to do that for all the drives at the same time.

 

If I replaced the AOC-SASLP-MV8 controllers with AOC-SAS2LP-MV8 controllers, I should see significant speed increase (at least 2x) on parity syncs, parity checks and data rebuilds during the time it's reading parity from all drives.

 

Additionally I should make sure the parity drive is connected to the motherboard SATA port. I'm not sure that's true at the moment.

 

Does that sound correct?

 

Anything else anyone could recommend? Should I try and stick as many of the 2TB disks onto the motherboard ports as I can as well?

You main bottleneck is DMI, which in your board is first gen, up to 1000MB/s, all 6 onboard ports + how many drives are in the bottom pcie x16(x4) controller have to share it, still should give you more than 15MB/s, even with 14 disks going through it it should be roughly 55Mb/s, you may be needing some tuning.

 

Speed won't improve by replacing the controllers, the most you can do with that board is to connect only 6 to the bottom controller plus the 6 onboard, with some tuning rebuilds should start at ~65MB/s.

 

The SASLP with no other bottlenecks is not the fastest card but it's capable of 80MB/s fully loaded.

 

 

Thanks for the info. I was reading that SATA controller benchmarking thread of yours but somehow missed the DMI section.

 

I'll do speed tests on the drives after the current data rebuild completes to see if I'm having issues with one or more of the drives first. I'll also doublecheck seating on the controllers and the cable connections.

 

Possibly a stupid question...how do I identify which one of my controllers is the "bottom controller"?

The black top PCIe x16 slot, where the graphics card usually goes is a CPU slot, i.e., the 16 lanes are connected directly to the CPU, the red bottom slot, although physically x16, only has 4 lanes (x4 electrically) and it's connected to the PCH (south bridge replacement), together with the onboard SATA ports and all other slots and devices, all these share the DMI connection from the PCH to the CPU.

Thanks. I also verified this from the motherboard manual.

 

The data rebuild should finish sometime this evening. I'll start tweaking and troubleshooting the drives when that's done.

The default disk settings are IMO not optimal, especially for bigger arrays, and while these are not universal settings I find them a pretty good starting point and usually considerably better than default with most hardware, all theses are only available on v6.2, if you are on v6.1 set the ones that are.

 

Settings -> Disk Settings:

 

Wow! Some of those numbers are hugely different from the defaults. I'd like to try them myself, but what's the trade-off? More memory needed, perhaps?

Wow! Some of those numbers are hugely different from the defaults. I'd like to try them myself, but what's the trade-off? More memory needed, perhaps?

 

Yes but it's only a few MB per disk, unless you have <2GB of RAM it wont make any noticeable difference.

Thanks. I'll have a play and run a parity check with the new values.

I was doing roughly 15MB/s up until 2TB and the speed just picked up to 94MB/s after passing the 2TB mark.

 

Thinking a little more about this, this speed seems rather slow even without any tuning, so post your diagnostics, better if grabbed during a rebuild, it could be a hardware issue, e.g., some Samsung disks that are extra slow on some controllers.

The syslog has been completely quiet during this rebuild. Literally not logging anything at all after the log entry about starting the data rebuild. The 2TB and 4TB drives spun down at some point, but those are the only log entries in there. It's got a little less than an hour to go and it's humming along pretty nicely at 87MB/s.

 

I'm suspecting one or more of the older drives are getting close to end of their life, but we'll see. Like I said, I'll do some digging after the data rebuild is done. Hopefully I'll find something.

I just finished running the diskspeed.sh utility on my array.

 

I don't see anything abnormal in there with the exception of one of the 2TB disks having kind of a choppy chart. I'll run the diskspeed script with more sampling points on that drive next.

 

2TB WDC EARS drives manufactured in 2010 are the slowest, but the numbers look similar to other reports at slightly less than 100MB/sec.

Do you have any Samsung disks? If yes, model?

Do you have any Samsung disks? If yes, model?

 

There are three Samsung ST2000DL004s in there.

 

This is the data drive inventory:

 

1 Hitachi HDS5C3020ALA632

3 Samsung ST2000DL004

3 Seagate ST4000VN000

4 WDC WD20EARS (soon 3)

2 WDC WD20EFRX

2 WDC WD40EZRX

3 WDC WD60EZRZ (soon 4)

 

Here's the chart for two of the Samsung drives sampled at every 1% this time. This looks a little odd to me, but the average speeds are still way up there, so I'm not sure if that's anything to worry about.

Looks fine.

I've rearranged the SATA connections.

 

Parity disk is now connected to a SATA port on the motherboard. According to old syslogs it was previously attached to the SATA controller connected via DMI. That controller also had 8 drives connected to it while the controller directly connected to the CPU only had 6. That's now also been corrected.

 

diskspeed report looks pretty much the same.

 

I ran the unraid-tunable-tester tool overnight and just changed the tunables to the Unthrottled values recommended by the script:

 

md_num_stripes="4480"

md_write_limit="2048"

md_sync_window="2048"

 

I am now running a parity check to see if there is any improvement.

 

It might be a little early to say, because the check hasn't run for too long yet, but it looks much better at the start than before all these changes. I'm currently seeing a 40+ MB/sec speed.

Looks like the changes are working quite nicely. I'm getting roughly 60MB/sec parity check speed at 780GB mark.

 

Thanks a lot for your help Johnnie! Really, really appreciate it. This is going to cut days off the process of replacing the older drives with new ones.

That's about the best you can expect with that board and your current config due to its 1st gen DMI and "fake" PCIe 2.0 on the PCH.