Bandwidth Limit

[1812]

6 minutes ago, johnnie.black said:

 

If I understand correctly the expander is using 2 wide links, one to each server, so both wide ports are in use so the 8 is correct, but obviously in that configuration it doesn't represent dual link.

 

Also, your speeds are more consistent with a SATA 1.5gbps single link, are you sure the disks are linking at sata 3gbps? a single link @ sata2 is good for 1100MB/s

You would think there would be 1100MB/s of bandwidth available.... 

 

All disks show sata3 as current connection in smart.  diagnostics from the 3 disk non parity array attached. 

brahms3-diagnostics-20170430-1029.zip

[JorgeB]

Right but with my largest disk being 8TB they should split at 1TB free. Or is it per share? I have my 8TB's for media and 2TB/3TB's for everything else. That would make sense if they are splitting at 1.5TB free.

It's per share

[JorgeB]

4 minutes ago, 1812 said:

You would think there would be 1100MB/s of bandwidth available.... 

 

All disks show sata3 as current connection in smart.  diagnostics from the 3 disk non parity array attached. 

brahms3-diagnostics-20170430-1029.zip

 

3 disk server is normal, you have a 3tb RED, max speed of that one is 150MB/s, so 3 x 150= 450MB/s, the other server looks slower than it should, post those diags, maybe I'll spot something.

[1812]

3 minutes ago, johnnie.black said:

 

3 disk server is normal, you have a 3tb RED, max speed of that one is 150MB/s, so 3 x 150= 450MB/s, the other server looks slower than it should, post those diags, maybe I'll spot something.

 

I tink a few off the sub 1TB disks are just dog slow on that one....

brahms1-diagnostics-20170430-1342.zip

[HellDiverUK]

3 hours ago, ashman70 said:

Just wanted to say your drive temps are kind of high, mind you if that is during a parity check I guess its not too bad. What kind of case are you using?

 

The highest I see is 43C, which isn't high at all.  55C is high.

[1812]

Forgot I did this a short time ago @johnnie.black. Shows my slow disks in that server... The math should work right for my results I think?

[lonnie776]

1 minute ago, 1812 said:

Forgot I did this a short time ago @johnnie.black. Shows my slow disks in that server... The math should work right for my results I think?

Why have you been keeping this software from me! what is it and how can I get it?

[Squid]

11 minutes ago, lonnie776 said:

Why have you been keeping this software from me! what is it and how can I get it?

 

[lonnie776]

Mmmmm... Nerd porn.

[JorgeB]

51 minutes ago, 1812 said:

 

I tink a few off the sub 1TB disks are just dog slow on that one....

brahms1-diagnostics-20170430-1342.zip

 

Nothing jumps out, those slower disks by themselves don't explain that starting speed, you're also using the default tunables, doubt that that's the only reason for the lower speed but try these and see if it improves:

 

Tunable (md_num_stripes): 4096

Tunable (md_sync_window): 2048

Tunable (md_sync_thresh): 2000

[lonnie776]

So I just passed the 25% mark (2TB) on my parity check and here again you can see that 425MB/s limit, aside from a few stray read ops. Always the same array limit of 425MB/s no matter the number of drives. Do you think it's that SAS Expander?

[1812]

5 hours ago, johnnie.black said:

 

Nothing jumps out, those slower disks by themselves don't explain that starting speed, you're also using the default tunables, doubt that that's the only reason for the lower speed but try these and see if it improves:

 

Tunable (md_num_stripes): 4096

Tunable (md_sync_window): 2048

Tunable (md_sync_thresh): 2000

 

 

No real changes using those settings:

 

 

 

 

[lonnie776]

Hmm... your stats graph... is it always that flat on top?

[1812]

Just now, lonnie776 said:

Hmm... your stats graph... is it always that flat on top?

 

For the purpose of this testing, which is running a parity check, it wavers within a few MB/s of that number. Seems like some sort of ceiling. I've never paid that much close attention to it. I might try to get it to push over to see if the SAS expander is the limit, or if i can run the parity check and move some data from one SSD to another across the expander and get a higher number. Give me a few minutes to see.

[lonnie776]

Sounds great. My graph looks the same aside from a few kick backs from writes.

[1812]

OK, parity check runs at 500ishMB/s. Transfer a 25GB .img file from one SSD to another via unassigned devices (2gbps max speed) bumps reads up to 730MB/s plus the 215 writes.

 

All parity disks are accessed via mini SAS on the expander card, which is listed as a 4x port. That is going through a 4x cable I posted earlier to the MD1000 which lists the connective as "One x4 3GB SAS (SFF 8470)." The SSD's are in a forward cage in the server. 

 

So for me, clearly there is a limiting factor either on the mini-sas out, the cable, or the MD1000. I've swapped EMM"s before when I was troubleshooting what ended up being a phantom problem so I believe those are fine. 

 

The SAS expander is on the latest firmware.

 

How can I verify the cable is correct aside from packaging? Because technically something appears to only be running at half capacity.

 

 

 

[lonnie776]

Hmm... Very interesting. I believe mine is running on the latest firmware as well however my HBA is not. I'll try re-cabling and a firmware update once my parity check is done as I should be able to get 8x with my setup. I'll also test my cache drives the same way you tested your but mine will be from cache to cache.

 

Also you could always run a multi-meter across the cable to test that all the pins are actually connected and have low resistance.

Edited May 1, 2017 by lonnie776

[1812]

5 minutes ago, lonnie776 said:

Hmm... Very interesting. I believe mine is running on the latest firmware as well however my HBA is not. I'll try re-cabling and a firmware update once my parity check is done as I should be able to get 8x with my setup. I'll also test my cache drives the same way you tested your but mine will be from cache to cache.

 

Also you could always run a multi-meter across the cable to test that all the pins are actually connected and have low resistance.

 

What type of PCIe slot is your hba in? 2 or 3?

[lonnie776]

PCIe 3.0 x16 running in x8 i believe.

[1812]

32 minutes ago, lonnie776 said:

PCIe 3.0 x16 running in x8 i believe.

 

 

my hba is in a PCIe 2 in either an x4 or x8... I'd have to check... But at a minimum it would still allow 2GB/s of data even if x4.....  and you're going to have way more bandwidth available.

 

perplexing....

 

Edited May 1, 2017 by 1812

[lonnie776]

FYI here are the results from a copy from ssd to ssd (Cache to Cache Raid 10) which is also connected to the hp sas expander. Me thinks I see a common thread. 

 

an x8 vs x4 test should show a doubling of speed.

 

[JorgeB]

There appears to be a limit somewhere affecting you both, I can only say that my HP expander total bandwidth is always 1100/2200MB/s in single/dual link.

[itimpi]

19 hours ago, lonnie776 said:

Right but with my largest disk being 8TB they should split at 1TB free. Or is it per share? I have my 8TB's for media and 2TB/3TB's for everything else. That would make sense if they are splitting at 1.5TB free.

The split points due to high water allocation are in relation to the largest data disk in the system and not to individual drive sizes..    Therefore since you mention the largest disk is 8TB they become 4TB, 2TB, 1TB, 0.5TB etc.   If you are seeing something different then it might need looking at more closely.    Also notice that the Share Split Level setting over-rides the allocation method setting if there is a conflict when choosing the drive for a particular file.

[lonnie776]

 

5 hours ago, johnnie.black said:

There appears to be a limit somewhere affecting you both, I can only say that my HP expander total bandwidth is always 1100/2200MB/s in single/dual link.

2 questions for you Johnnie.

 

1. Are you using the newer 12Gb/s model of the HP sas expander?

2. Can you check your Expander and see how it's cabled? Also there should be LED's near the back IO plate, can you see what yours look like?

 

I have mine cabled with the HBA going to the two ports marked specifically for the HBA. Also I saw a note on the expander about the two cables needing to be the same length going to the HBA... Makes sense. Are yours the same length?

[JorgeB]

24 minutes ago, lonnie776 said:

1. Are you using the newer 12Gb/s model of the HP sas expander?

 

Older SAS2/SATA2 model, SATA only links at 3gbps, hence the bandwidth limits, this is how they work on an expander:

 

Each wide port has total bandwidth of 4 x the linking speed, so for this case, since it links at sata2 speed, or 300MB/s, max theoretical bandwidth is 4 x 300MB/s, of those 1100MB/s are what I measured as the real world usable bandwidth, which is pretty good considering for example PCIe which has usually only about 70% usable.

 

With dual link bandwidth doubles to 2400MB/s, 2200 usable.

 

A SAS2/SATA3 expander, like the Intel RES2SV240, doubles those speeds, since each wide ports links at 4 x 600MB/s, tested real speed is 2200MB/s for a single link and 4400MB/s for a dual link (the latter one needs a PCIe 3.0 HBA to use all available bandwidth).

 

24 minutes ago, lonnie776 said:

2. Can you check your Expander and see how it's cabled? Also there should be LED's near the back IO plate, can you see what yours look like?

 

Can't easily access it now to check the leds, but I'm going to upgrade it to an Intel later this week, so I'll check then.

 

As for the cables, same length between HBA and expander is probably a good idea, and mine are, but on most expanders, including this HP, you can use whatever ports you want for in/out, all ports work in either direction.

Edited May 1, 2017 by johnnie.black