[Partially SOLVED] Is there an effort to solve the SAS2LP issue? (Tom Question)

[garycase]

... The SSDs are ACS-2, and the parity check speed on the SAS2LP doubled from ~100MB/s to >200MB/s.

 

Clearly that shows that the assumption that the change is only needed for drives that interface with ATA8-ACS  is NOT correct ==> but it really doesn't matter.    Simple fact is that simply setting the nr_requests to 8 resolves this issue !!

 

Now we just need a tunable parameter setting on the disk settings page so we don't have to apply this every time we boot 

 

One unresolved part of this mystery is why it's unique to v6 (v5 also defaults to nr_requests = 128) ... but I'm fine with that just being one of life's little mysteries since we now know how to avoid the issue.

 

[eschultz]

... The SSDs are ACS-2, and the parity check speed on the SAS2LP doubled from ~100MB/s to >200MB/s.

 

Clearly that shows that the assumption that the change is only needed for drives that interface with ATA8-ACS  is NOT correct ==> but it really doesn't matter.    Simple fact is that simply setting the nr_requests to 8 resolves this issue !!

 

Now we just need a tunable parameter setting on the disk settings page so we don't have to apply this every time we boot 

 

One unresolved part of this mystery is why it's unique to v6 (v5 also defaults to nr_requests = 128) ... but I'm fine with that just being one of life's little mysteries since we now know how to avoid the issue.

 

This surprised me too as I didn't see any performance difference on spinning HDDs that interface with ACS-2/3 when I kept them at the 128 nr_requests default... but this clearly is making a huge impact for these SSDs.

 

Tom and I are still experimenting.  Setting the nr_requests to 8 makes a world of difference for parity checks but does this negatively affect performance for normal read/write operations for anyone? 

[Russ Uno]

One unresolved part of this mystery is why it's unique to v6 (v5 also defaults to nr_requests = 128) ... but I'm fine with that just being one of life's little mysteries since we now know how to avoid the issue.

Maybe because, in theory at least 64 bit systems are faster then 32 bit making it too fast for the controller...

just a thought.

[dlandon]

Tom and I are still experimenting.  Setting the nr_requests to 8 makes a world of difference for parity checks but does this negatively affect performance for normal read/write operations for anyone?

 

I implemented the nr_requests change on my server with 7 data and parity disk with newer SSD cache drive.  The drives are all WD Reds.  I've never had any issues with parity speed.  I don't know if there is a change in parity speed, the monthly parity check will run tomorrow and I will check. 

 

As far as normal read/write operations, there doesn't seem to be any degradation.  I haven't done anything very scientific though.

[wgstarks]

but you really only need to change the drives that have the lower ATA spec 'ATA8-ACS'. (You can see what the ATA Version is from the Identity tab on the disk details page).

What about ATA version= ATA/ATAPI-7 T13/1532D revision 1?

This is what my oldest HD's are using.

[Frank1940]

 

Tom and I are still experimenting.  Setting the nr_requests to 8 makes a world of difference for parity checks but does this negatively affect performance for normal read/write operations for anyone?

 

Wild thought.  Why not change it just for the Parity Check and then change it back to 128 when the parity check is complete.  I believe most of us have the parity check scheduled for a time when the server is not being used or other usage is low.

[JorgeB]

After testing with 8 faster 120GB SSDs I’m satisfied that this fix (or workaround) solves the issue, this result is very close to what I get with a Dell H310 on the same server, more than enough speed for the fastest HDDs on the market and then some.

 

default

 

Notice [TESTV6] - Parity check finished (0 errors)
Duration: 20 minutes, 53 seconds
Average speed: 95.8 MB/sec

 

nr_requests=8

 

Notice [TESTV6] - Parity check finished (0 errors)
Duration: 6 minutes, 31 seconds
Average speed: 307.0 MB/sec

 

 

Also it does not appear to have a negative impact on writes, at least on parity syncs and disk rebuilds:

 

default

 

Notice [TESTV6] - Parity sync: finished (0 errors)
Duration: 6 minutes, 1 second
Average speed: 332.5 MB/sec

Notice [TESTV6] - Data rebuild: finished (0 errors)
Duration: 5 minutes, 59 seconds
Average speed: 334.4 MB/sec

 

nr_requests=8

 

Notice [TESTV6] - Parity sync: finished (0 errors)
Duration: 6 minutes, 1 second
Average speed: 332.5 MB/sec

Notice [TESTV6] - Data rebuild: finished (0 errors)
Duration: 6 minutes, 6 seconds
Average speed: 328.0 MB/sec

 

 

Many thanks to LT and Eric for fixing an issue that although not critical was very frustrating.

 

P.S: like Eric pointed out, the disk tunable settings can have an huge impact on the parity check speed with the SAS2LP, with these SSDs and nr_reqs=8 the default settings gave me ~150MB/s, after running tunables tester ended up with 300MB/s+

 

[mr-hexen]

Tom and I are still experimenting.  Setting the nr_requests to 8 makes a world of difference for parity checks but does this negatively affect performance for normal read/write operations for anyone?

 

I implemented the nr_requests change on my server with 7 data and parity disk with newer SSD cache drive.  The drives are all WD Reds.  I've never had any issues with parity speed.  I don't know if there is a change in parity speed, the monthly parity check will run tomorrow and I will check. 

 

As far as normal read/write operations, there doesn't seem to be any degradation.  I haven't done anything very scientific though.

 

Same here. It usually takes my 3TB parity system 10 hours to complete. Let's see what happens.

[TUMS]

Excellent news!  I still have 2 sas2lp cards and I guess I will add them to a new server that I plan on building at the end of the year.  Saves me from having to buy more lsi 2008 based cards. Good work guys!

[opentoe]

Very cool news!

 

I had the slow parity check speeds with these drives. All were hooked up to two SAS2LP cards. Changed the cards to Dell H310's and the slow speeds went away. I have both my SAS2LP cards sitting in anti-static bags and can re-install them if more testing is needed. What would cause this to only affect some Marvell chipsets?

 

 

Parity        WDC 4001FAEX-00MJRA0_WD-WCC1F0127613 - 4 TB

Disk 1 WDC_WD2002FAEX-007BA0_WD-WMAWP0099064 - 2 TB

Disk 2 HGST_HDN724040ALE640_PK2338P4HE525C - 4 TB

Disk 3 Hitachi_HDS723020BLA642_MN5220F32HHMJK - 2 TB

Disk 4 HGST_HDN724040ALE640_PK2338P4HEPH8C - 4 TB

Disk 5 Hitachi_HDS723020BLA642_MN1220F30803DD - 2 TB

Disk 6 WDC_WD2002FAEX-007BA0_WD-WMAWP0284322 - 2 TB

Disk 7 HGST_HDN724040ALE640_PK1334PCJY9BRS - 4 TB

Disk 8 WDC_WD4001FAEX-00MJRA0_WD-WCC130263021 - 4 TB

Disk 9 WDC_WD4003FZEX-00Z4SA0_WD-WCC130966733 - 4 TB

Disk 10 HGST_HDN724040ALE640_PK2334PCGYD32B - 4 TB

Disk 11 WDC_WD2001FASS-00U0B0_WD-WMAUR0142521 - 2 TB

Disk 12 WDC_WD2002FAEX-007BA0_WD-WCAY01300087 - 2 TB

 

What I found to cause the slowdowns was when there was a drive using ATA Version 'ATA8-ACS' + Marvell chip + default nr_requests at 128.

 

If a drive has ATA Version 'ACS-2' or 'ACS-3' then changing the nr_requests doesn't matter for that device, it'll run fast unless it has to wait on a slower disk in the array.  It was just easier to have everyone here change their nr_requests to 8 for all array devices... but you really only need to change the drives that have the lower ATA spec 'ATA8-ACS'. (You can see what the ATA Version is from the Identity tab on the disk details page).

 

All of my drives have that spec except one drive. Nice find!

 

[garycase]

Wild thought.  Why not change it just for the Parity Check and then change it back to 128 when the parity check is complete.  I believe most of us have the parity check scheduled for a time when the server is not being used or other usage is low.

 

Interesting thought.    But I think it's easier to simply provide a "tunables" setting to change it.  Perhaps a "better" fix is to both (a) provide a tunables setting so it can be changed AND (b) set the default to 8.    I suspect that it would almost never be changed      [Although if it was that easy to change users could experiment  with intermediate values to see just where it becomes a problem ... i.e. it may work fine at 16, 32, etc.]    Just for grins I did a couple disk copies from the server and there's certainly no impact on performance for those -- but of course they're limited by the Gb network, so that's not the same as a parity sync or disk rebuild ... but Johnnie's tests have shown that those aren't impacted either (and if they're not impacted on his very fast SSDs, they're certainly not going to be a problem with spinners).

 

[switchman]

 

For anyone that doesn't know and wants a quick way to do this:

for i in {a..t}; do echo 8 > /sys/block/sd$i/queue/nr_requests; done

With {a..t} matching the first and last disks in your array.

 

Anyway to make this dynamic find the first and last disks in the system?

[BRiT]

Just do a to z range. It won't break anything if you don't have those disks, at worst it will just display a message saying it does not exist.

 

[Frank1940]

Wild thought.  Why not change it just for the Parity Check and then change it back to 128 when the parity check is complete.  I believe most of us have the parity check scheduled for a time when the server is not being used or other usage is low.

 

Interesting thought.    But I think it's easier to simply provide a "tunables" setting to change it.  Perhaps a "better" fix is to both (a) provide a tunables setting so it can be changed AND (b) set the default to 8.    I suspect that it would almost never be changed      [Although if it was that easy to change users could experiment  with intermediate values to see just where it becomes a problem ... i.e. it may work fine at 16, 32, etc.]    Just for grins I did a couple disk copies from the server and there's certainly no impact on performance for those -- but of course they're limited by the Gb network, so that's not the same as a parity sync or disk rebuild ... but Johnnie's tests have shown that those aren't impacted either (and if they're not impacted on his very fast SSDs, they're certainly not going to be a problem with spinners).

 

I googled this parameter a couple of days ago and could not really find a good description (at least to me) of what exactly it did.  But I got the feeling that increasing the number improves the performance on systems that have a lot of processes that are doing independent data read and writes.  (Think of servers that are handling ticket and seat bookings as an extreme example.  There were some instances of people continuing to see improvements in performance after they increase it to 32,768. I also notice that almost everyone who was talking about the value was setting the parameter to a power of 2.)

[garycase]

 

For anyone that doesn't know and wants a quick way to do this:

for i in {a..t}; do echo 8 > /sys/block/sd$i/queue/nr_requests; done

With {a..t} matching the first and last disks in your array.

 

Anyway to make this dynamic find the first and last disks in the system?

 

As Brit noted, you could just do it from a-z;  or if you don't want a lot of messages about non-existent drives, just do it from a-X, where X is the last drive in your system.    I may be wrong, but I believe if you look carefully you'll see that all your drives are assigned starting with a and going up => the only ones that won't be part of the actual array are your USB flash drive; any cache drives you have; and any unassigned drives you have.    But invoking this for those won't hurt anything (and if you stop it at the highest assigned letter will eliminate any error messages).

 

Hopefully this will be moot when the next release comes out => since Limetech knows this fixes the issue, I'd think they'll either make 8 the new default or provide a "tunable" in disk settings to make it easy to change.

 

... Anxiously awaiting v6.1.4 with this change  :)

 

[mr-hexen]

Tom and I are still experimenting.  Setting the nr_requests to 8 makes a world of difference for parity checks but does this negatively affect performance for normal read/write operations for anyone?

 

I implemented the nr_requests change on my server with 7 data and parity disk with newer SSD cache drive.  The drives are all WD Reds.  I've never had any issues with parity speed.  I don't know if there is a change in parity speed, the monthly parity check will run tomorrow and I will check. 

 

As far as normal read/write operations, there doesn't seem to be any degradation.  I haven't done anything very scientific though.

 

Same here. It usually takes my 3TB parity system 10 hours to complete. Let's see what happens.

 

My parity check finished with no noticeable difference in speed, 8 hours (10 hours was before I removed my 2TB drives, doh).

[tjambro]

I went from starting out about 45MB/s to over 110MB/s.

 

This was the jump I experienced using 8 for nr_requests.  I was able to ultimately increase it from 110MB/s to 125-130MB/s by tweaking the md_num_stripes and md_sync_window values from the Disk Settings page.

 

For reference, the default values are:

md_num_stripes = 1280

md_sync_window = 384

 

With my 14 drive array (including parity disk), I used these values:

md_num_stripes = 3840

md_sync_window = 1728

 

You're mileage may vary, but I encourage you to at least experiment with the md_sync_window value to see what you can top out at.

 

I ran the tunables script with just under 105MB/s.  With it's settings implemented and the nr 8 settings, I just completed a parity check with an average of 90.6MB/s. It actually started out higher and decreased over time which was surprising to me. Still much better than my last parity check which was around 50 or 55MB/s. Went from 19+ hours to 12:15.  I'll take it!

[JorgeB]

It actually started out higher and decreased over time which was surprising to me.

 

That’s how it should be, the sequential read speed slows down as the disk moves from the outer to the inner tracks, I have mostly WD green drives, parity check starts at ~150MB/s and ends at ~80MB/s, average speed in the end is a little over 100MB/s.

[tjambro]

It actually started out higher and decreased over time which was surprising to me.

 

That’s how it should be, the sequential read speed slows down as the disk moves from the outer to the inner tracks, I have mostly WD green drives, parity check starts at ~150MB/s and ends at ~80MB/s, average speed in the end is a little over 100MB/s.

 

That makes sense Johnnie, thanks. Reminds me of my laserdisc days with CAV and CLV. 

[EdgarWallace]

Can you or Tom see a mechanism to explain the rarer but much more serious issue of data corruption with the SAS2LP?  I'm wondering if the issue is not just a delay/backup in the queue processing, but a queue overflow or overwrite of an I/O, that might explain the more serious issues a few users have seen, like false (and repeatable) parity check errors.  Unfortunately, those users are the ones most likely to have sold off or trashed their SAS2LP cards, but they are the ones we most need to test this.

 

I was not affected by the parity check speed issue but I was facing parity sync errors which I had never seen before (Test&Backup Server). Any advise from anyone or even LT how to deal with this?

[RobJ]

Can you or Tom see a mechanism to explain the rarer but much more serious issue of data corruption with the SAS2LP?  I'm wondering if the issue is not just a delay/backup in the queue processing, but a queue overflow or overwrite of an I/O, that might explain the more serious issues a few users have seen, like false (and repeatable) parity check errors.  Unfortunately, those users are the ones most likely to have sold off or trashed their SAS2LP cards, but they are the ones we most need to test this.

 

I was not affected by the parity check speed issue but I was facing parity sync errors which I had never seen before (Test&Backup Server). Any advise from anyone or even LT how to deal with this?

 

Test it!  Try the change and see if you still have the same issues.  I've been hoping to hear from those like you, with other issues using Marvell controllers, especially the SAS2LP.

 

My own test resulted in almost identical times, essentially 14 hours.  The change in nr_requests resulted in 3.5 minutes faster, which given so many other factors, is not statistically significant.  But it's one more data point that there isn't a negative effect using other controllers (SiI3132 and ASM1061).

[garycase]

... My own test resulted in almost identical times, essentially 14 hours.

 

I presume this was with non-Marvell-based controllers ... is that correct?    Apparently the most common controllers that have caused this issue are the SAS2LP and the 1430SA ... but I'm sure there are others (perhaps some motherboards that use Marvell secondary controllers to add extra SATA ports).

 

It's still interesting that v5 didn't have this issue; and that all works fine with parity syncs and disk rebuilds.    But the key thing is the nr_requests change FIXES the issue, and doesn't seem to have any negative impact on performance otherwise.

 

[JorgeB]

...Apparently the most common controllers that have caused this issue are the SAS2LP and the 1430SA ... but I'm sure there are others (perhaps some motherboards that use Marvell secondary controllers to add extra SATA ports).

 

That’s interesting, for me the 1430SA has been very consistent since V5, but maybe the hardware used makes a difference.

 

1430SA with 4 SSDs (MB/s):

 

V5.0.6	V6.0.0	V6.0.1	V6.1.0	V6.1.1	V6.1.2	V6.1.3  V6.1.3 (nr_reqs=
213.4	214.9	213.4	213.4	213.4	214.9	214.9   209.3

 

 

Besides the SAS2LP, I did find some small gains on the SASLP with the tweak, back to V5 speeds.

 

SASLP with 8 SSDs (MB/s):

 

V5.0.6	V6.0.0	V6.0.1	V6.1.0	V6.1.1	V6.1.2	V6.1.3 	V6.1.3 (nr_reqs=
80.6	72.4	73.3	70.1	71.0	71.3	71.3	80.4

 

 

But anyone experiencing lower parity check speed since V5 should try it to see if it makes any difference.

 

 

[warhead]

My own test resulted in almost identical times, essentially 14 hours.  The change in nr_requests resulted in 3.5 minutes faster, which given so many other factors, is not statistically significant.  But it's one more data point that there isn't a negative effect using other controllers (SiI3132 and ASM1061).

 

During the early stages of the parity check on my SAS2LP setup (first 10-20% of the array maybe?) the change to nr_requests more than doubled the speed but after that it started tapering off and eventually settled on the old speeds.

After the parity check completed, the average speed and duration remained pretty much the same as before with leaving nr_requests as default at 128

[JorgeB]

My own test resulted in almost identical times, essentially 14 hours.  The change in nr_requests resulted in 3.5 minutes faster, which given so many other factors, is not statistically significant.  But it's one more data point that there isn't a negative effect using other controllers (SiI3132 and ASM1061).

 

During the early stages of the parity check on my SAS2LP setup (first 10-20% of the array maybe?) the change to nr_requests more than doubled the speed but after that it started tapering off and eventually settled on the old speeds.

After the parity check completed, the average speed and duration remained pretty much the same as before with leaving nr_requests as default at 128

 

Because you have 4 different size of disks it’s normal for the speed to vary greatly, at about 20% you will be on the last third of your 1.5TB disks, this will bring your speed down way down to below 100Mb/s, from 30% to 40% will be on the last quarter of your 2Tb disks, so again speed will be low, then from 40% to 60% you’re on the last third of you 3TB disks, speed should pick up a little after 60% but soon after you’re going to reach the inner tracks of your 5TB drives, so low speed again.

 

So, you’ll never have great parity check speed you so many different sizes, in your case the biggest difference with the tweak should be between 0 and 15% of your array.