[Partially SOLVED] Is there an effort to solve the SAS2LP issue? (Tom Question)

[RobJ]

FTP site to dump old versions

 

host ftp.diggsnet.com

user [email protected]

pw unraid

 

Thank you!  I've placed beta3, beta4, beta5, beta5a, beta6b, and beta6 there, that's date-issued order and their dates are very close to correct.  Let me know if any others are wanted.  I cannot remember what 6b was about, but it was about half way between 5a and 6.

 

I would ask anyone posting unRAID releases to always point to the Release Notes wiki page (MD5SUMs section), for the MD5's for all releases.  We want to avoid any possibility of a "tampered with" distro.

[Fireball3]

Rebooted server. Shut off all VM's. Turned as much off as I could. Started a fresh parity check and only getting 56MB/sec. Going to take over a day. Since I don't perform parity checks monthly, never really worried about the speeds but that's pretty slow. I'll see if I can grab those Dell H310's from work. On the wiki it indicates they were crossflashed with LSI9211 but the Dell Perc H310 has an LSISAS2008 on it. Weird. Don't want to brick any cards.

See this post to flash your H310 controller cards.

http://lime-technology.com/forum/index.php?topic=12767.msg259006#msg259006

 

With respect to my problem of reduced parity check speeds.

http://lime-technology.com/forum/index.php?topic=42629.msg406745#msg406745

 

I found a Core2Duo in my pool of spares and replaced the Celeron.

Fixed the problem, CPU usage is now at ~80% during parity check.

Speed ist nice around ~100MB/s.

The C2D TDP is 65W vs. 35W of the Celeron but as this is my backup machine

I'm fine with that also.

Don't know if there is something like that, but the requirements for unRAID should

mention a multi-core processor as a "must".

[JorgeB]

After some more testing I found that because I used different tunable settings in the beta 6 test the big drop off is not there, I’m sorry for that, there is some performance lost going from beta 3 to beta 6, but it is still a lot faster than beta 14, so it was not a single change up to beta 6, in fact, I re-tested all versions again to make sure the settings are all the same and it shows that it’s losing performance gradually since v5 beta12, so maybe it’s not a single setting that was changed.

 

Version  - AVG SPD

 

V5B12a  - 196.2

V5B14    - 184.4

V5RC16c - 125.1

V5.0.6    - 123.9

V6B1      - 125.1

V6B3      - 125.5

V6B4      - 120.9

V6B5a    - 120.2

V6B6b    - 110.6

V6B6      - 103.9

V6B14b  -  67.2

V6B15    -  58.4

V6.1        -  51.8

V6.1.1    -  51.8

 

 

 

 

 

 

Thank you!  I've placed beta3, beta4, beta5, beta5a, beta6b, and beta6 there, that's date-issued order and their dates are very close to correct.  Let me know if any others are wanted.  I cannot remember what 6b was about, but it was about half way between 5a and 6.

 

 

Rob when you have the time please upload betas 7 through 13, just in case there’s one version where I can identify a big drop in performance, if not I’ll give up on this issue.

 

At this point I would be very happy with an average speed above 100Mb/s, like the early V6 betas.

 

[mr-hexen]

Thinking out loud here.... is this possibly a CPU ramping issue? It's been reported that CPU is at 100% and getting a 2 core + CPU somewhat helps the problem...?

[JorgeB]

Thinking out loud here.... is this possibly a CPU ramping issue? It's been reported that CPU is at 100% and getting a 2 core + CPU somewhat helps the problem...?

 

Not in my case, in fact it’s the opposite, the slower the parity check, the lower the CPU load, but never goes over 35-40%.

 

For what I read in the forums from other users, when the CPU is pinned parity check is about 25Mb/s.

 

 

[mr-hexen]

This is a peculiar issue that's for sure. I wonder if Lime-Tech has been able to re-produce in their test systems?

 

What a HUGE difference in parity check speeds from early 5 RC's to 5.0 final even!!

 

 

@Johnnie, now I know you did not do FULL parity checks as there's no way you would have been done already, but If you sat at the webGUI hitting refresh that might slow them down... perhaps the tunables script is a less invasive test?

[Frank1940]

Thinking out loud here.... is this possibly a CPU ramping issue? It's been reported that CPU is at 100% and getting a 2 core + CPU somewhat helps the problem...?

 

Not in my case, in fact it’s the opposite, the slower the parity check, the lower the CPU load, but never goes over 35-40%.

 

For what I read in the forums from other users, when the CPU is pinned parity check is about 25Mb/s.

 

Perhaps this question has not been asked.  Are you keeping the GUI open in your browser while you are doing these tests?  If you are I would suggest starting the parity check, close the GUI window, and check back three or four hours later to see what effect the operation of updating the GUI has on the speed.  (I have not doubt there is some effect but the issue is "How much?". )

[JorgeB]

This is a peculiar issue that's for sure. I wonder if Lime-Tech has been able to re-produce in their test systems?

 

What a HUGE difference in parity check speeds from early 5 RC's to 5.0 final even!!

 

 

@Johnnie, now I know you did not do FULL parity checks as there's no way you would have been done already, but If you sat at the webGUI hitting refresh that might slow them down... perhaps the tunables script is a less invasive test?

 

All tests are full parity checks on my test server, I’m using small ssds so it’s quick.

 

I keep the browser closed during the check and monitor the processes in the console using top.

 

[RobJ]

Thank you!  I've placed beta3, beta4, beta5, beta5a, beta6b, and beta6 there, that's date-issued order and their dates are very close to correct.  Let me know if any others are wanted.  I cannot remember what 6b was about, but it was about half way between 5a and 6.

 

Rob when you have the time please upload betas 7 through 13, just in case there’s one version where I can identify a big drop in performance, if not I’ll give up on this issue

 

I've uploaded the additional betas plus some more, for anyone to test with.

 

But be aware these are old versions with bugs and incompatibilities compared to current systems!  Do NOT use on production systems!  Make sure you have a complete backup of your flash drive and restore it for each test, as they may make incompatible changes to it.  Even test systems with XFS or BTRFS drives should not be started with early v6 betas or any v5 release!  These releases are for testing only!  Do not allow any writes to the data drives.  Disable any plugins and dockers that might write to the array.

[TODDLT]

Thinking out loud here.... is this possibly a CPU ramping issue? It's been reported that CPU is at 100% and getting a 2 core + CPU somewhat helps the problem...?

 

I had this thought too, as I was seeing a ramping up and down of my CPU in V6 even at rest.  Someone (back in this thread) suggested it was due to "Scan User Shares" being turned on in Folder Caching.  I turned that off and still see the ramping issue occurring.  See these attached charts which were taken while the server is at rest.

 

i tried in V5 last night as well and this doesn't occur.

 

EDIT (oops had the wrong image, now corrected)

 

[JonathanM]

Thank you!  I've placed beta3, beta4, beta5, beta5a, beta6b, and beta6 there, that's date-issued order and their dates are very close to correct.  Let me know if any others are wanted.  I cannot remember what 6b was about, but it was about half way between 5a and 6.

 

Rob when you have the time please upload betas 7 through 13, just in case there’s one version where I can identify a big drop in performance, if not I’ll give up on this issue

 

I've uploaded the additional betas plus some more, for anyone to test with.

 

But be aware these are old versions with bugs and incompatibilities compared to current systems!  Do NOT use on production systems!  Make sure you have a complete backup of your flash drive and restore it for each test, as they may make incompatible changes to it.  Even test systems with XFS or BTRFS drives should not be started with early v6 betas or any v5 release!  These releases are for testing only!  Do not allow any writes to the data drives.  Disable any plugins and dockers that might write to the array.

I would suggest a password protected archive as a container for the known dangerous versions, (beta7 and 8?) with the password only given by direct communication. I would hate for someone to start playing around with one of the silent data corruption betas without full knowledge of what they are risking.

[TODDLT]

I also ran short parity checks last night in V5.0.5 and v6.1.1.  Both were with the SAS2LP card installed.  I am not seeing any appreciable increase CPU usage (maybe a little).

Sometime today I will try to put the SASLP in and see if it's any different. 

 

Here are V5. images

 

[jonp]

Thank you!  I've placed beta3, beta4, beta5, beta5a, beta6b, and beta6 there, that's date-issued order and their dates are very close to correct.  Let me know if any others are wanted.  I cannot remember what 6b was about, but it was about half way between 5a and 6.

 

Rob when you have the time please upload betas 7 through 13, just in case there’s one version where I can identify a big drop in performance, if not I’ll give up on this issue

 

I've uploaded the additional betas plus some more, for anyone to test with.

 

But be aware these are old versions with bugs and incompatibilities compared to current systems!  Do NOT use on production systems!  Make sure you have a complete backup of your flash drive and restore it for each test, as they may make incompatible changes to it.  Even test systems with XFS or BTRFS drives should not be started with early v6 betas or any v5 release!  These releases are for testing only!  Do not allow any writes to the data drives.  Disable any plugins and dockers that might write to the array.

I would suggest a password protected archive as a container for the known dangerous versions, (beta7 and 8?) with the password only given by direct communication. I would hate for someone to start playing around with one of the silent data corruption betas without full knowledge of what they are risking.

I would suggest not posting any links to beta 7 or 8...period.  That's just asking for trouble.

[TODDLT]

V6 images for my above post

[RobJ]

Thank you!  I've placed beta3, beta4, beta5, beta5a, beta6b, and beta6 there, that's date-issued order and their dates are very close to correct.  Let me know if any others are wanted.  I cannot remember what 6b was about, but it was about half way between 5a and 6.

 

Rob when you have the time please upload betas 7 through 13, just in case there’s one version where I can identify a big drop in performance, if not I’ll give up on this issue

 

I've uploaded the additional betas plus some more, for anyone to test with.

 

But be aware these are old versions with bugs and incompatibilities compared to current systems!  Do NOT use on production systems!  Make sure you have a complete backup of your flash drive and restore it for each test, as they may make incompatible changes to it.  Even test systems with XFS or BTRFS drives should not be started with early v6 betas or any v5 release!  These releases are for testing only!  Do not allow any writes to the data drives.  Disable any plugins and dockers that might write to the array.

I would suggest a password protected archive as a container for the known dangerous versions, (beta7 and 8?) with the password only given by direct communication. I would hate for someone to start playing around with one of the silent data corruption betas without full knowledge of what they are risking.

I would suggest not posting any links to beta 7 or 8...period.  That's just asking for trouble.

 

I would completely agree, but ALL of these older versions can be dangerous!  They absolutely should not be used!  Backward compatibility has NEVER been assured!  The flash MUST be restored after them.  No writing to the array should be allowed AT ALL!

 

In fact, I don't understand how he can even start the array with a v5 release and all XFS data drives.  I wonder if that skews the results.

 

And even mounting current XFS drives with older v6 betas could cause trouble to them, as the XFS mounting options have changed, are not necessarily backward compatible.

 

This is only a temporary availability, strictly for testing only, and then only with the understanding of the strong warnings above.

[TODDLT]

 

I would completely agree, but ALL of these older versions can be dangerous!  They absolutely should not be used!  Backward compatibility has NEVER been assured!  The flash MUST be restored after them.  No writing to the array should be allowed AT ALL!

 

In fact, I don't understand how he can even start the array with a v5 release and all XFS data drives.  I wonder if that skews the results.

 

And even mounting current XFS drives with older v6 betas could cause trouble to them, as the XFS mounting options have changed, are not necessarily backward compatible.

 

This is only a temporary availability, strictly for testing only, and then only with the understanding of the strong warnings above.

 

V6 defaults to XFS.  I replaced a couple drives but haven't added any.  Would they format XFS or stay as the drives they replaced?

 

EDIT: never mind, found my answer, they are RFS, not XFS

[JorgeB]

In fact, I don't understand how he can even start the array with a v5 release and all XFS data drives.  I wonder if that skews the results.

 

This was one of the first things I tested, the type of filesystem and the fact that drives are or not formatted has zero influence on the speed of a parity check. 

 

Thanks for uploading the betas, I’ll post the results later today.

 

 

 

[bkastner]

One question to the Masses.  What FS are you running for your array storage and cache?  I am running XFS for the array and 3 SSD's in a BTRFS raid pool.    Just wondering if this is an XFS file system issue or what.   

 

Checking out the release notes for the Linux kernal that was implemented for beta6 as well..  3.15.0 maybe there is something in there that started to hose us.

 

I am XFS across the board, and completed a parity check this morning after upgrading to 6.1.1

 

Last checked on Mon 07 Sep 2015 08:45:41 AM EDT (today), finding 28 errors.

Duration: 17 hours, 33 minutes, 40 seconds. Average speed: 94.9 MB/sec

 

This was my one on 6.1.0 on Sept 1st:

 

Last checked on Tue 01 Sep 2015 05:45:14 PM EDT (yesterday), finding 0 errors.

Duration: 17 hours, 45 minutes, 13 seconds. Average speed: 93.9 MB/sec

 

So, fractionally faster, but not much.

 

I have M1015s that will arrive later this week. Once I swap them out I will do another parity check and see how it goes with only swapping out the controllers.

[garycase]

Definitely interested in how the M1015's do.    Until the recent spate of issues with the SAS2LP's, that's definitely the card I'd have purchased if I was adding more SATA ports.    But now I'm gun-shy r.e. those cards and would likely avoid them ... either in favor of the older SASLP's or the M1015.

 

Fortunately I don't need any additional ports at the moment ... although I may around Christmas time, as I plan to build a new server this winter.

 

[WeeboTech]

I would suggest not posting any links to beta 7 or 8...period.  That's just asking for trouble.

 

I would agree.

The mere act of mounting reiserfs filesystems writes into the meta data which could cause the reiserfs silent corruption way into the future.

 

When reiserfs starts to fail in the future, it looks like a problem with the kernel. It's very hard to decipher the real issue.

 

Unless a tester is going to disable all mounting or reformat any reiserfs drives, I would stay away from the problem versions.

[mr-hexen]

Thank you!  I've placed beta3, beta4, beta5, beta5a, beta6b, and beta6 there, that's date-issued order and their dates are very close to correct.  Let me know if any others are wanted.  I cannot remember what 6b was about, but it was about half way between 5a and 6.

 

Rob when you have the time please upload betas 7 through 13, just in case there’s one version where I can identify a big drop in performance, if not I’ll give up on this issue

 

I've uploaded the additional betas plus some more, for anyone to test with.

 

But be aware these are old versions with bugs and incompatibilities compared to current systems!  Do NOT use on production systems!  Make sure you have a complete backup of your flash drive and restore it for each test, as they may make incompatible changes to it.  Even test systems with XFS or BTRFS drives should not be started with early v6 betas or any v5 release!  These releases are for testing only!  Do not allow any writes to the data drives.  Disable any plugins and dockers that might write to the array.

I would suggest a password protected archive as a container for the known dangerous versions, (beta7 and 8?) with the password only given by direct communication. I would hate for someone to start playing around with one of the silent data corruption betas without full knowledge of what they are risking.

I would suggest not posting any links to beta 7 or 8...period.  That's just asking for trouble.

 

Jon, have you guys been able to reproduce in your labs?

[jonp]

Thank you!  I've placed beta3, beta4, beta5, beta5a, beta6b, and beta6 there, that's date-issued order and their dates are very close to correct.  Let me know if any others are wanted.  I cannot remember what 6b was about, but it was about half way between 5a and 6.

 

Rob when you have the time please upload betas 7 through 13, just in case there’s one version where I can identify a big drop in performance, if not I’ll give up on this issue

 

I've uploaded the additional betas plus some more, for anyone to test with.

 

But be aware these are old versions with bugs and incompatibilities compared to current systems!  Do NOT use on production systems!  Make sure you have a complete backup of your flash drive and restore it for each test, as they may make incompatible changes to it.  Even test systems with XFS or BTRFS drives should not be started with early v6 betas or any v5 release!  These releases are for testing only!  Do not allow any writes to the data drives.  Disable any plugins and dockers that might write to the array.

I would suggest a password protected archive as a container for the known dangerous versions, (beta7 and 8?) with the password only given by direct communication. I would hate for someone to start playing around with one of the silent data corruption betas without full knowledge of what they are risking.

I would suggest not posting any links to beta 7 or 8...period.  That's just asking for trouble.

 

Jon, have you guys been able to reproduce in your labs?

Nope.

[mr-hexen]

Do you guys have SAS2LP's?

[JorgeB]

Tested some other betas, new results in blue, to confirm that filesystem as no impact on parity check for this round of tests all drives are Reiserfs formatted.

 

All results are of a full parity check, I ran some tests more than once with same version and results are very consistent, sometimes duration is the same down to the second.

 

Version -  AVG SPD - (*)

 

V5B12a  - 196.2 - (196.2)

V5B14    - 184.4

V5RC5    - 186.1

V5RC8a  - 126.7

V5RC12a - 146.1

V5RC16c - 125.1

V5.0.6    - 123.9

V6B1      - 125.1 - (189.7)

V6B3      - 125.5

V6B4      - 120.9

V6B5a    - 120.2

V6B6b    - 110.6

V6B6      - 103.9

V6B7      - 56.4

V6B8      - 55.6

V6B9      - 50.8

V6B10a  - 57.3

V6B12    - 63.8

V6B13    - 71.7

V6B14    - 69.7

V6B14a  - 67.3

V6B14b  - 67.2

V6B15    - 58.4

V6.0.1    - 57.3

V6.1        - 51.8

V6.1.1    - 51.8 - (196.2)

 

* Edited to add average speed on same server with 6 drives on onboard ports and 2 on adaptec 1430SA, without using the SAS2LP.

 

There are some interesting results, but nothing I can make sense of and I doubt this issue can be solved by changing a setting, but speed drops to half in beta 7, I’m more of a hardware guy, any ideas looking at the release notes?

 

btrfs-progs: update to 3.14.2
docker: update to 1.1.2
emhttp: assign any number of slots to array and cache pool
emhttp: implemented multi-device btrfs cache pool feature
emhttp: support btrfs, reiserfs, xfs filesystems for array devices
emhttp: handle multiple key files in /boot/config, but no more keys files in /boot
emhttp: add indicator for shares that only exist on the cache
libvirt: update to 1.2.7
linux: update to version 3.16
CONFIG_NR_CPUS: changed from 16 to 256
CONFIG_XFS_FS: include XFS file system
CONFIG_XFS_POSIX_ACL: include XFS ACL support
add "pci: Enable overrides for missing ACS capabilities" patch (for KVM vfio/PCI pass-through)
misc: console coloring for kool kids
php: update to 5.4.30
add support for curl, openssl, zlib, bz2 and gmp
qemu: update to 2.1.0
add "Enable -cpu option to hide KVM" patch (for Nvidia)
samba: update to 4.1.11
shfs: fixed improper handling of cache floor
slack: added htop-1.0.2
slack: added xfsprogs-3.1.11
unraid: correct overlapped I/O logic in driver; remove write-throttling algorithm
webGui: add "check filesystem" functions on DiskInfo page.
webGui: several misc. bug fixes

 

At this point I’d be very happy with a speed similar to beta 6, but I’m prepared to accept there’s nothing Limetch can do about this and move on.

[BRiT]

Items that stand out that could be double-checked --

 

linux: update to version 3.16

unraid: correct overlapped I/O logic in driver; remove write-throttling algorithm