unRAID OS Version:

 

Description:

 

Parity Checks *can* take longer (~10% is what I'm seeing in one example) under 6.0-beta12 compared to 6.0-beta10a times.

 

How to reproduce:

 

Compare Parity Check times on the same array, one with beta10a and the other with beta12.

 

Expected results:

 

I'd expect Parity Check times to be "comparable" on the same array between the two releases.

 

Actual results:

 

My most recent results are as follows:

 

6.0-beta10a Parity Check time=35734sec

Duration: 9 hours, 55 minutes, 34 seconds. Average speed: 112.0 MB/sec

 

6.0-beta12 Parity Check time=39319sec

Duration: 10 hours, 55 minutes, 19 seconds. Average speed: 101.8 MB/sec

 

Other information:

 

In the small amount of experimentation I've undertaken so far, the problem seems to manifest more so for high-throughput situations (around 2GB/s total) than for less-demanding configurations.

 

My primary server is 16 x 4TB, in two SAS enclosures each connected via SFF-8088.  Theoretical speeds then would be 24Gb/s, or 2.4GB/s.  Of course there will be other sources of overhead that will chip away at this number.

 

Under beta10a I see ~135MB/s (peak) Parity Check speeds, times 16 is 2.16GB/s.  Under beta12 I could only see low 120MB/s peak Parity Check speeds, times 16 is around 1.95GB/s.

 

The only changes between the two runs were to the bzimage and bzroot files the server boots from.

 

If there is a beta11 release I would be willing to try it, to help determine between which two releases the performance loss was introduced.

 

This "investigation" started when I upgraded from beta10a to beta13 and found it to be even slower than beta12 (which I subsequently downgraded to and tested).

 

Ideally we could have the nice new features I see in the beta12 and beyond releases, but with the previous levels of disk performance.  I'm hoping not to have to choose between the two.

 

I ran some tests on my backup server (20 x 3TB) and found these peak Parity Check speeds:

 

beta10a: 67.x MB/s

beta12: low 60s MB/s

beta13: low 40s MB/s

 

Let me know if there is any other information I can provide to help with this.

Have you tried this: http://lime-technology.com/forum/index.php?topic=29009.0 to see if there's any speed to be had?

 

 

Hi Squid, that looks like an interesting tool.  I will definitely check it out.

 

My immediate thought is: did those disk tuning parameters *change* between OS releases?  If so that could easily explain what I'm seeing . . . if not then it *seems* like it ought to perform as before, assuming no additional overhead was introduced elsewhere (i.e. unrelated to the tuning parameters).

 

I just checked my main server, which is back to beta10a, and two of the tunable parameters are shown under Disk Settings:

md_num_stripes: 1280

md_sync_window: 384

 

Possibly the third parameter (md_write_limit) was introduced after beta10a?

 

I guess I'll fire up my backup server under beta12 and see what values those parameters have there.  If they're different I can try setting them to the default beta10a values above to see if that helps.  Although I don't know what I'd want to set the third parameter to (md_write_limit) since I'm not seeing that value reported under beta10a.

 

(Main server is busy doing a Data Rebuild, having swapped out the last WD drive for a Seagate.)

I don't think that those tunables have changedin years.  As Pauven states:

Since unRAID servers can be built in a limitless number of configurations, it is impossible for Tom and Lime-Technology to know what tunable parameters are correct for each system.  Different amounts of memory and various hardware components (especially HD controllers) directly affect what values work best for your system.  To play it safe, Lime-Technology delivers unRAID with 'safe' stock values that should work with any system, including servers with only 512MB RAM.

 

The third parameter used to exist, but I believe it was dropped somewhere along the line in v6 (but the script itself still works - just ran it yesterday against my 2 b13 servers

 

But, the difference in speed could be a million different things.  Kernel versions, background processes, RAM usage, etc.

 

I wouldn't expect much difference in your main server, but your backup server might show a big increase. 

But, the difference in speed could be a million different things.  Kernel versions, background processes, RAM usage, etc.

 

I tried as much as possible to limit those variables.  I have no plugins or other tasks running during the Parity Checks.  Same machine, RAM, etc. between the two test results.  Only different bzimage/bzroot files.

 

I very repeatably get within a minute or two of my Parity Check times on the same OS release, so I have to believe I'm *not* seeing run-to-run variations in performance.

 

Kernel version I suppose is a possibility; I see 3.16.3 versus 3.17.4 there.

 

I guess I can try tuning those parameters to see what effect that has.

 

Ideally someone who knows what changed between those versions relating to disk throughput/performance can chime in and either say nothing changed there or point out a possible explanation.

 

I'll also go through the changelog myself in search of something like that.

 

Squid, is it the case that your Parity Check times weren't impacted by your upgrade from beta10a (assuming you ever ran that version)?

To my knowledge, this is the only explanation of what the settings are / do is: http://lime-technology.com/forum/index.php?topic=4625.msg42091#msg42091

 

Honestly, I've never really paid a whole lot of attention to the raw speeds, so I couldn't tell you if between 10 and 12/13 it dropped.  On the first of the month, I just check in the morning to see that it completed in roughly the same time period.  To me, a 1 hour difference is roughly the same time period.

 

 

All I do know is that prior to that script being introduced, I tried playing by hand with those tunables, and never really got anywhere.

 

After the script, my one server went from ~80MB/s to ~115MB/s peak.  Definitely a noticeable improvement in time to completion.

 

My older server went from ~40MB/s to ~90MB/s peak.  A Huge improvement in time to completion.

 

Try the script and see what it does for you

Thanks for your help Squid; I will definitely try that tuning script, although I feel it's orthogonal to what I'm seeing.

 

After powering up my backup server with beta12 I noticed the CPU load fluctuating despite no activity on my part.  On my other servers the CPU is pretty flat when I'm not using them (no Dashboard there, just going by load averages reported by, for example, the 'top' command).

 

Sure enough, 'top' reports wildly varying load averages on my backup server under beta12, even when sitting idle, all disks spun down.  Started to suspect Dynamix.  Found this topic where mention is made of Dynamix potentially impacting Parity Check times: http://lime-technology.com/forum/index.php?topic=30939 . . . admittedly there did not seem to be concurrence that it could actually have an impact, but I didn't read all 80+ pages.

 

Dug into it a bit more and found lots of smartctl processes coming and going, no doubt part of Dynamix trying to keep the displayed SMART status of the drives current.  I have another, "testbed" server, currently with just two drives installed, I use it to test drive speeds, controller speeds, etc., and I booted it with beta12, and I don't see the same problem, so that leads me to believe the large number of drives on my backup server helps multiply the smartctl overhead.

 

As an experiment (which was suggested in that topic I linked to above) I moved the smartctl binary aside, and sure enough, the load averages came down.  There was still the occasional bump, which seemed to correlate with a php process coming and going, and I attributed that to the web GUI, so I closed the browser window into that, and now my load averages are pretty flat at 0.0.

 

Next step is to try a Parity Check under these experimental conditions (smartctl binary moved aside and close the browser on the web GUI).  I see that there is an option on the page update frequency to 'disable page updates while parity operation is running', I'm pretty sure I had that turned on for my earlier tests that took an hour longer to run.

 

I understand that the time a Parity Check takes is not *that* important, my concern is more about how long a Data Rebuild takes, because the entire time one of *those* is running, the array is vulnerable to a drive failure.  10% isn't that much but I wouldn't call it negligible either.  I went to great lengths to get my Parity Check times down as low as possible (drives, controllers, motherboards), and was finally satisfied when I got it under 10 hours, so to lose an hour after all that effort is disheartening.

The widely varying load averages is actually a bug in the intel pstate drivers. There is a patch to fix that by disabling it via the syslinux config file. This has greatly stabilized my load averages even with dynamix doing its hdparm thing.

The widely varying load averages is actually a bug in the intel pstate drivers. There is a patch to fix that by disabling it via the syslinux config file. This has greatly stabilized my load averages even with dynamix doing its hdparm thing.

Unless you have RealTime updates turned on when parity checks are active (settings/display settings)

The widely varying load averages is actually a bug in the intel pstate drivers. There is a patch to fix that by disabling it via the syslinux config file. This has greatly stabilized my load averages even with dynamix doing its hdparm thing.

Unless you have RealTime updates turned on when parity checks are active (settings/display settings)

 

As an I said originally, even with dynamix doing its thing, ive seen greatly stabilized load averages and cpu speeds with the intel pstate syslinux fix. The cpu throttling up and down isnt an issue and has no impact.

 

The hdparm havin an impact on parity check is an entirely different matter. Its not cpu load related.

I've narrowed it down to those numerous smartctl invocations, I see a group of them every ten seconds, I'm guessing one per drive in the array.

 

Here is the latest experiment:

- boot unRAID 6.0-beta12 on primary server (16 x 4TB)

- make sure Dynamix page updates are completely disabled

- initiate a Parity Check

- check estimated speed over a period of several minutes: ~122MB/s

- check 'top' for presence of smartctl invocations: confirmed

- move smartctl binary aside

- check 'top' for absence of smartctl invocations: confirmed

- check estimated speed over a period of several minutes: ~136MB/s

 

So it looks like at the very worst I could use beta12 (and beyond) if I also use the workaround of moving the smartctl binary aside before a Parity Check (or Parity Sync, or Data Rebuild, or disk clearing operation).  And of course move it back in place after the disk-heavy operation to allow the GUI to continue to monitor the drives.

 

*Ideally* there could be a way to have those disk-heavy operations disable (or at least throttle way back) those repeated multiple smartctl invocations while they are in progress.  I understand the need to check on the health of the drives periodically.  I suggest that such checking be balanced against other considerations, like disk throughput speeds.

An easier way would be to maybe disable system notifications.  Not sure, but it may not make all those smartctl calls

Is that something I can do via settings or do you mean that unRAID could be changed to disable system notifications during disk-heavy operations?

 

I'm looking under Notification Settings and the closest I can see to disabling it is 'Notification entity: None', which is what it apparently defaults to since I've not made any changes there.

Not sure, but I'm thinking system notifications.  Notifications entities just states how the notifications themselves are delivered.

 

I don't believe there currently is any method for disabling it during disk-heavy operations.

 

[thinking aloud here]

I suppose if you're bash savvy you could change rename smartctl and make a script in its place to only run the executable if a parity check / rebuild is not running.

Does your system have a Haswell processor?

 

There have been some issues with CPU throttling/scaling with the Haswells.  Beta 14 has a couple changes related to this, although I don't believe the issue is fully resolved yet.  Update to that and see if this helps.

 

Squid: I could build such a script *if* I knew how to test if a parity check/etc. is running.

 

Garycase: My primary server *does* have a Haswell processor (i3-4330).  But the problem also happens on my backup server, which is Xeon-base (L5420).

 

Downloading beta14 now.

 

 

I believe that this will do it:

 

cat /proc/mdcmd | grep 'mdResync='

If the value > 0, a parity rebuild / check is in progress

 

Bonienl is the outright expert on this stuff however.

Thanks Squid.

 

It seems to work to move the smartctl binary to (for example) Xsmartctl, and put this script in its place:

 

#!/bin/bash
if grep ^mdResync=0\$ /proc/mdcmd
then
  Xsmartctl
fi

 

Now I just need to add the couple of commands to do that to my go script.

 

I'm still hoping that something can be done officially in unRAID to get the lost performance back.

As long as you're aware that while you're doing that, you also will not get any over temperature warnings, etc

As long as you're aware that while you're doing that, you also will not get any over temperature warnings, etc

 

Yep.  Managed for 3+ years without them, that's *ever*, not just only during syncs/rebuilds.  And I've yet to lose a drive (in an unRAID array), with nearly 70+ drives in play all told.  Did lose a RAID5 enclosure once and got the contents back with a rebuild.

 

Thanks again for your help Squid, garycase, and BRiT; I still have the tunables-tester and beta14 to try, but I feel that I have successfully worked around this disk throughput problem.  I do still hope that a future release allows me to remove the workaround.

Thanks Squid.

 

It seems to work to move the smartctl binary to (for example) Xsmartctl, and put this script in its place:

 

#!/bin/bash
if grep ^mdResync=0\$ /proc/mdcmd
then
  Xsmartctl
fi

 

Now I just need to add the couple of commands to do that to my go script.

 

I'm still hoping that something can be done officially in unRAID to get the lost performance back.

 

The calling of smartctl happens as part of emhttp, will ask / work together with LT to see how this can be improved, as in: how to avoid performance impact.

 

The calling of smartctl happens as part of emhttp, will ask / work together with LT to see how this can be improved, as in: how to avoid performance impact.

 

Great, thanks bonienl!

 

I've been satisfied with the features and performance of unRAID for many years, to me one of the attractions is its "lean"ness, it's not a big, bloated solution.  With leanness typically comes good performance, but when overhead creeps in, that attraction is diminished.  Fortunately I can work around it for now, and hopefully the performance loss can be regained.

 

To Lime Technology: keep up the good work!

Just finished the Parity Check under beta12 with smartctl moved aside:

 

Duration: 9 hours, 26 minutes, 46 seconds.  Average speed: 117.6 MB/sec

 

That's 31 seconds less than the previous Parity Check, under beta10a:

 

Duration: 9 hours, 27 minutes, 17 seconds.  Average speed: 117.5 MB/sec

 

I'd say those results pretty well prove that the repeated/numerous smartctl invocations were the reason for the slowdown.

 

(For those wondering where I gained nearly half an hour over my previously-reported times for this array, I recently swapped out the last WD drive in the array for a Seagate.)

That are great average speeds

 

That are great average speeds

 

Thanks!  I worked hard for those.

 

I just realized a couple of things about my workaround, one minor, the other not:

- that script could be improved slightly with a -q option to grep

- emhttp is apparently exec'ing smartctl, and doesn't like it being a script as opposed to a binary executable.

 

So I've broken my drive temperature readouts in the web GUI.

 

I guess I'll put the real smartctl back in place and resort to manually moving it aside before a disk-heavy operation.  Bummer.