Slow Transfers / Parity Checks? I have a theory

[DoeBoye]

Hi folks,

 

I have a theory re: the newer linux kernels in the v5 release candidates: I noticed that my transfer speeds had dropped significantly, a few rcs ago, and after much exploration, found that my unraid system was now defaulting to the onDemand governor of my AMD X4 CPU. On this setting, all 4 cores dropped to 800mhz, and would not always speed up when transfers started. (Note: Also, apparently the up_threshold value was increased some kernels ago, so the load needs to be greater before the cores speed up)

 

I set the governor on all 4 procs to 'performance', which forced cpus to max speed (2200mhz for me), and my transfer speeds increased substantially (an increase of around 30 MB/s). I haven't tested this fully yet with parity checks, but was thinking that perhaps this is why some people are seeing much slower transfers/parity checks with these newer release candidates.

 

If anyone would mind doing a quick check, I'd be very curious to see if it is affecting others. Here's the info I think is important:

 

1. Are you finding transfers and/or parity checks to be slower then before (with v5beta and 4.7) and what version are you on?

2. What CPU do you have?

3. What governor are you currently running?

4. Does your cpu scale up in speed when transferring files over the network?

5. Does switching your cpu to the  'performance' governor change the speeds you are seeing?

 

Here's the commands that work for me. This may not be the same for all cpus. A quick search will probably give you the right one if it doesn't:

 

3. What governor:

cat /sys/devices/system/cpu/cpu0/cpufreq/scaling_governor

4. Does your cpu scale up the speed while transferring: I use the cpu info page of unmenu

5. Switch governor:

sudo sh -c "echo performance > /sys/devices/system/cpu/cpu0/cpufreq/scaling_governor"
sudo sh -c "echo performance > /sys/devices/system/cpu/cpu1/cpufreq/scaling_governor" 
sudo sh -c "echo performance > /sys/devices/system/cpu/cpu2/cpufreq/scaling_governor" 
sudo sh -c "echo performance > /sys/devices/system/cpu/cpu3/cpufreq/scaling_governor"  

NOTE: Modify to match the number of cores your cpu is running.

 

Here's the original thread when I first noticed and wanted to modify the up_threshold so the cores speed up at lower loads:

 

http://lime-technology.com/forum/index.php?topic=22206.msg196705#msg196705

 

This may be completely off base, but I figured it would be worthwhile finding out if I'm a unique case or not. I have a feeling the results might be surprising...

[jumperalex]

Is there another governor available besides performance?  Something that will scale sooner?  I sure don't want my CPU ramped up all the time.

[jumperalex]

Seems there are only those two ... so I tried it ... no transfer difference writing to the cache drive.  62MB/s according to terracopy.  With 4.7 and early betas I was getting on the order of 80-90 with spikes to 100. 

[DoeBoye]

I'm not entirely sure if it changes depending on the cpu. In my case, I only have onDemand and performance available. That said, running at 'full tilt' is certainly not harmful to the cpu, as that is the speed it was built for. The downside is the increased electricity usage. My system pulls about 10 more watts on performance, as opposed to onDemand.

 

There is a way to set ondemand to me more 'sensitive' to load, but unfortunately it was not included in this kernel (see other thread i linked to). If we had the option to change the up_threshold value, we could use ondemand, and have it speed up at lower loads (Currently my understanding is it is set to ramp up at 90%. I would set it down to something like 30%).

 

I have emailed Tom to see if it would be a simple matter to include in the next iteration, but I certainly don't want to if it delays in any way 5.0 final!

 

In my eyes, that would be the best of both worlds (And how it used to be in earlier Linux kernels). Power when required, but electricity savings otherwise.

 

If that can't happen, I believe we *can* set the min frequency value. This sets the lowest frequency the cpu is allowed to drop to in ondemand mode. In my case, I figure if I set that to 1200mhz-1600mhz instead of 800mhz, I might not see the drop in performance and still save a bit of electricity...

 

Though I have yet to test it, this is the command that allows one to change this value:

sudo sh -c "echo 1200000 > /sys/devices/system/cpu/cpu0/cpufreq/scaling_min_freq"

[DoeBoye]

Interesting. How low does your cpu's frequency drop to when using the ondemand governor?

[jumperalex]

Yeah I know it can handle it.  but my cooling is based on minimal thermal load and a desire to keep noise, sound, and dust to a minimum  

 

It idles st 1350mhz and hits 2700mhz on performance.  During parity checks and transfers, the cpu stays at 1350 with ondemand and I can see 2700 of course when I change to performance.  But even more strange ... see my new post in the RC thread.  I'll give you the highlight: when testing the governor theory I saw 110+MB/s parity check from start through 3.9% with the ondemand gov

[DoeBoye]

Yeah I know it can handle it.  but my cooling is based on minimal thermal load and a desire to keep noise, sound, and dust to a minimum  

 

It idles st 1350mhz and hits 2700mhz on performance.  During parity checks and transfers, the cpu stays at 1350 with ondemand and I can see 2700 of course when I change to performance.  But even more strange ... see my new post in the RC thread.  I'll give you the highlight: when testing the governor theory I saw 110+MB/s parity check from start through 3.9% with the ondemand gov

 

Very odd... Wish I could help explain that!! .

 

re: min frequency. Looks like my cpu drops much lower then yours does, but at least it is also not ramping up when transferring so looks like the up threshold is the same...

 

Would you be interested in using the code i included above to set your minimum frequency lower (I would do it in steps to make sure there are no issues) to see at what point your transfers start to get affected? If you do, keep in mind to set both your cpus (cpu0 and cpu1)

[jumperalex]

I'm willing ... but I don't know of my CPU supports lower speeds ... here goes nothing

[jumperalex]

"scaling_available_frequencies — Lists available frequencies for the CPU, in KHz"

 

and mine show 2700000 and 1350000  I think this CPU just doesn't have many options.  I remember that from cpuID when it was part of my desktop

[DoeBoye]

Looks like mine has a couple extra options (800, 1400, 1700, 2200). I tried at 1400mhz, and my transfers are at full speed.

 

I have a feeling this issue I'm experiencing may only affect those with low power cpus that scale back substantially.

 

If anyone running a Sempron would care to run the test and post back. Here's the command to find out frequencies supported that jumperalex was talking about:

sudo cat /sys/devices/system/cpu/cpu0/cpufreq/scaling_available_frequencies

 

I was really hoping this might be the problem, but I have my doubts now...

[Influencer]

I was interested so I gave this a try.

 

I am running a sempron, and my available scaling freq's are :

2800000 / 2800MHz

2000000 / 2000MHz

1600000 / 1600MHz

800000  /  800MHz

 

With set ondemand, the frequency bounces between 800 and 2800 during the parity check. Setting to performance it of course stays steady at 2800MHz.

 

I of course just started the parity check to test this theory for you, but in the initial start of the test I noticed the following:

Last night when doing a parity check it started at ~40MB/s. It took a while to go up, but eventually I got a maximum of around 94MB/s and an average of 75 MB/s.

 

Tonight, I started the parity check with ondemand still set, and checked my current freq via unmenu. It was bouncing between 800MHz and 2800 MHz. My initial parity check speed was ~50MB/s. I switched to telnet and set the scaling to performance, double checked in unmenu that it was pegged at 2800MHz constantly, and then checked my parity speed, for the past few minutes it has sat consistently at ~99MB/s, and as I just checked tipped over to 101 MB/s, which is the fastest I've seen on this build.

 

I think you are onto something, may not be the full solution, but is definitely an option for users seeing slower speeds.

 

[jowi]

My Intel i3 doesnt show anything when i use the governer command, i guess this i a specific AMD problem? As far as i can see, everyone who reacts here has some AMD cpu...

[DoeBoye]

This is interesting... Thanks for testing this out JumperAlex + Influencer! If anyone else has some spare time to test transfers and parity checks, I'd love to see the results.

 

Definitely do the available freq. check and include it in your postings.

 

Maybe this does have some impact on speeds, even if it isn't the cure-all .

 

I would love to be able to set the up_threshold lower so I don't need to run my server with the performance governor all the time. If it could spin up to max speed at something like 30% load instead of 95% (Like it appears to be currently), it would be the perfect combination of power saving and performance!

[DoeBoye]

My Intel i3 doesnt show anything when i use the governer command, i guess this i a specific AMD problem? As far as i can see, everyone who reacts here has some AMD cpu...

 

It's probably just 'worded' differently, as Intel does have speedstep...

 

I'll look into it and see if I can find the correct wording for Intel and post back so we can get Intel folks in on the fun

[DoeBoye]

A thought: Anyone trying this, make sure you have it enabled in your bios. It would be called something like:

 

Speedstep, Cool and Quiet, PowerNow!, or ACPI

 

 

 

[DoeBoye]

Jowi: What version of UnRaid are you using? If I remember correctly, power saving features were not included by default in earlier versions.

[drawz]

I never could get my Atom 230 to Speedstep in unRAID - the proper kernel module wasn't included as far as I could tell. This makes me wonder if other Intel CPUs are not adjusting their clock speed, which in turn might explain the discrepancy in parity/transfer rates?

 

I switched to an HP N40L Microserver (AMD Turion N40L), which does down clock to 800MHz by default in RC6 as described by others. Despite being a faster processor, it does seem to run parity check/sync more slowly. Of course, this is with a different controller and CPU combo. I did actually upgrade to a faster parity drive with the Microserver build, so it's especially odd that it would be slower. The unRAID build remained the same (RC6-test2).

 

If this really is the issue, I agree that the solution is not to simply run at maximum speed (and therefore heat/power), but to improve the scaling governor.

[jowi]

Jowi: What version of UnRaid are you using? If I remember correctly, power saving features were not included by default in earlier versions.

v5.0 rc5.

[PeterB]

My Intel i3 530 has ten frequencies available, ranging from 1197MHz up to 2926MHz.  The scaling governor is set to 'ondemand'.

 

Even during parity check, running at over 110MB/s, the processor speed is reported as 1197MHz.

 

During the parity check, top reports my cpu usage at around 2%.

 

I guess that my system is way over specified for basic unRAID use!

 

However, this doesn't explain why I was seeing serious performance issues on an earlier RC when others were reporting good performance.  Perhaps that was an issue with specific sata interface drivers?

[kizer]

On my Sempron 140 the farthest I can go with that path you posted is

/sys/devices/system/cpu/cpu0

 

I'm running

v5.0 rc5

 

 

Looks like a ModProbe gave me a little info regarding my failure to find that path.

http://lime-technology.com/forum/index.php?topic=12292.msg120250#msg120250

 

sudo cat /sys/devices/system/cpu/cpu0/cpufreq/scaling_available_frequencies

Available speeds are:

2700000

1900000

1500000

800000

[downloadski]

Jowi: What version of UnRaid are you using? If I remember correctly, power saving features were not included by default in earlier versions.

v5.0 rc5.

On my system it seems to be:

:/sys/devices/system/cpu/cpu0/cpufreq# cat scaling_available_frequencies

2501000 2500000 2400000 2300000 2200000 2100000 2000000 1900000 1800000 1700000 1600000

 

so it under the path: /sys/devices/system/cpu/cpu0/cpufreq#

 

i find:

 

root@Tower2:/sys/devices/system/cpu/cpu0/cpufreq# cat scaling_available_governors

ondemand performance

 

root@Tower2:/sys/devices/system/cpu/cpu0/cpufreq# cat scaling_governor

ondemand

 

root@Tower2:/sys/devices/system/cpu/cpu0/cpufreq# cat scaling_min_freq

1600000

 

root@Tower2:/sys/devices/system/cpu/cpu0/cpufreq# cat scaling_max_freq

2501000

 

root@Tower2:/sys/devices/system/cpu/cpu0/cpufreq# cat scaling_setspeed

<unsupported>

 

root@Tower2:/sys/devices/system/cpu/cpu0/cpufreq# cat scaling_cur_freq

1600000

 

 

 

my system does parity check and is on the lowes setting it seems:

 

model name      : Intel® Xeon® CPU E3-1265L V2 @ 2.50GHz

stepping        : 9

microcode      : 0x12

cpu MHz        : 1600.000

[downloadski]

if i put performance in the scaling governor:

 

root@Tower2:/sys/devices/system/cpu/cpu0/cpufreq# echo performance > scaling_governor

 

 

I see the speed go up:

 

root@Tower2:/sys/devices/system/cpu/cpu0/cpufreq# cat /proc/cpuinfo

processor      : 0

vendor_id      : GenuineIntel

cpu family      : 6

model          : 58

model name      : Intel® Xeon® CPU E3-1265L V2 @ 2.50GHz

stepping        : 9

microcode      : 0x12

cpu MHz        : 2501.000

cache size      : 8192 KB

 

[DoeBoye]

if i put performance in the scaling governor:

 

root@Tower2:/sys/devices/system/cpu/cpu0/cpufreq# echo performance > scaling_governor

 

 

I see the speed go up:

 

root@Tower2:/sys/devices/system/cpu/cpu0/cpufreq# cat /proc/cpuinfo

processor      : 0

vendor_id      : GenuineIntel

cpu family      : 6

model          : 58

model name      : Intel® Xeon® CPU E3-1265L V2 @ 2.50GHz

stepping        : 9

microcode      : 0x12

cpu MHz        : 2501.000

cache size      : 8192 KB

 

Are you seeing any performance change during parity checks or transfers? I'm guessing, based on what we've seen so far, that with your cpu running at 1600 or 2500mhz, there's no difference, as you're minimum frequency is already enough to easily keep up...

[DoeBoye]

On my Sempron 140 the farthest I can go with that path you posted is

/sys/devices/system/cpu/cpu0

 

I'm running

v5.0 rc5

 

 

Looks like a ModProbe gave me a little info regarding my failure to find that path.

http://lime-technology.com/forum/index.php?topic=12292.msg120250#msg120250

 

sudo cat /sys/devices/system/cpu/cpu0/cpufreq/scaling_available_frequencies

Available speeds are:

2700000

1900000

1500000

800000

 

I kept meaning to post that people might have to modprobe to get it to work, but got distracted and never ran down the proper code  . Thanks for the link!

 

I had to do that a few years ago, when last I messed around with cpu frequencies, but it looks like it now works by default for me...

 

I'm assuming you did a

modprobe powernow-k8

to get your results right?

 

For those with Intel, as per the link kizer provided, give

 

modprobe acpi-cpufreq

a try if the initial command does not work.

[downloadski]

Are you seeing any performance change during parity checks or transfers? I'm guessing, based on what we've seen so far, that with your cpu running at 1600 or 2500mhz, there's no difference, as you're minimum frequency is already enough to easily keep up...

Sorry, the parity check was just done (64 Mbyte/sec average)

I switched between the 2 speeds while transfering to my cache drive. But that did not seem to make a difference.

However, i have a hitachi 1 TB 7200 RPM disk, and i suspect the 72 Mbyte/sec speed i see while copying is about the maximum it can do. (this is stable speed with Blu ray files of 10-20 GB)

 

I have a 2 port adapted raid card coming to me, i will try that with 2 of these 1 TB drives in raid-0 to see if i get more speed.