Throttle parity sync/check according to I/O


Recommended Posts

just provide min and max throttling settings to allow it to be tuned

 

This is how the linux md drivers have done it in the past with success.

 

When needed, you could adjust minimum and maximum levels. by echoing a value into /proc/

 

before unraid, I used this functionality to adjust values as I needed them on demand.

It worked fine for linux with the standard md driver, it would work fine here for unRAID.

I think we might be over analyzing this.

With these two values, the end user has what he needs to provide the service level required.

I'm sure examples of this could be found in various linux md.c/raid?.c driver code.

 

examples:

 

[pre]cat /proc/sys/dev/raid/speed_limit_max 200000 cat /proc/sys/dev/raid/speed_limit_min 1000

md: minimum _guaranteed_ reconstruction speed: 1000 KB/sec/disc. md: using maximum available idle IO bandwidth (but not more than 200000 KB/sec) for reconstruction.

echo 50000 >/proc/sys/dev/raid/speed_limit_min

 

watch cat /proc/mdstat

[/pre]

Link to comment

So it's already there? Just add webgui parameters to adjust it?

 

No it's not in the unRAID drivers/kernel.

The examples are of the standard linux md driver kernel interface.

I'm not sure what version this example represents.

 

I posted it because,

A. It's been thought of before by the kernel raid developers.

B. If it's worked fine for software raid all these years, it could work fine for unRAID.

C. by using these two values with a webgui interface, the end user could adjust as he sees fit.

 

If some value parity check/sync/generate speed over usability adjust accordingly.

If some value usability vs a check operation, adjust accordingly.

 

Link to comment

So it's already there? Just add webgui parameters to adjust it?

 

No.  What I would implement is a 'min' setting.  Set to 0 means "let all non-parity sync/check I/O have complete priority", and set to 100 would cause it to run the way it does now (generate parity sync/check I/O as fast as possible).

 

This is something I've wanted to put in for a while now in order to prevent parity check from interrupting media playback.

Link to comment

So it's already there? Just add webgui parameters to adjust it?

 

No.  What I would implement is a 'min' setting.  Set to 0 means "let all non-parity sync/check I/O have complete priority", and set to 100 would cause it to run the way it does now (generate parity sync/check I/O as fast as possible).

 

This is something I've wanted to put in for a while now in order to prevent parity check from interrupting media playback.

 

So if nothing else is going on, would parity check /sync/rebuilds work at max speed?    ... or will they be limited to the percentage set by this parameter?

 

Link to comment

So it's already there? Just add webgui parameters to adjust it?

 

No.  What I would implement is a 'min' setting.  Set to 0 means "let all non-parity sync/check I/O have complete priority", and set to 100 would cause it to run the way it does now (generate parity sync/check I/O as fast as possible).

 

This is something I've wanted to put in for a while now in order to prevent parity check from interrupting media playback.

 

So if nothing else is going on, would parity check /sync/rebuilds work at max speed?    ... or will they be limited to the percentage set by this parameter?

It would go at max speed.  What I want to do is introduce delays in parity sync/check stripe generation based on volume of normal I/O to array disks in conjunction with this new setting.

Link to comment

I am afraid that the implementation of this new idea could create problems.  If history has thought us anything, is that any new features around here end up bundled up into the md-mod driver making a mess of it.

 

If this is to be done right, we have to separate the ability to throttle from the I/O monitoring:

 

(1) First, implement a simple throttling ability in the md-mod driver, and expose the setting through /proc/mdcmd -- say, an integer number from 0 to 100 signifying the percentage of full speed desired for the check/build. (zero can have a special meaning).  The throttling has to be done right, making sure that it not only slows things down, but effectively releases resources to other processes. (whatever sleep in kernel space means).

 

(2) Only after you've done and tested (1) then you think of writing some daemon (in user space) that will monitor I/O and adjust the throttling setting accordingly.  The user has the option to not even start that daemon, and their system will run check/build at 100% speed, like before.  Or, the user can just manually (from the go script) set the trottle setting to 80 or something and again not bother with starting any realtime I/O monitoring.

 

To summarize my personal preference:

- I would like to have a configurable throttle;

- I don't need the real time I/O monitoring and adjusting.

 

As I will want to be able to turn off the latter, I don't want these two bundled together.

 

 

Link to comment

I really would like this feature, and it seems in high demand. I actually adjust my tunables so I have a little headroom because of this, which makes my parity checks ~2 hours longer. Watching BD ISOs during parity checks is impossible otherwise.

 

I would put this feature right up there with 64bit. Two features that we could use. The lowmemory issue caused by 32bit is also problematic, since many drives need much higher tunables to get the fastest parity checks possible. However, without this setting, many people would probably still not want to run them that high.

Link to comment

Whilst your doing that could you look at partial checks?

 

For example, do 100% over 7 days, so 14.5% a day?

 

I wake my server at 7AM to run mover and then directly after do some more of a check.

 

That means that any one point your disk is a maximum of 7 days since a check!

 

Rather than the once a month mine does it now and practically fixes the OP issue as it'll be running for a few hours at a time when nobody is using it...

Link to comment

Whilst your doing that could you look at partial checks?

 

For example, do 100% over 7 days, so 14.5% a day?

 

I wake my server at 7AM to run mover and then directly after do some more of a check.

 

That means that any one point your disk is a maximum of 7 days since a check!

 

Rather than the once a month mine does it now and practically fixes the OP issue as it'll be running for a few hours at a time when nobody is using it...

Nice idea.

Link to comment

Whilst your doing that could you look at partial checks?

 

For example, do 100% over 7 days, so 14.5% a day?

 

I wake my server at 7AM to run mover and then directly after do some more of a check.

 

That means that any one point your disk is a maximum of 7 days since a check!

 

Rather than the once a month mine does it now and practically fixes the OP issue as it'll be running for a few hours at a time when nobody is using it...

Nice idea.

 

Indeed it is:

 

The problem with this approach is that it is entirely possible that the throttle would have to be in place all the time.....

 

The first compromise should be making sure parity work as far as possible happens when unRAID is used the least. Notice I specifically dont say evening as again that will not be a universal assumption. unRAID should track historical usage stats anyway and this could be one use for them....

Link to comment

I like the idea of a contextually aware parity check system.

 

It tracks historical usage (far more accurately than I can) to figure out the best time to run the check but most importantly, it AUTO throttles based upon I/O. eg. I don't use my server between 11pm and 7am most days, at all - but my usenet downloads do.

 

unRAID should somehow know that if I start streaming a 50GB BD iso I'm going to be doing for upwards of 90 mins most likely. I have no idea how you would implement that, though.

 

As previously stated, contextual awareness is the key thing here.

 

Link to comment

Whilst your doing that could you look at partial checks?

 

For example, do 100% over 7 days, so 14.5% a day?

 

I wake my server at 7AM to run mover and then directly after do some more of a check.

 

That means that any one point your disk is a maximum of 7 days since a check!

 

Rather than the once a month mine does it now and practically fixes the OP issue as it'll be running for a few hours at a time when nobody is using it...

Nice idea.

 

Is that a 'I like it enough to implement it' or a 'I like it enough to think about implementing it'?  ;)

Link to comment

It would be useful if each add-on and the parity-check process had configurable priorities. For Example, Linux has a "nice" command that accepts values from -20 to 20. (There is also an "ionice" command that takes different values.) If we configure basic file service to run at level 0, then add-ons and parity-checks can be configured as higher or lower priority. The current behavior should be the default configuration.

Link to comment
  • 6 months later...

I really would like this feature, and it seems in high demand. I actually adjust my tunables so I have a little headroom because of this, which makes my parity checks ~2 hours longer. Watching BD ISOs during parity checks is impossible otherwise.

 

I would put this feature right up there with 64bit. Two features that we could use. The lowmemory issue caused by 32bit is also problematic, since many drives need much higher tunables to get the fastest parity checks possible. However, without this setting, many people would probably still not want to run them that high.

 

Hello tyrindor,

 

I was curious if I could ask what your settings for your tunables are so that you can use your server during parity checks?

 

There's only once a week I don't have need of my unRaid servers for more than ten hours ...

(unRaid Pro main, unRaid Pro backups, unRaid Plus testing, unRaid Plus in waiting)

 

If something goes wrong

I'm down for three days and if I do a parity check

I can't do anything else until the next week

(can you see too dependent on my unRaid servers?)

 

I've been wanting to re-balance disks between my unRaid servers ...

(for the reason that it's inaccessible during parity builds and parity checks mainly)

 

Thanks for your time,

Bobby

Link to comment