WeeboTech Posted November 5, 2013 Share Posted November 5, 2013 just provide min and max throttling settings to allow it to be tuned This is how the linux md drivers have done it in the past with success. When needed, you could adjust minimum and maximum levels. by echoing a value into /proc/ before unraid, I used this functionality to adjust values as I needed them on demand. It worked fine for linux with the standard md driver, it would work fine here for unRAID. I think we might be over analyzing this. With these two values, the end user has what he needs to provide the service level required. I'm sure examples of this could be found in various linux md.c/raid?.c driver code. examples: [pre]cat /proc/sys/dev/raid/speed_limit_max 200000 cat /proc/sys/dev/raid/speed_limit_min 1000 md: minimum _guaranteed_ reconstruction speed: 1000 KB/sec/disc. md: using maximum available idle IO bandwidth (but not more than 200000 KB/sec) for reconstruction. echo 50000 >/proc/sys/dev/raid/speed_limit_min watch cat /proc/mdstat [/pre] Link to comment
lionelhutz Posted November 5, 2013 Share Posted November 5, 2013 So it's already there? Just add webgui parameters to adjust it? Link to comment
WeeboTech Posted November 5, 2013 Share Posted November 5, 2013 So it's already there? Just add webgui parameters to adjust it? No it's not in the unRAID drivers/kernel. The examples are of the standard linux md driver kernel interface. I'm not sure what version this example represents. I posted it because, A. It's been thought of before by the kernel raid developers. B. If it's worked fine for software raid all these years, it could work fine for unRAID. C. by using these two values with a webgui interface, the end user could adjust as he sees fit. If some value parity check/sync/generate speed over usability adjust accordingly. If some value usability vs a check operation, adjust accordingly. Link to comment
lionelhutz Posted November 5, 2013 Share Posted November 5, 2013 OK, it still might not be too difficult to incorporate that throttling code instead of deleting it. Link to comment
jbartlett Posted November 5, 2013 Share Posted November 5, 2013 One thing to keep in mind is that this throttle will improve the WAF. -John * Wife Acceptance Factor Link to comment
NAS Posted November 5, 2013 Share Posted November 5, 2013 madness, I say, madness If this gets implemented just make sure it can be turned off. Link to comment
limetech Posted November 5, 2013 Author Share Posted November 5, 2013 So it's already there? Just add webgui parameters to adjust it? No. What I would implement is a 'min' setting. Set to 0 means "let all non-parity sync/check I/O have complete priority", and set to 100 would cause it to run the way it does now (generate parity sync/check I/O as fast as possible). This is something I've wanted to put in for a while now in order to prevent parity check from interrupting media playback. Link to comment
garycase Posted November 5, 2013 Share Posted November 5, 2013 So it's already there? Just add webgui parameters to adjust it? No. What I would implement is a 'min' setting. Set to 0 means "let all non-parity sync/check I/O have complete priority", and set to 100 would cause it to run the way it does now (generate parity sync/check I/O as fast as possible). This is something I've wanted to put in for a while now in order to prevent parity check from interrupting media playback. So if nothing else is going on, would parity check /sync/rebuilds work at max speed? ... or will they be limited to the percentage set by this parameter? Link to comment
limetech Posted November 6, 2013 Author Share Posted November 6, 2013 So it's already there? Just add webgui parameters to adjust it? No. What I would implement is a 'min' setting. Set to 0 means "let all non-parity sync/check I/O have complete priority", and set to 100 would cause it to run the way it does now (generate parity sync/check I/O as fast as possible). This is something I've wanted to put in for a while now in order to prevent parity check from interrupting media playback. So if nothing else is going on, would parity check /sync/rebuilds work at max speed? ... or will they be limited to the percentage set by this parameter? It would go at max speed. What I want to do is introduce delays in parity sync/check stripe generation based on volume of normal I/O to array disks in conjunction with this new setting. Link to comment
Finka Posted November 6, 2013 Share Posted November 6, 2013 I am afraid that the implementation of this new idea could create problems. If history has thought us anything, is that any new features around here end up bundled up into the md-mod driver making a mess of it. If this is to be done right, we have to separate the ability to throttle from the I/O monitoring: (1) First, implement a simple throttling ability in the md-mod driver, and expose the setting through /proc/mdcmd -- say, an integer number from 0 to 100 signifying the percentage of full speed desired for the check/build. (zero can have a special meaning). The throttling has to be done right, making sure that it not only slows things down, but effectively releases resources to other processes. (whatever sleep in kernel space means). (2) Only after you've done and tested (1) then you think of writing some daemon (in user space) that will monitor I/O and adjust the throttling setting accordingly. The user has the option to not even start that daemon, and their system will run check/build at 100% speed, like before. Or, the user can just manually (from the go script) set the trottle setting to 80 or something and again not bother with starting any realtime I/O monitoring. To summarize my personal preference: - I would like to have a configurable throttle; - I don't need the real time I/O monitoring and adjusting. As I will want to be able to turn off the latter, I don't want these two bundled together. Link to comment
tyrindor Posted November 7, 2013 Share Posted November 7, 2013 I really would like this feature, and it seems in high demand. I actually adjust my tunables so I have a little headroom because of this, which makes my parity checks ~2 hours longer. Watching BD ISOs during parity checks is impossible otherwise. I would put this feature right up there with 64bit. Two features that we could use. The lowmemory issue caused by 32bit is also problematic, since many drives need much higher tunables to get the fastest parity checks possible. However, without this setting, many people would probably still not want to run them that high. Link to comment
Interstellar Posted November 12, 2013 Share Posted November 12, 2013 Whilst your doing that could you look at partial checks? For example, do 100% over 7 days, so 14.5% a day? I wake my server at 7AM to run mover and then directly after do some more of a check. That means that any one point your disk is a maximum of 7 days since a check! Rather than the once a month mine does it now and practically fixes the OP issue as it'll be running for a few hours at a time when nobody is using it... Link to comment
limetech Posted November 12, 2013 Author Share Posted November 12, 2013 Whilst your doing that could you look at partial checks? For example, do 100% over 7 days, so 14.5% a day? I wake my server at 7AM to run mover and then directly after do some more of a check. That means that any one point your disk is a maximum of 7 days since a check! Rather than the once a month mine does it now and practically fixes the OP issue as it'll be running for a few hours at a time when nobody is using it... Nice idea. Link to comment
mr-hexen Posted November 13, 2013 Share Posted November 13, 2013 All this moot anyways since at present unraid does not "schedule" a parity check. I'd like to see that added first. Link to comment
NAS Posted November 13, 2013 Share Posted November 13, 2013 Whilst your doing that could you look at partial checks? For example, do 100% over 7 days, so 14.5% a day? I wake my server at 7AM to run mover and then directly after do some more of a check. That means that any one point your disk is a maximum of 7 days since a check! Rather than the once a month mine does it now and practically fixes the OP issue as it'll be running for a few hours at a time when nobody is using it... Nice idea. Indeed it is: The problem with this approach is that it is entirely possible that the throttle would have to be in place all the time..... The first compromise should be making sure parity work as far as possible happens when unRAID is used the least. Notice I specifically dont say evening as again that will not be a universal assumption. unRAID should track historical usage stats anyway and this could be one use for them.... Link to comment
ironicbadger Posted November 19, 2013 Share Posted November 19, 2013 I like the idea of a contextually aware parity check system. It tracks historical usage (far more accurately than I can) to figure out the best time to run the check but most importantly, it AUTO throttles based upon I/O. eg. I don't use my server between 11pm and 7am most days, at all - but my usenet downloads do. unRAID should somehow know that if I start streaming a 50GB BD iso I'm going to be doing for upwards of 90 mins most likely. I have no idea how you would implement that, though. As previously stated, contextual awareness is the key thing here. Link to comment
Interstellar Posted November 22, 2013 Share Posted November 22, 2013 Whilst your doing that could you look at partial checks? For example, do 100% over 7 days, so 14.5% a day? I wake my server at 7AM to run mover and then directly after do some more of a check. That means that any one point your disk is a maximum of 7 days since a check! Rather than the once a month mine does it now and practically fixes the OP issue as it'll be running for a few hours at a time when nobody is using it... Nice idea. Is that a 'I like it enough to implement it' or a 'I like it enough to think about implementing it'? Link to comment
dgaschk Posted November 23, 2013 Share Posted November 23, 2013 It would be useful if each add-on and the parity-check process had configurable priorities. For Example, Linux has a "nice" command that accepts values from -20 to 20. (There is also an "ionice" command that takes different values.) If we configure basic file service to run at level 0, then add-ons and parity-checks can be configured as higher or lower priority. The current behavior should be the default configuration. Link to comment
perfessor101 Posted June 23, 2014 Share Posted June 23, 2014 I really would like this feature, and it seems in high demand. I actually adjust my tunables so I have a little headroom because of this, which makes my parity checks ~2 hours longer. Watching BD ISOs during parity checks is impossible otherwise. I would put this feature right up there with 64bit. Two features that we could use. The lowmemory issue caused by 32bit is also problematic, since many drives need much higher tunables to get the fastest parity checks possible. However, without this setting, many people would probably still not want to run them that high. Hello tyrindor, I was curious if I could ask what your settings for your tunables are so that you can use your server during parity checks? There's only once a week I don't have need of my unRaid servers for more than ten hours ... (unRaid Pro main, unRaid Pro backups, unRaid Plus testing, unRaid Plus in waiting) If something goes wrong I'm down for three days and if I do a parity check I can't do anything else until the next week (can you see too dependent on my unRaid servers?) I've been wanting to re-balance disks between my unRaid servers ... (for the reason that it's inaccessible during parity builds and parity checks mainly) Thanks for your time, Bobby Link to comment
Recommended Posts