[Plugin] Mover Tuning


Recommended Posts

if i could configure it as i wrote in my previous post, the mover would run with proper timing which results in running 1-2x a day.

Now i need to run the mover on schedule every 1-3 hours and the cache disk still can be overloaded. How is that supposed to be better?

In fact I am trying to reduce amount of times the mover has to run...

Edited by pingu3000
Link to comment
1 hour ago, pingu3000 said:

if i could configure it as i wrote in my previous post, the mover would run with proper timing which results in running 1-2x a day.

Now i need to run the mover on schedule every 1-3 hours and the cache disk still can be overloaded. How is that supposed to be better?

In fact I am trying to reduce amount of times the mover has to run...

Why do you not have the Minimum Free Space setting for the Cache drive set to a suitable value so that if space gets low new files start by-passing the cache and going directly to the array?  This means that at least things keep working all the time.   Writes that by-pass the cache will tend to be slower, but is your work pattern one where this will get noticed in practice?    If you are regularly filling up your cache then maybe you should be investing in larger cache?

Link to comment
Quote

Why do you not have the Minimum Free Space setting for the Cache drive set to a suitable value so that if space gets low new files start by-passing the cache and going directly to the array?  This means that at least things keep working all the time.   Writes that by-pass the cache will tend to be slower, but is your work pattern one where this will get noticed in practice?    If you are regularly filling up your cache then maybe you should be investing in larger cache?

 

i want to maximize the usage of my cache drive and minimize the amount of times the array has to spin up. main advantages are the speed and longevity of my disk drives then there are others like noise and power consumption. usually the size of my cache drive is enough. though sometimes it happens that a large amount of data is moved to the server, in these cases i want the cache drive to be emptied once it is (almost) full. since that would mean the cache drive could be continued to be filled up while the mover is running i would have much better transfer speeds. also i do not want the cache drive to be used arbitrarily or with me having to check if it is full or not, i want it to be predictable and reliably being used all times. outside that i'd probably be fine with the mover running once a day. of course i could just buy a much larger cache drive with space i wouldn't need most of the time, but that would be a waste of resources since 80 or 90% of the time the current one is fine. 

i am a bit surprised that i have to defend my use case here while it seems like a pretty basic and logic approach of using the cache drive. to me it even seems that my idea is the go-to implementation and like an oversight of LT to not implement that feature.

i'd still like to know if it is doable to implement such a feature via script or else.

Edited by pingu3000
Link to comment
8 hours ago, pingu3000 said:

cache drive could be continued to be filled up while the mover is running i would have much better transfer speeds.

Cache would be full and couldn't continue to be filled up until mover moves things to the array since writes to the array are slower than writes to cache. When cache is full it can't go any faster than writing to the array. What would actually happen depends on your Minimum Free setting for cache (in Global Share Settings). If it is set larger than the largest file you will write then it will overflow to the array. If not then there is some chance that a large file will fail when cache runs out of space.

 

And of course, writing the overflow to the array will be impacted by the writes mover is trying to do at the same time, so everything is going to be even slower.

 

I really think you are expecting impossible things. Do you know about such things as seek times, for example?

 

Link to comment

well, i also have to admit that a move by free space is probably the better user case instead of a cron job by time,

im happy as i increased the crons to check and mover just will turn on when space is like this, even if still have to rely on my daily check.

 

i setted my 1 tb cache disk to 85 %, so lower 150 gb it will move and im not in the user case that i ll fill more then 150 gigs / day, but i understand

the question too and i also would prefer like described, this daily check could be lowered to an hourly job if i would think about going closer to the edge,

but as im still old school and usually would say 20 % should be always free ... ;) so i can live now perfectly with this helpful script.

 

May a note, let it run 10 minute wise, then its almost like u want it ... ;)

 

and about impossible, my docker size warnings coming also frequently when im on the edge, so i d say its not impossible, its currently not designed

to work with this argument (free space), so with this script its "tricked" todo so on the given cron´s.

Link to comment
On 10/25/2018 at 1:40 AM, trurl said:

Cache would be full and couldn't continue to be filled up until mover moves things to the array since writes to the array are slower than writes to cache. When cache is full it can't go any faster than writing to the array. What would actually happen depends on your Minimum Free setting for cache (in Global Share Settings). If it is set larger than the largest file you will write then it will overflow to the array. If not then there is some chance that a large file will fail when cache runs out of space. 

  

And of course, writing the overflow to the array will be impacted by the writes mover is trying to do at the same time, so everything is going to be even slower. 

  

I really think you are expecting impossible things. Do you know about such things as seek times, for example? 

  

I am still amazed how hard you try to defend the current primitive functioning of the cache drive while it could very obviously be way more refined.

 

If the mover starts moving at 10% capacity left of the cache drive, i could NOT fill the cache drive faster with network uploads (to the server) and internet downloads (on the server) than it is emptying by the data being written to the array. This means the cache drive will always be used to it's (90%) maximum and the array would only be spinned up when necessary. The cache drive will never be completely full and data written to the array will only be done by the mover.

 

I brought up the network speed argument before and i think since using turbowrite and a new intel network card i actually might be able to write faster to the array over lan than the 400 Mbit/s from before. That point might be mute after testing. If not, it would still be my biggest gripe.

 

I understand your point about the minimum free setting, it doesn't concern me much though. i transfer files of less than 10-20% capacity of my cache disk.

 

Before starting with quizzes we should understand each other, meaning you should understand my point. :P

Link to comment

I can't seem to get my mover to work... Am I doing something wrong with the settings?   I set it to 40% but after 200/500GB are utilized nothing happens on its own.  It will just keep going to the brink of capacity on the SSD and dockers will just stop running if they're on the cache.

Capture.PNG

 

EDIT: just saw on the previous page to set it to hourly and it will basically check to see if the threshold is reached *per hour*

 

Will try that out now

 

 

 

UPDATE: Next hour has started and still nothing, mover is not triggered at all.  My cache is nearing 70% full, and my threshold is 40%...

 

Mover only starts up again after I remove the plugin and keep the mover on hourly.

Edited by pro9c3
Link to comment

not sure if i missed something here.

But checking syslog there is no new data there at all saying that it didnt run for some reason.

But i've tried a few setting changes and each hour mover doesn't start.

 

Any pointers of something i might have missed or do i need to reboot after installing the plugin?

mover.thumb.PNG.0078f4d3c769d419ca137b7bda4f439b.PNG

disk.thumb.PNG.568152de8e08c2613aac1ad8d78580da.PNG

Link to comment
On ‎10‎/‎27‎/‎2018 at 6:07 PM, pingu3000 said:

If the mover starts moving at 10% capacity left of the cache drive, i could NOT fill the cache drive faster with network uploads (to the server) and internet downloads (on the server) than it is emptying by the data being written to the array. This means the cache drive will always be used to it's (90%) maximum and the array would only be spinned up when necessary. The cache drive will never be completely full and data written to the array will only be done by the mover.

I'm trying to understand your statement.

Your idea might work if you have an SSD, but with a spinner, your write throughput (to the cache) will dramatically slow down once you have reads in parallel!

Same and even worse, with your array. Given an SSD cache, you will probably fill up the cache as you can't copy to the array fast enough.

You will have to run the numbers to see if it works under any circumstance.

Writes to idle cache ~110MB/s (Gbit LAN limit)

Writes to idle array ~40-50MB/s

Writes to cache while simultaneusly copying to array ~???

Writes to array while simultaneusly reading/streaming data ~???

You can easily lock down your array if you copy mass data during unfavorable times of high user activity - with or without mover.

Therefore it's not wise to let the cache run into a high usage level and just rely on this logic.

Link to comment
On 10/28/2018 at 3:07 AM, pingu3000 said:

I am still amazed how hard you try to defend the current primitive functioning of the cache drive while it could very obviously be way more refined.

 

If the mover starts moving at 10% capacity left of the cache drive, i could NOT fill the cache drive faster with network uploads (to the server) and internet downloads (on the server) than it is emptying by the data being written to the array. This means the cache drive will always be used to it's (90%) maximum and the array would only be spinned up when necessary. The cache drive will never be completely full and data written to the array will only be done by the mover.

 

I brought up the network speed argument before and i think since using turbowrite and a new intel network card i actually might be able to write faster to the array over lan than the 400 Mbit/s from before. That point might be mute after testing. If not, it would still be my biggest gripe.

 

I understand your point about the minimum free setting, it doesn't concern me much though. i transfer files of less than 10-20% capacity of my cache disk.

 

Before starting with quizzes we should understand each other, meaning you should understand my point. :P

I get what you are saying here, you could fill the cache drive to 100% while mover is running.

I have experienced this and if you have unraid setup incorrectly dockers will crash which is probably what you are getting at here.

 

Under Shares make sure Cache Drive is set to Prefer on any of the Shares you have set to Only.

If the Cache drive fills too 100% any files that need to be changed will be written to the array, when mover runs next time it will have free space and move those files back to the Cache Drive. Its the same as wanting those files on your Cache Drive only but in the event of 100% full Cache the system has a place to keep operating.

You have to also Include some drives from your array within the Share where you want this overflow to take place.

The only thing that is going to happen here is any accesses to that file that is on the array instead of the cache drive will read at array speed.

Once Mover runs again that file will now be at Cache Drive at Cache speed.

 

Completely opposite in my opinion the Unraid setup between Cache and Array is amazing.

If you use the Prefer option on your Share you will run into a bad day, once the Cache Drive nears 100% you will get I/O errors and at somepoint docker crashes or system lockup.

 

If you are running into some weird problems like Disk Full when the Cache Drive is only at 70% or 80%, that is a BTRFS bug you can run a rebalance on the Cache Drive to clear the space.

Look at your Cache Drive Used Space then click on the Cache drive on the left under balance you should see "Data, single: total=83.01GiB, used=80.65GiB"

See if the total is around what your used space is in Unraid.

You can manually press Balance and it will clean it up for you.

 

I have a cron job that runs "btrfs balance start -dusage=75 /mnt/cache" every week just to clean it up.

 

Might be useful actually if this plugin checked the Cache Folder in DF and compared it with the BTRFS Filesystem Database if they don't match run rebalance.

Maybe another plugin should keep an eye on that, anyway i find the Cron job enough to do the job.

 

 

 

Link to comment
9 hours ago, Maticks said:

Under Shares make sure Cache Drive is set to Prefer on any of the Shares you have set to Only.

If the Cache drive fills too 100% any files that need to be changed will be written to the array

As I mentioned before this will depend on Minimum Free for cache in Global Share Settings. Unraid will only overflow if cache has less than Minimum Free. It decides when it begins to write the file which disk to use. If you fill up cache to 100% it is too late for it to decide.

 

On 10/27/2018 at 12:07 PM, pingu3000 said:

understand your point about the minimum free setting, it doesn't concern me much though. i transfer files of less than 10-20% capacity of my cache disk.

It isn't clear if he really understood my point or not since the reason he gives for not being concerned about it is not really relevant to the point I was making.

Link to comment
  • 3 weeks later...

Another option to add to the plugin which will make it even better.

We have the check every hour on the Cronjob if not at the threshold or over that percent don't run the Mover.

 

How about an exception, if all Data Disks are spun up already during the cronjob check you might as well run the Mover.

Since 90% of people running this plugin are to not move unless at X Threshold to stop disks from being spun up, if they are already all spinning why wait to the threshold and force all drives to spin up then.

 

Just an idea for offloading the data to the parity array when it makes sense too.

 

Edited by Maticks
Link to comment

Is it possible to add rules so that after certain time, the mover is forced to stop.

 

My situation is that, im adding 500GB of data to my unraid. When mover starts, it wants to move these 500GB of data which take a long time.  When mover is running, unraid is slow and certain application stops working(Steam).

I want the mover to run while I am sleeping (1.00am till 7.00am). At 7.00, mover scripts is killed, and files that are currently moving will continue moving. No new files will be moved.

 

Right now, when I want movie script to stop I use

pkill mover

and wait few hours.

 

Make sense?

Link to comment
3 hours ago, publicENEMY said:

Tried that. Got


mover: command not found

 

That output makes zero sense, as even with the plugin installed, a mover script is still available.

 

This plugin is a wrapper for the existing mover script.  No modifications get made to the existing script, so manual controls are a hair different.

 

Your pkill command kills the wrapper, but doesn't kill the original mover that's running.  Hence why you have to wait a couple of hours.

 

What you want to do is

mover.old stop
Link to comment

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.

Guest
Reply to this topic...

×   Pasted as rich text.   Restore formatting

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.