Jump to content
Squid

[Plugin] Mover Tuning

120 posts in this topic Last Reply

Recommended Posts

15 hours ago, Squid said:

Added in two more tunables:

 

Priority of the mover process itself (Normal, Low, Very Low) and

I/O priority of the process (Normal, Low, and Idle)

 

Setting these *may* help with the odd user where the mover process interferes with the normal operation of the GUI / other apps.  Note that if your system pauses all I/O streams if/when a disk is required to spin up that these tunables will not help out all (In which case, do what I did -> adjust Kodi's (eg) caching settings)

Awsome update @Squid. I'm already using the plugin, but don' really have the CPU issue people are mentioning, but seen a lot of people whining about it, so very nice addition to an already awesome plugin :) Tnx for all the hard work :)

 

Share this post


Link to post
Posted (edited)
Quote

Added in two more tunables:

 

Priority of the mover process itself (Normal, Low, Very Low) and

I/O priority of the process (Normal, Low, and Idle)

 

Setting these *may* help with the odd user where the mover process interferes with the normal operation of the GUI / other apps.  Note that if your system pauses all I/O streams if/when a disk is required to spin up that these tunables will not help out all (In which case, do what I did -> adjust Kodi's (eg) caching settings)

Thanks @Squid!  Mover used to slow down all running docker apps until it was done.  I can now run it without it negatively impacting my system after lowering the cpu and I/O priority.  This update is great!

Edited by Iciclebar

Share this post


Link to post

Just be aware that setting it to very low and idle may significantly impact the amount of time it takes to move the files depending upon what else is going on in the system when mover runs

Sent from my NSA monitored device

Share this post


Link to post

Thank you for adding that in Squid, i just tracked down my problems to Mover yesterday.

Sent over a Beer your way.

Share this post


Link to post
2 hours ago, Maticks said:

Sent over a Beer your way.

Thank you.  Always appreciated :) 

Share this post


Link to post

I've just discovered this plugin. It really seems to scratch an itch with the mover, so thanks a bunch. I do have a feature request, though:

 

I have a large cache drive (4tb SSD), and it would be helpful if I could use set it up so, during a scheduled move, turbo write is used only if the cache drive is filled to a certain percent. That way small scheduled moves would happen without turbo write, but big ones that need to write faster to clear the cache would spin up all the drives.

 

Also, and maybe it's asking for too much, but it'd be great if we could:

 

A) stop the mover manually; and

B) automatically stop scheduled moves if they haven't finished by a certain time.

 

My need for those last two may be abrogated by being able to set the mover priority lower / cancel it during parity scans, but I haven' had a chance to test the low priority much.

 

Anyway, thanks again!

Share this post


Link to post
Posted (edited)

Been having a lot of new errors in my system logs for the last 3 days or so. After studying them a little bit, it seems the error only happens when the mover is running. And lines up with the time that I updated this plugin and started using the new priority settings.

 

Here's a piece of my logs: https://pastebin.com/xVpiJDfY

 

I've tried it with and without turbowrite on, and same errors. 

 

Anyone else see any new errors that occur while mover is running?

unraid-diagnostics-20190630-0016.zip

Edited by drawmonster

Share this post


Link to post
1 minute ago, drawmonster said:

Here's a piece of my logs: 

Always best practice to attach the full diagnostics.

Share this post


Link to post

All the errors are from python which is 100% nothing to do with mover or the plugin, unless mover is attempting to move something that your python script doesnt like being moved

Sent via telekinesis

Share this post


Link to post

Great plugin! I've got a feature request:

 

- Ability to set a mover stop time (fixed time or relative to starting time).

 

I've got a rather large cache drive (8TB) and I got my mover scheduled for the night ours so it doesn't impact user performance. But when a lot of the cache drive has been filled by daily operations the mover runs into daytime and impacts the day performance. I would love the feature to tell it to stop the mover (I invoke the command manually via the web terminal > 'mover stop') on a certain time or after x hours of running.

Share this post


Link to post
5 hours ago, Mevius said:

Great plugin! I've got a feature request:

 

- Ability to set a mover stop time (fixed time or relative to starting time).

 

I've got a rather large cache drive (8TB) and I got my mover scheduled for the night ours so it doesn't impact user performance. But when a lot of the cache drive has been filled by daily operations the mover runs into daytime and impacts the day performance. I would love the feature to tell it to stop the mover (I invoke the command manually via the web terminal > 'mover stop') on a certain time or after x hours of running.

Just use user scripts and have a script that runs at whatever time with the mover stop command.  Doesn't do anything if its not running.

 

Share this post


Link to post

Thanks for the plugin, squid.  I do have a "feature" request, if you could call it that


Currently, I modify mover (now called mover.old) to ignore certain directories in certain user shares and print log messages about it.

 

I had noticed after I installed your plugin that mover.old's logs no longer print.  Digging into it, it's due to it using "echo" and it's caller used to previously pipe mover's output to logger (ie. /usr/local/sbin/mover(.old) |& logger).  I can easily add that line into your mover.php (and I do), though I assume an update would get rid of it.

 

Would this be possible, so mover.old's existing logs would also print?

Share this post


Link to post

I've added this to my own mover script, but a good feature for some would be to have certain docker containers shut down before mover runs and restarted after. For example, Radarr and Sonarr both like to get currupt databases if you run mover and they weren't set to use the cache drive as their appdata directory specifically. I don't have any appdata on my cache drive, as it's only used for speeding up my downloads and file transfers. 

Share this post


Link to post
Thanks for the plugin, squid.  I do have a "feature" request, if you could call it that

Currently, I modify mover (now called mover.old) to ignore certain directories in certain user shares and print log messages about it.
 
I had noticed after I installed your plugin that mover.old's logs no longer print.  Digging into it, it's due to it using "echo" and it's caller used to previously pipe mover's output to logger (ie. /usr/local/sbin/mover(.old) |& logger).  I can easily add that line into your mover.php (and I do), though I assume an update would get rid of it.
 
Would this be possible, so mover.old's existing logs would also print?
Honestly I never enable mocer logging, so never checked it at all. Will pump out an update in the next day or so

Ashamed to admit that I no longer live up to my teenage ethos of "it's better to burn out than fade away"

Share this post


Link to post
On 8/20/2019 at 11:40 AM, gooner said:

Would this be possible, so mover.old's existing logs would also print?

Should be fixed on today's update

Share this post


Link to post

is it possible to move files immediately after they have finished writing? (like a physical HDD cache?)

 

So it can write to cache (fast) then trickle feed it to array. keeping the cache empty and only using it as, well, a a cache for new files?

Share this post


Link to post

Closest you can get is hourly itll run (or even say every 10 minutes if you set a force move custom cron)

Sent from my NSA monitored device

Share this post


Link to post
14 hours ago, Squid said:

Closest you can get is hourly itll run (or even say every 10 minutes if you set a force move custom cron)

Sent from my NSA monitored device
 

Ah thanks, I'll keep it set to overnight, as the mover always messes up my Kodi playback. (I can't change the Kodi cache, but that's a different Problem)

 

Is it possible to limit the speed,  only write to the array at 4mbs for example.

Edited by TheBuz

Share this post


Link to post

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.

Guest
Reply to this topic...

×   Pasted as rich text.   Restore formatting

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.