Jump to content

[Plugin] Mover Tuning

Squid

By Squid
in Plugin Support

Recommended Posts

Recommended

Posted by hugenbdd,

How to ignore a SINGLE file 1.) Find the path of the file you wish to ignore.     ls -ltr /mnt/cache/Download/complete/test.txt         root@Tower:/# ls -ltr /mnt/cache/Download/complete/test.txt         -rwxrwxrwx 1 root root 14 Oct 27 11:32 /mnt/cache/Download/complete/test.txt* 2.) Copy the complete path used.       /mnt/cache/Download/complete/test.txt 3.) Create a text file to hold the ignore list.     vi /mnt/user/appdata/mover-ignore/mover_ignore.txt 4.) Add a file pa
Recommended by Squid
  • Like

4 reactions

Go to this post
Recommended

Posted by Swarles,

FAQ - Frequently Asked Questions I'm putting these here for some of the common problems, and will try to update it as more questions/issues come along. How do I enable logs / provide logs? Mover wont run after installing Mover Tunning? Mover isn't ignoring files or folders? Issue Reporting Template  
Recommended by JonathanM
  • Upvote
  • Thanks

3 reactions

Go to this post
Reynald Explorer

Reynald

Posted
Posted (edited)

Hi all,

I made a new version of the plugin. This fork replaces @hugenbdd's plugin which is not maintained anymore and will land in community applications if your feedbacks are positive.

 

It worth the upgrade if you want:

- Automatic age

- Compatibility with 7.0.0 without playing with symlinks

- Minor fixes

Please see README / Changelog for more informations: R3yn4ld/ca.mover.tuning (github.com).

 

I use it hourly and it shall work correctly. I had not made extensive testing on 7.0.0 however (just a couple of days, seems OK). Please report any bug/problem.

 

Here is the plugin: ca.mover.tuning.plg

 

Please remove hugenbdd's one before installing (at the moment via Plugins/Install plugin/insert above URL).

Please note that you'll have to reconfigure mover tuning.

 

Advise: enable Test Mode and verify the Mover_Tuning_{date}T{time}list in the /tmp/Mover directory. This file contain the files that would be moved by original mover with Test Mode off.

If everything seems OK you can turn off Test mode.

 

@SimonF, would you please check my plugin script and tell me if it seems OK for you to publish on CA? I've worked on renaming and symlinking files and hope I did not break reverse compatibility (should be OK but I'd like a review please)

 

 

 

Edited by Reynald
Links and typo
  • Like 2
  • Thanks 1
Link to comment
  • Replies 1.8k
  • Created
  • Last Reply

Top Posters In This Topic

Popular Days

Top Posters In This Topic

Popular Days

Popular Posts

Squid
Squid

CA Mover Tuning       A simple little plugin (mainly created for my own purposes) that will let you fine tune the operation of the mover script On scheduled runs of mover, o

hugenbdd
hugenbdd

New Version 07-03-2023   HUGE thanks/Shout-out to @Swarles for the help with updating code fixes and adding a great new feature.  You can now have some customization per share.  It's great t

hugenbdd
hugenbdd

How to ignore a SINGLE file 1.) Find the path of the file you wish to ignore.     ls -ltr /mnt/cache/Download/complete/test.txt         root@Tower:/# ls -ltr /mnt/cache/Download/complete/test

Posted Images

Yock Rookie

Yock

Posted
Posted

Unraid 7 beta has a bug where the setting "Tunable (md_write_method)" does not apply until after a reboot of the server.

 

I THINK it's what stopping the mover plugin from working for me as i have that setting set to read/modify/write, in the plugin i want it set to force turbo write on (as in reconstructive write) and so the mover doesn't start for me even with this fix to the plugin.

Link to comment
Reynald Explorer

Reynald

Posted
Posted

Oh OK I see.

 

Is there another workaround than rebooting the server that you can me point to so I can see if something can be made by mover plugin please?

However, I cannot see how this bug can prevent mover plugin to work, as there is no error control on the Force Turbo Write piece of code

 

There is probably something else.

 

Are list created in /tmp/Mover ?

Is test mode enable ? (it shall be set to "no" if above lists are OK )

Link to comment
Yock Rookie

Yock

Posted
Posted

I don't think there is a workaround.

 

It's sadly not possible for me right now to get near to anything administrative stuff on my server atm as i'm not home and don't want to use a computer/network not owned by me for that.

But what i remember from logs is all it did was say something about starting the whole process, setting turbo writes on then next line was setting turbo writes off again and then nothing more.

Test mode was disabled.

 

Sorry i'm of practically no help but as said i'm not near the server atm. :/

Link to comment
erf89 Noob

erf89

Posted
Posted
On 6/30/2024 at 3:34 PM, wuudogg said:

Thinking I may need to rm /usr/local/sbin/move first ?

 

Edit confirmed.

 

Issued from terminal:

rm /usr/local/sbin/move

ln -s /usr/libexec/unraid/move /usr/local/sbin/move

 

Working great. Thank you!

Saved me from rolling back/pulling my hair out! Thanks! 😄

Link to comment
Reynald Explorer

Reynald

Posted
Posted (edited)
25 minutes ago, Yock said:

I don't think there is a workaround.

 

It's sadly not possible for me right now to get near to anything administrative stuff on my server atm as i'm not home and don't want to use a computer/network not owned by me for that.

But what i remember from logs is all it did was say something about starting the whole process, setting turbo writes on then next line was setting turbo writes off again and then nothing more.

Test mode was disabled.

 

Sorry i'm of practically no help but as said i'm not near the server atm. :/

 

 No problem Yock.

 

The behavior you describe is likely that you have filter preventing moving files. 

Between the two turbo writes line in the log you should see something like: 

ionice $ioLevel nice -n $niceLevel $age_mover_str

or 

ionice $ioLevel nice -n $niceLevel /usr/local/sbin/mover.old $options

 

Here is the behavior I have with 7.0.0-beta.1 and 2024-07-07 mover plugin:

image.thumb.png.b14d5d35a1253f1a033f3c57e5d03066.png

 

(edit: it seems that force turbo write is working as we have mdcmd returns)

Edited by Reynald
  • Like 1
Link to comment
Yock Rookie

Yock

Posted
Posted

The plugin was with default settings except for the force turbo writers.

 

Sounds good that it works, i will look into it again when i get back to the server. Thanks for the help!

  • Like 1
Link to comment
Yock Rookie

Yock

Posted
Posted

Couldn't help my self so borrowed a secure computer and tested.

 

Jul 8 12:27:49 YockS emhttpd: shcmd (87): /usr/local/sbin/mover start &> /dev/null &

Jul 8 12:27:49 YockS kernel: mdcmd (36): set md_write_method 1

Jul 8 12:27:49 YockS kernel:

Jul 8 12:27:49 YockS kernel: mdcmd (37): set md_write_method 0

Jul 8 12:27:49 YockS kernel:

 

Is what i get.

Link to comment
Yock Rookie

Yock

Posted
Posted

Just noticed on the picture my loggin is off.. Would probably help with that on... So...

 

Jul 8 12:55:56 YockS kernel: mdcmd (46): set md_write_method 1

Jul 8 12:55:56 YockS kernel:

Jul 8 12:55:56 YockS move: Usage: /usr/local/sbin/mover.old start [-e] <disk_name>

Jul 8 12:55:56 YockS move: /usr/local/sbin/mover.old stop|status

Jul 8 12:55:56 YockS move: <disk_name> must match pattern 'disk[0-9]*'

Jul 8 12:55:56 YockS kernel: mdcmd (47): set md_write_method 0

Jul 8 12:55:56 YockS kernel:

Link to comment
Reynald Explorer

Reynald

Posted
Posted

Please enable Mover logging.

 

What is the first parameter of Mover Tuning settings ? This one:

image.png.5181114302d5de9e6c91f749e41c0a81.png

 

With your settings, Mover Tuning filter are applied only on scheduled (cron) run, that is on the 28th of the month.

You need to set this to also invoke Mover Tuning on "Move Now" button press:

image.png.3fb891c9a0d0c72f8e57e2a0b2f0495a.png

Link to comment
Reynald Explorer

Reynald

Posted
Posted
1 minute ago, Yock said:

Just noticed on the picture my loggin is off.. Would probably help with that on... So...

 

Jul 8 12:55:56 YockS kernel: mdcmd (46): set md_write_method 1

Jul 8 12:55:56 YockS kernel:

Jul 8 12:55:56 YockS move: Usage: /usr/local/sbin/mover.old start [-e] <disk_name>

Jul 8 12:55:56 YockS move: /usr/local/sbin/mover.old stop|status

Jul 8 12:55:56 YockS move: <disk_name> must match pattern 'disk[0-9]*'

Jul 8 12:55:56 YockS kernel: mdcmd (47): set md_write_method 0

Jul 8 12:55:56 YockS kernel:

 

OK so there is a bug with the mover.old invocation for unraid 7.0.0+.
As a workaround, you can keep your default parameters and set this to Yes:

image.png.7ab0308e48e3ae3ccb08a3f3eb114c0f.png

or remove this plugin as you do not use filters

  • Thanks 1
Link to comment
Yock Rookie

Yock

Posted
Posted

Haha nearly killed my moms pets now when i yelled out in joy.

It is working now! But if i have schedule on i can't manually move and vice versa, no biggie though. As a switch between turbo write on and off is really what i wanted anyways.

 

I can't thank you enough for your help!

  • Thanks 1
Link to comment
Reynald Explorer

Reynald

Posted
Posted
1 minute ago, Yock said:

It is working now! But if i have schedule on i can't manually move and vice versa, no biggie though.

 

I'll investigate this one after looking at this error: 

move: Usage: /usr/local/sbin/mover.old start [-e] <disk_name>

 

Link to comment
SimonF Mentor

SimonF

Posted
Posted
10 hours ago, Reynald said:

 

  

 

Hi all,

I made a new version of the plugin. This fork replaces @hugenbdd's plugin which is not maintained anymore and will land in community applications if your feedbacks are positive.

 

It worth the upgrade if you want:

- Automatic age

- Compatibility with 7.0.0 without playing with symlinks

- Minor fixes

Please see README / Changelog for more informations: R3yn4ld/ca.mover.tuning (github.com).

 

I use it hourly and it shall work correctly. I had not made extensive testing on 7.0.0 however (just a couple of days, seems OK). Please report any bug/problem.

 

Here is the plugin: ca.mover.tuning.plg

 

Please remove hugenbdd's one before installing (at the moment via Plugins/Install plugin/insert above URL).

Please note that you'll have to reconfigure mover tuning.

 

Advise: enable Test Mode and verify the Mover_Tuning_{date}T{time}list in the /tmp/Mover directory. This file contain the files that would be moved by original mover with Test Mode off.

If everything seems OK you can turn off Test mode.

 

@SimonF, would you please check my plugin script and tell me if it seems OK for you to publish on CA? I've worked on renaming and symlinking files and hope I did not break reverse compatibility (should be OK but I'd like a review please)

 

 

 

Looks good to me. symlink is created correctly. You will need an xml and to speak to @Squidfor ca

  • Like 1
  • Thanks 1
Link to comment
Reynald Explorer

Reynald

Posted
Posted

I plan to add "Automatic age" to array->cache mover for shares with cache set to "Prefer" (= Array -> Cache).
I also plan to add some "Smart caching" option like I did in the following (deprecated) script: https://forums.unraid.net/topic/92126-smart-caching-script/

 

 

Both of these function will require a rewrite of plugin internal filelist functions, but I think it worth it.

 

My plan is to, under the hood:

  1. build a single list of files that were created/modified since the last plugin run (first run will be quite long)
  2. add (in a future version, tunable with option) seeding torrent, played series, and other "preventing to sleep" things
  3. sort the list by age (as actually done by "Automatic Age" settings)
    Then with the list containing all needed information to do smart things:
  4. emptying old files from cache to array
  5. smartly fill the cache 

How do you all feel about this? Any advice/feature request?

 

  • Like 3
Link to comment

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.

Guest
Reply to this topic...

×   Pasted as rich text.   Restore formatting

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.

×
Go to topic listing
×
×
  • Create New...