[Plugin] Mover Tuning

[Reynald]

7 hours ago, RonneBlaze said:

The current version 2024.08.06.0813 is working great so far, it has moved all the files from shares that are set to Cache > Array but it is leaving the empty folders on the cache drive.. Is that normal or did i miss a setting?

There is no cleaning function yet, I'm going to add it (if you wanna wait a little before cleaning, so you can report if it works)

 

I will also relax starting verification for those like @ramjam824 who don't have an array but raid pool instead.

 

Coming soon

[Reynald]

2024-08-07

• Allow operation without array if multiple pools (R3yn4ld) Unraid 7.0.0.beta2 may be required for this to work (6.x gui might not allow to have pool as Primary and Secondary) (cc @ramjam824 please test and report :))
• Add cleanup empty folder function (R3yn4ld): Will delete empty folder if file have been moved (cc @RonneBlaze)

[zoggy]

Went to try out Reynald version 2024.08.07.0054  now that I upgraded to unraid 7b2, however it errors:

 

Fatal error: cache not enabled and less than 2 pools set up.

 

Not sure why its saying that, I do have a cache pool (1 drive) and an array.

 

[Faceman]

Just gotta say, Love the rapid development going on here, massive improvements in functionality and usability.

 

 

 

 

[RonneBlaze]

43 minutes ago, zoggy said:

Went to try out Reynald version 2024.08.07.0054  now that I upgraded to unraid 7b2, however it errors:

 

Fatal error: cache not enabled and less than 2 pools set up.

 

Not sure why its saying that, I do have a cache pool (1 drive) and an array.

 

Getting the same thing with Unraid 6.12.11 using plugin version 2024.08.07.0054 

[alturismo]

4 hours ago, Reynald said:

2024-08-07

can also confirm

 

Aug  7 05:00:01 AlsServerII root: *********************************************** Mover Tuning Plugin version version=2024.08.07.0054 *********************************************
Aug  7 05:00:01 AlsServerII root: Fatal error: cache not enabled and less than 2 pools set up.

 

also, cache and array are there

 

 

may also as note, errors lets the mover "stall"

 

 

not a big deal as we always can kill it

 

root@AlsServerII:~# /usr/local/emhttp/plugins/ca.mover.tuning/age_mover stop
/usr/local/emhttp/plugins/ca.mover.tuning/age_mover: line 1126: kill: (3406321) - No such process
mover: stopped
root@AlsServerII:~#

 

cant find any missmatch Options due the changes ...

[Reynald]

7 hours ago, Reynald said:

2024-08-07

• Allow operation without array if multiple pools (R3yn4ld) Unraid 7.0.0.beta2 may be required for this to work (6.x gui might not allow to have pool as Primary and Secondary) (cc @ramjam824 please test and report :))

 

Sorry guys I was sleeping and couldn't correct the bug earlier.

 

Oups, the error is due to the new possibility to run without array.

 

Pushing a patch in a few minutes

[Reynald]

2024.08.07.0858 

• Fix bug introduced by "Allow operation without array if multiple pools" preventing mover to run if less than 2 pools installed.
• Allow operation without array if multiple pools [R3yn4ld]. Unraid 7.0.0.beta2 may be required for this to work (6.x gui might not allow to have pool as Primary and Secondary)
• Add cleanup empty folder function[R3yn4ld]

[alturismo]

1 hour ago, Reynald said:

• Fix bug introduced by "Allow operation without array if multiple pools" preventing mover to run if less than 2 pools installed.
•  

thanks, updated and back to normal again

[methanoid]

16 hours ago, Reynald said:

I'm not yet used with file filtering, it is still the same code for this part as the outdated version you use.

Anyway, to get the latest plugin, follow the link in my signature: where it says: "I take over CA Mover Tuning. "

 

And suddenly everything "just works"... thank you.  Didnt test the "ignore files" cos it worked and said "under 50% full so not doing any moving". Which is good!

[ramjam824]

@Reynald Still not working for me.

 

 

[zoggy]

updating to the latest version, it resolves the fatal error but does not move anything...

 

to comment you are showing:

Info: you may optionally install 'bc' from 'NerdTools' to have more precise size value

IMHO: this should not be mentioned in logs, as nerdtools is not really recommended for unraid 7.x nor alive anymore. if you think bc should be included, just included with the plugin...  or offer up an option to enable more precise sizes and blip on there about telling the user to go install bc.

 

Then, your plugin has "Test Mode (dry run):" defaulting to yes.. and as this is buried middle way down in the options it makes it easy to miss (!it got me). I'd recommend moving this to the very top (or bottom like old plugin) or something for better visibility.

Edited August 7 by zoggy

[Reynald]

4 hours ago, ramjam824 said:

@Reynald Still not working for me.

 

 

Hummm, OK, I will investigate.

Can you please: post the result of this command in terminal: 

ls /boot/config/pools/*.cfg | wc -l

 

[Reynald]

3 hours ago, zoggy said:

updating to the latest version, it resolves the fatal error but does not move anything...

 

to comment you are showing:

Info: you may optionally install 'bc' from 'NerdTools' to have more precise size value

IMHO: this should not be mentioned in logs, as nerdtools is not really recommended for unraid 7.x nor alive anymore. if you think bc should be included, just included with the plugin...  or offer up an option to enable more precise sizes and blip on there about telling the user to go install bc.

Thanks for your comment. Is it still not moving anything or was it test mode?

Good point for bc. Anyway, as unraid 7 is based on slackware 15, as recent 6.x versions, I don't see a problem (I myself choose to continue using nerdpack as long as repo is alive), but yes, I will definitly think about this point

 

Quote

Then, your plugin has "Test Mode (dry run):" defaulting to yes.. and as this is buried middle way down in the options it makes it easy to miss (!it got me). I'd recommend moving this to the very top (or bottom like old plugin) or something for better visibility.

I will move the option on top in second position, that makes sense  .
I will also remove this default when the plugin will be stable enough, I've completly rewritten some part and I consider that it's safer to check in test mode, before going live, at the moment.

Edited August 7 by Reynald

[Faceman]

I'm seeing some minor errors in the logs related to files with a $ in the filename, which is usually a valid symbol, but perhaps something isn't being escaped or parsed properly somewhere.

for example :

 

stat: cannot statx '/mnt/disk7/Media/Music/Metallica/1987 - The .98 E.P.-Garage Days Re-Revisited/Art/Front.jpg

 

Theres meant to be a $5 in there and it's getting messed up, lots of other examples too.

[alturismo]

19 minutes ago, Faceman said:

related to files with a $ in the filename

you may should also consider to rename them as special characters pretty often will end up with issues ... not only mover related, may also then with other things (apps) coming across ... and <$> is a so called special character

[Faceman]

I know they need to be carefully handled in bash scripts, but they've been fine for about 10 years so something has changed recently that doesn't like them.

 

I'm renaming them as I go anyway.

[Reynald]

1 hour ago, Faceman said:

I know they need to be carefully handled in bash scripts, but they've been fine for about 10 years so something has changed recently that doesn't like them.

 

I'm renaming them as I go anyway.

I'll investigate, I did my best on escaping, but here, it's a command which call a program which call a program.

 

It shall be fine if you use ctime Yes however

[ramjam824]

15 hours ago, Reynald said:

Hummm, OK, I will investigate.

Can you please: post the result of this command in terminal: 

ls /boot/config/pools/*.cfg | wc -l

 

 

[ZGief]

Hey all,

Recently I noticed my mover job getting stuck. Looking at the logs i see the following:

 

Aug  8 08:46:33 Tower move: /usr/local/emhttp/plugins/ca.mover.tuning/age_mover: line 467: * 0 / 100: syntax error: operand expected (error token is "* 0 / 100")

 

I don't think I have any crazy settings.

 

Looking at that line of code it seems like maybe $POOLSIZE isn't being set maybe?

 

Any advice / guidance would be appreciated!

Edited August 8 by ZGief
accidentally submit

[Reynald]

3 hours ago, ramjam824 said:

 

Ok, that's what I expected. I will modify checking function to accommodate that. 

[Reynald]

33 minutes ago, ZGief said:

Hey all,

Recently I noticed my mover job getting stuck. Looking at the logs i see the following:

 

Aug  8 08:46:33 Tower move: /usr/local/emhttp/plugins/ca.mover.tuning/age_mover: line 467: * 0 / 100: syntax error: operand expected (error token is "* 0 / 100")

 

I don't think I have any crazy settings.

 

Looking at that line of code it seems like maybe $POOLSIZE isn't being set maybe?

 

Any advice / guidance would be appreciated!

*0/100 is strange, I would expect 70/100 and 20/100.

 

Do you have a share setting override?

[ZGief]

19 minutes ago, Reynald said:

Do you have a share setting override?

 

I'm assuming you're talking about this setting on the shares "Override Mover Tuning settings for this share:"

 

 

I have 3 shares. Two are caches and don't have this setting. The other is set to 'no' as shown above.

[Reynald]

OK @ZGiefthank you.

It's then something about your options (I admit I've never tested with age=no and no other filters).

I will do some tests reproducing your configuration.

Also, your test mode is set to yes, that does not explain the errors but explain why files are not moved

 

[ZGief]

10 minutes ago, Reynald said:

It's then something about your options (I admit I've never tested with age=no and no other filters).

I will do some tests reproducing your configuration.

Also, your test mode is set to yes, that does not explain the errors but explain why files are not moved

 

Really appreciate your help btw!

 

I have been trying different setting and still getting that error. From your advice above I tried changing age to 'yes', forgot i had test on yes, changed that. Added all the logs below in case that is helpful. 

 

Happy to re-run with any settings for experimentation. Just let me know. 

 

Also, I have to run: `mover stop` to get it to stop hanging after that error.

 

 

Aug  8 10:50:33 Tower emhttpd: shcmd (5659): /usr/local/sbin/mover |& logger -t move &
Aug  8 10:50:33 Tower root: Starting Mover
Aug  8 10:50:33 Tower root: ionice -c 2 -n 0 nice -n 0 /usr/local/emhttp/plugins/ca.mover.tuning/age_mover start
Aug  8 10:50:33 Tower move: *************************************************** Mover Tuning Plugin version 2024.08.07.0858 *************************************************
Aug  8 10:50:33 Tower move: Log Level: 1
Aug  8 10:50:33 Tower move: ----------------------------------------------------------------- Global settings ---------------------------------------------------------------
Aug  8 10:50:33 Tower move: Using global moving threshold: 70 %
Aug  8 10:50:33 Tower move: Using global freeing threshold: 20 %
Aug  8 10:50:33 Tower move: Age: 10
Aug  8 10:50:33 Tower move: After script: /mnt/ssd_cache_docker/appdata/qbittorrent/qBittorrent/scripts/resume-qbit.sh
Aug  8 10:50:33 Tower move: Before script: /mnt/ssd_cache_docker/appdata/qbittorrent/qBittorrent/scripts/pause-qbit.sh
Aug  8 10:50:33 Tower move: Launching before script: /mnt/ssd_cache_docker/appdata/qbittorrent/qBittorrent/scripts/pause-qbit.sh
Aug  8 10:50:33 Tower move: Qbit pause starting to pause cache torrents and run mover
Aug  8 10:50:33 Tower move: Qbit pause completed pausing torrents
Aug  8 10:50:33 Tower move: Before script finished
Aug  8 10:50:33 Tower move: ***************************************************************** FILTERING FILES ***************************************************************
Aug  8 10:50:33 Tower move: Info: you may optionally install 'bc' from 'NerdTools' to have more precise size value
Aug  8 10:50:33 Tower move: ------------------------------------------------------------ Processing appdata share -----------------------------------------------------------
Aug  8 10:50:33 Tower move: Primary storage: ssd_cache_docker - used:   2 % (4 GiB)
Aug  8 10:50:33 Tower move: Secondary storage: none
Aug  8 10:50:33 Tower move: Share Information: Name: appdata - Path: /mnt/ssd_cache_docker/appdata
Aug  8 10:50:33 Tower move: Mover action: no action, only ssd_cache_docker used (cache:only).
Aug  8 10:50:33 Tower move: => Nothing will be moved. Share usage is taken into account in the calculation of the threshold for other shares.
Aug  8 10:50:33 Tower move: Calculating share usage... (can take a moment)
Aug  8 10:50:33 Tower move: ssd_cache_docker/appdata used: 447 MiB
Aug  8 10:50:33 Tower move: -------------------------------------------------------------- Processing data share ------------------------------------------------------------
Aug  8 10:50:33 Tower move: Primary storage: ssd_cache_array - used:  42 % (97 GiB)
Aug  8 10:50:33 Tower move: Secondary storage: user0
Aug  8 10:50:33 Tower move: Share Information: Name: data - Path: /mnt/ssd_cache_array/data
Aug  8 10:50:33 Tower move: Moving threshold: 70% (163 GiB) ; Freeing threshold: 20% (46 GiB)
Aug  8 10:50:33 Tower move: Mover action: ssd_cache_array->user0 (cache:yes). Pool is below moving threshold percentage:  42% < 70%.
Aug  8 10:50:33 Tower move: => Skipping
Aug  8 10:50:33 Tower move: ------------------------------------------------------------ Processing domains share -----------------------------------------------------------
Aug  8 10:50:33 Tower move: df: /mnt/cache: No such file or directory
Aug  8 10:50:33 Tower move: df: /mnt/cache: No such file or directory
Aug  8 10:50:33 Tower move: df: /mnt/cache: No such file or directory
Aug  8 10:50:33 Tower move: /usr/local/emhttp/plugins/ca.mover.tuning/age_mover: line 467: * 20 / 100: syntax error: operand expected (error token is "* 20 / 100")