[Plugin] Mover Tuning

Reynald · August 4

1 hour ago, PPH said:

Hi @Reynald,

Could you advise where I set the value for the "global freeing threshold" please?

Thanks.

Sure, it's there in settings (wording may change a little):

image.png.cb7e0ec05b463a8e1f314f2feb97609c.png

Edited August 4 by Reynald

PPH · August 4

48 minutes ago, Reynald said:

Sure, it's there in settings (wording may change a little):

Hi @Reynald,

Thanks.

It's a little strange as I had already set this value to 75%:

Settings.png.a618c0679a097bebd46bd787d7b9fd72.png

And when changed to 80%, the same error persists:

move: /usr/local/emhttp/plugins/ca.mover.tuning/age_mover: line 421: * 80 / 100: syntax error: operand expected (error token is "* 80 / 100")

I also tried changing to 85% and 85% matching your settings but the same error occurs:

move: /usr/local/emhttp/plugins/ca.mover.tuning/age_mover: line 421: * 85 / 100: syntax error: operand expected (error token is "* 85 / 100")

Thanks.

PPH · August 4

30 minutes ago, PPH said:

Hi @Reynald,

Thanks.

It's a little strange as I had already set this value to 75%:

And when changed to 80%, the same error persists:

move: /usr/local/emhttp/plugins/ca.mover.tuning/age_mover: line 421: * 80 / 100: syntax error: operand expected (error token is "* 80 / 100")

I also tried changing to 85% and 85% matching your settings but the same error occurs:

move: /usr/local/emhttp/plugins/ca.mover.tuning/age_mover: line 421: * 85 / 100: syntax error: operand expected (error token is "* 85 / 100")

Thanks.

Hi @Reynald,

I wonder if it has anything to do with the fact that I have used dashes "-" in my Pool Names (e.g. my current (SSD-based) cache pool is called "os-pool" (without the double-quotes))?

Is this possibly being filtered or not extracted due to this naming standard?

Thanks.

Reynald · August 4

Thanks for your tests.

The setting is taken into account. In the different logs you show me here, pool usage is missing, but in the diagnostic zip it's the threshold.

I'm uploading a new version that has better verbosity so the log will help us.

I'm in the final testing stage and update shall land within a couple of hours

PPH · August 4

1 minute ago, Reynald said:

Thanks for your tests.

The setting is taken into account. In the different logs you show me here, pool usage is missing, but in the diagnostic zip it's the threshold.

I'm uploading a new version that has better verbosity so the log will help us.

I'm in the final testing stage and update shall land within a couple of hours

Hi @Reynald,

Thank you.

I also did some investigation and saw in an old "mover_tuning" file that it was reporting that I had a Share .CFG file present that was no longer valid and that the "Mover Tuning" plugin was "skipping" this share. I deleted the Share .CFG file from the /config/shares folder and the "Mover Tuning" plugin is now working and no longer erroring (although I still have the plugin in 'test' mode at the moment).

I'm not sure if this was a change in the latest version of the plugin (that I installed today) as this "missing" Share has been present for quite some time and previously did not cause the plugin to error.

I have sent you a DM with the log files for the CTIME/MTIME issue I am encountering (no rush on these though).

Again, thanks for your help and a great unRAID plugin!

Reynald · August 4

Yes, a user had an error a couple of page ago regarding a deleted share, so I added a check and some verbosity.

It is in release notes

image.png.ab6c92a066473521dd77421ad18aae90.png

However, as deleted shares are skipped, it shall not affect moving operations but can effectively throw some errors.

I may have a deeper look, please @everyone do not hesitate to copy/paste some logs snippets in this thread when you see errors, so I can debug or/and improve error handling.

Reynald · August 4

Oh, and already replied DMs

alturismo · August 4

2 hours ago, Reynald said:

Yes, a user had an error a couple of page ago regarding a deleted share, so I added a check and some verbosity.

was also me but i took this as Ocassion to cleanup my "deleted" Shares ...

but yes, errors where gone after the update, but as you updated so quick i immediately cleaned up here too, had a bad

conscience ...

thanks for your efforts in this

Reynald · August 5

Oups, almost forgot to announce:

2024.08.04.2112

Unraid 7.0.0 beta2 Secondary storage Compatibility (R3yn4ld) minor enhancements (6.12 mover action naming) and... can now move between pools (tested on 7.0.0-beta2) !
Fix find not ignoring hidden files (R3yn4ld) (Thanks to helpful-tune3401)
2024-08-04.0054:
Fix default Settings handling causing a "Unary operator" bug (R3yn4ld) (thanks to Alturismo)
Add freeing threshold option (R3yn4ld)

This one is a big one, bringing pool to pool moving. I ran it all the day. Please test in test mode, check /tmp/ca.mover.tuning/Mover_action_xxx.list

Jaycedk · August 5

After installing update 2024.08.04.2112 i'm getting the following errors.

Unraid 6.12.11

Aug  5 10:13:31 JaycedkUnraid emhttpd: shcmd (1206952): /usr/local/sbin/update_cron
Aug  5 10:13:55 JaycedkUnraid emhttpd: shcmd (1207025): /usr/local/sbin/mover |& logger -t move &
Aug  5 10:13:55 JaycedkUnraid root: Starting Mover
Aug  5 10:13:55 JaycedkUnraid root: ionice -c 2 -n 0 nice -n 0 /usr/local/emhttp/plugins/ca.mover.tuning/age_mover start
Aug  5 10:13:55 JaycedkUnraid move: ******************************************************************* MOVER START *****************************************************************
Aug  5 10:13:55 JaycedkUnraid move: Log Level: 1
Aug  5 10:13:55 JaycedkUnraid move: ----------------------------------------------------------------- Global settings ---------------------------------------------------------------
Aug  5 10:13:55 JaycedkUnraid move: Mover Tuning Plugin version 2024.08.04.2112
Aug  5 10:13:55 JaycedkUnraid move: Using global moving threshold: 70 %
Aug  5 10:13:55 JaycedkUnraid move: Using global freeing threshold:  %
Aug  5 10:13:55 JaycedkUnraid move: Age: Automatic (smart caching)
Aug  5 10:13:55 JaycedkUnraid move: ***************************************************************** FILTERING FILES ***************************************************************
Aug  5 10:13:55 JaycedkUnraid move: ---------------------------------------------------------- Processing Arr-Scripts share ---------------------------------------------------------
Aug  5 10:13:55 JaycedkUnraid move: /usr/local/emhttp/plugins/ca.mover.tuning/age_mover: line 424: 1000203837440 *  / 100: syntax error: operand expected (error token is "/ 100")
Aug  5 10:13:57 JaycedkUnraid flash_backup: adding task: /usr/local/emhttp/plugins/dynamix.my.servers/scripts/UpdateFlashBackup update

Reynald · August 5

Hi,

I thought I fixed this bug, but the solution seems not working for everyone

Can you please share /boot/config/plugins/ca.mover.tuning/ca.mover.tuning.cfg

Jaycedk · August 5

Sure

testmode="no"
logging="yes"
movenow="yes"
version="2024.08.04.2112"
moverDisabled="no"
movingThreshold="70"
freeingThreshold=""
age="yes"
daysold="-1"
advancedFilters="no"
ctime="no"
sizef="no"
sparsnessf="no"
filelistf="no"
filetypesf="no"
ignoreHidden="no"
parity="no"
moverNice="0"
moverIO="-c 2 -n 0"
enableTurbo="no"
rebalanceShares="no"
synchronizeCache="no"
advancedOptions="no"
force="no"
beforeScript=""
afterScript=""
omovercfg="no"
ca.mover.tuning.cfg

Reynald · August 5

OK thanks, I will add something in plugin installation script.

In the mean time, to solve the bug, just change and apply this setting:
image.png.05ac8f57ecd418c478fa4e1f190db7ae.png

(edit: I realize help text contain a mistake, I will change to "Set to the amount of disk space used on the Primary (cache) drive after the mover ran.")

Edited August 5 by Reynald

Jaycedk · August 5

Will do and report back 👍

Jaycedk · August 5

After resetting the setting, and applying,

This line was set to

freeingThreshold="70"

and no longer freeingThreshold=""

And mover is again running a expected.

Thank you for the quick responce 👍

Freender · August 5

After updating to 2024.08.04.2112 mover crashed at night (the logs were disabled, not sure what happened),
my scheduler page was not showing any mover properties as well.

In the config

/boot/config/plugins/ca.mover.tuning/ca.mover.tuning.cfg I see 2 new empty parameters:

movingThreshold=
freeingThreshold=

changing movingThreshold to 85% resolved an issue

movingThreshold="85"
freeingThreshold=

Reynald · August 5

Great thank you (well, not the crash, but the empty parameters). And thanks for report.

I'm not sure about that the way I'm handling these new parameters in the plugin installation/updating script then:

ca.mover.tuning/plugins/ca.mover.tuning.plg at 2024-08-04-release2 · R3yn4ld/ca.mover.tuning (github.com)

Maybee $old_tresh value is not correctly interpreted during update.

I've added error control in age_mover script and will rework installation script.

Edited August 5 by Reynald

Reynald · August 5

New version:

2024.08.05.2004

Rebalance shares: Enhance previous "Repair Primary" option. Renamed it "Rebalance shares". This will move files from shares to their primary and secondary storage if spread elsewhere. May imply moving older files from Primary->Secondary or Secondary->Primary if allowed (cache:prefer or cache:yes) to free some space.

Note: "Run-once" is a writting error in the setting, it is meant to be "yes"
Bug fixes
Normaly freeingThreshold="" bug is gone

Edited August 5 by Reynald

Reynald · August 5

2024-08-06

Added Resynchronize all Primary files to Secondary option (R3yn4ld): Resynchronize all Primary files to Secondary. This will resynchronize the Primary (cached) files on both Primary and Secondary (array) so they are backed up and parity protected. All files will be synchronized again independently of modification time. This can be a long operation. Run-once setting will reset back to No after next run
Minor bug fixes and improvements (R3yn4ld)

alturismo · August 6

6 hours ago, Reynald said:

Minor bug fixes and improvements (R3yn4ld)

possible there is one back ? [: no: unary operator expected ...

Aug  6 05:00:02 AlsServerII root: ------------------------------------------------------------- Processing Media share ------------------------------------------------------------
Aug  6 05:00:02 AlsServerII root: Primary Pool: cache - used:  55 % (1014.74 GiB)
Aug  6 05:00:02 AlsServerII root: Secondary Pool: user0
Aug  6 05:00:02 AlsServerII root: Share Information: Name: Media - Path: /mnt/cache/Media
Aug  6 05:00:02 AlsServerII root: Moving threshold: 90% (1.63 TiB) ; Freeing threshold: 70% (1.27 TiB)
Aug  6 05:00:02 AlsServerII root: Mover action: cache->user0 (cache:yes). Pool is below moving threshold percentage:  55% < 90%.
Aug  6 05:00:02 AlsServerII root: /usr/local/emhttp/plugins/ca.mover.tuning/age_mover: line 521: [: no: unary operator expected
Aug  6 05:00:02 AlsServerII root: => Skipping
Aug  6 05:00:02 AlsServerII root: ----------------------------------------------------------- Processing Nextcloud share ----------------------------------------------------------

root@AlsServerII:~# cat /boot/config/plugins/ca.mover.tuning/ca.mover.tuning.cfg
moverDisabled="no"
moverNice="0"
moverIO="-c 2 -n 0"
age="yes"
daysold="-1"
ctime="no"
sizef="no"
sparsnessf="no"
filelistf="yes"
filelistv="/mnt/user/system/mover/mover_ignore.txt"
filetypesf="no"
parity="yes"
enableTurbo="no"
logging="yes"
force="no"
ignoreHidden="yes"
beforeScript=""
afterScript=""
omovercfg="no"
movenow="yes"
testmode="yes"
version=2024.08.06.0255
advancedFilters="no"
repairPrimary="no"
synchronizeCache="no"
advancedOptions="no"
movingThreshold="90"
freeingThreshold="70"
rebalanceShares="no"
resynchronizeCache="no"
root@AlsServerII:~#

Reynald · August 6

48 minutes ago, alturismo said:

possible there is one back ? [: no: unary operator expected ...

Of course it is

However it's not the same bug because now (lack of) configuration shall be handled correctly.

Found the cause, it's about the new RESYNCHRONIZECACHE option. I forgot a space in the code (see on Github). Bug effect: resynchronize Primary->Secondary option won't work for share that does not met moving threshold.

Error will be corrected in next version.

Reynald · August 6

2024.08.06.0813

Bug fixes (R3yn4ld):
- Resynchronize not working for share below moving threshold.
- Internal mover moving files from Secondary to Primary instead of syncing (you may Resynchronize to correct the effect)
2024.08.06.0255
Added Resynchronize all Primary files to Secondary option (R3yn4ld): Resynchronize all Primary files to Secondary. This will resynchronize the Primary (cached) files on both Primary and Secondary (array) so they are backed up and parity protected. All files will be synchronized again independently of modification time. This can be a long operation. Run-once setting will reset back to No after next run
Minor bug fixes and improvements (R3yn4ld)

RonneBlaze · August 6

The current version 2024.08.06.0813 is working great so far, it has moved all the files from shares that are set to Cache > Array but it is leaving the empty folders on the cache drive.. Is that normal or did i miss a setting?

methanoid · August 6

On 8/2/2024 at 5:39 PM, Reynald said:
It might also be a stupid bug (I won't tell, because you are using an old version and due to respect I have to other devs of this plugins, I won't tell the previous dev is stupid, he's not, probably, certainly,... )

Can you please post the log ?

We will focus on a line stating:
skipfilelist string: find "/mnt/ ....

CA Mover Tuning version 2023.12.19 - my CA Autoupdate thinks that IS the latest version? Or did you mean unRAID version which also reports as latest Stable?

I enabled Mover logging but not sure where to find the "mover.log" specifically... but found this under Logging

Aug 6 19:14:27 Uberserver emhttpd: shcmd (3581410): /usr/local/sbin/mover |& logger -t move &
Aug 6 19:14:27 Uberserver move: mover: started
Aug 6 19:14:27 Uberserver move: file: /mnt/cache/data/backups/2024-08-04/uberserver-flash-backup-20240804-2305.zip
Aug 6 19:14:29 Uberserver move: file: /mnt/cache/data/backups/2024-08-04/plex_database_backup.tar.gz
Aug 6 19:14:31 Uberserver move: file: /mnt/cache/data/backups/2024-08-05/uberserver-flash-backup-20240805-2305.zip
Aug 6 19:14:41 Uberserver move: file: /mnt/cache/data/backups/2024-08-05/plex_database_backup.tar.gz
Aug 6 19:14:43 Uberserver move: file: /mnt/cache/data/backups/NZB_Archive.zip
Aug 6 19:14:44 Uberserver emhttpd: spinning down /dev/sdk
Aug 6 19:14:46 Uberserver emhttpd: spinning down /dev/sdd

So it is completely ignoring my "ignore.files" content. I also added a CR at end of the last line just in case.

I *thought* I had found the issue. Mover Tuning settings are fine but I have to make sure the Share Mover Tuning Settings are NOT set to Override... OK I am an idiot, I accept, but maybe might help if instructions said when enabling Tuning that you have to make sure "shares" are set appropriately? Hope thats constructive criticism?

But it didnt change anything to have the Share called Data "Override Mover Tuning settings for this share:" set to No....

I'm stuck

Reynald · August 6

I'm not yet used with file filtering, it is still the same code for this part as the outdated version you use.

Anyway, to get the latest plugin, follow the link in my signature: where it says: "I take over CA Mover Tuning. "

[Plugin] Mover Tuning

Recommended Posts

Link to comment

Top Posters In This Topic

Popular Days

Top Posters In This Topic

Popular Days

Popular Posts

Squid

hugenbdd

hugenbdd

Posted Images

Link to comment

Link to comment

Link to comment

Link to comment

Link to comment

Link to comment

Link to comment

Link to comment

Link to comment

Link to comment

Link to comment

Link to comment

Link to comment

Link to comment

Link to comment

Link to comment

Link to comment

Link to comment

Link to comment

Link to comment

Link to comment

Link to comment

Link to comment

Link to comment

Join the conversation