[Plugin] Mover Tuning

foux · June 16, 2023

8 minutes ago, hugenbdd said:

No, can you verify the folder was created?

/tmp/Mover

from the console

ls -ltr /tmp/Mover

Yes, the folder was created, and here is its content (after another run of course) :

14:58:04 with root in /mnt/ssd_pool/medias at Vili took 35m 46.3s …
➜ ls -ltr /tmp/Mover
total 38960
-rw-r--r-- 1 root root  4907973 Jun 16 10:25 Mover_Tuning_2023-06-16T102527.list
-rw-r--r-- 1 root root  1272063 Jun 16 10:26 Custom_Mover_Tuning_2023-06-16T102527.list
-rw-r--r-- 1 root root  3777980 Jun 16 10:26 Custom_Mover_Directory_2023-06-16T102527.list
-rw-r--r-- 1 root root        0 Jun 16 10:26 Cache_Tuning_2023-06-16T102527.list
-rw-r--r-- 1 root root        0 Jun 16 10:26 Custom_Cache_Tuning_2023-06-16T102527.list
-rw-r--r-- 1 root root 14657783 Jun 16 10:28 Mover_Tuning_2023-06-16T102527.log
-rw-r--r-- 1 root root  2758622 Jun 16 10:35 Mover_Tuning_2023-06-16T103533.list
-rw-r--r-- 1 root root  1272063 Jun 16 10:35 Custom_Mover_Tuning_2023-06-16T103533.list
-rw-r--r-- 1 root root  1628629 Jun 16 10:35 Custom_Mover_Directory_2023-06-16T103533.list
-rw-r--r-- 1 root root        0 Jun 16 10:36 Cache_Tuning_2023-06-16T103533.list
-rw-r--r-- 1 root root        0 Jun 16 10:36 Custom_Cache_Tuning_2023-06-16T103533.list
-rw-r--r-- 1 root root  8365138 Jun 16 10:37 Mover_Tuning_2023-06-16T103533.log
-rw-r--r-- 1 root root   245258 Jun 16 14:22 Mover_Tuning_2023-06-16T142244.list
-rw-r--r-- 1 root root    18350 Jun 16 14:22 Custom_Mover_Tuning_2023-06-16T142244.list
-rw-r--r-- 1 root root   228809 Jun 16 14:22 Custom_Mover_Directory_2023-06-16T142244.list
-rw-r--r-- 1 root root        0 Jun 16 14:22 Cache_Tuning_2023-06-16T142244.list
-rw-r--r-- 1 root root        0 Jun 16 14:22 Custom_Cache_Tuning_2023-06-16T142244.list
-rw-r--r-- 1 root root   738814 Jun 16 14:22 Mover_Tuning_2023-06-16T142244.log

hugenbdd · June 16, 2023

2 minutes ago, foux said:

Yes, the folder was created, and here is its content (after another run of course) :

14:58:04 with root in /mnt/ssd_pool/medias at Vili took 35m 46.3s …
➜ ls -ltr /tmp/Mover
total 38960
-rw-r--r-- 1 root root  4907973 Jun 16 10:25 Mover_Tuning_2023-06-16T102527.list
-rw-r--r-- 1 root root  1272063 Jun 16 10:26 Custom_Mover_Tuning_2023-06-16T102527.list
-rw-r--r-- 1 root root  3777980 Jun 16 10:26 Custom_Mover_Directory_2023-06-16T102527.list
-rw-r--r-- 1 root root        0 Jun 16 10:26 Cache_Tuning_2023-06-16T102527.list
-rw-r--r-- 1 root root        0 Jun 16 10:26 Custom_Cache_Tuning_2023-06-16T102527.list
-rw-r--r-- 1 root root 14657783 Jun 16 10:28 Mover_Tuning_2023-06-16T102527.log
-rw-r--r-- 1 root root  2758622 Jun 16 10:35 Mover_Tuning_2023-06-16T103533.list
-rw-r--r-- 1 root root  1272063 Jun 16 10:35 Custom_Mover_Tuning_2023-06-16T103533.list
-rw-r--r-- 1 root root  1628629 Jun 16 10:35 Custom_Mover_Directory_2023-06-16T103533.list
-rw-r--r-- 1 root root        0 Jun 16 10:36 Cache_Tuning_2023-06-16T103533.list
-rw-r--r-- 1 root root        0 Jun 16 10:36 Custom_Cache_Tuning_2023-06-16T103533.list
-rw-r--r-- 1 root root  8365138 Jun 16 10:37 Mover_Tuning_2023-06-16T103533.log
-rw-r--r-- 1 root root   245258 Jun 16 14:22 Mover_Tuning_2023-06-16T142244.list
-rw-r--r-- 1 root root    18350 Jun 16 14:22 Custom_Mover_Tuning_2023-06-16T142244.list
-rw-r--r-- 1 root root   228809 Jun 16 14:22 Custom_Mover_Directory_2023-06-16T142244.list
-rw-r--r-- 1 root root        0 Jun 16 14:22 Cache_Tuning_2023-06-16T142244.list
-rw-r--r-- 1 root root        0 Jun 16 14:22 Custom_Cache_Tuning_2023-06-16T142244.list
-rw-r--r-- 1 root root   738814 Jun 16 14:22 Mover_Tuning_2023-06-16T142244.log

Was this second run working? I believe there is a bug on the first run of Mover after installing. Second and following times it should work.

foux · June 16, 2023

Here are the logs and content of the folder, after a fresh run after deleting /tmp/Mover (before running of course) :

14:59:36 with root in /mnt/ssd_pool/medias at Vili …
➜ tail -f /var/log/syslog | grep "move:"
Jun 16 14:59:57 Vili move: Log Level: 1
Jun 16 14:59:57 Vili move: /usr/local/emhttp/plugins/ca.mover.tuning/age_mover: line 73: /tmp/Mover/Mover_Tuning_2023-06-16T145957.log: No such file or directory
Jun 16 14:59:57 Vili move: /usr/local/emhttp/plugins/ca.mover.tuning/age_mover: line 73: /tmp/Mover/Mover_Tuning_2023-06-16T145957.log: No such file or directory
Jun 16 14:59:57 Vili move: /usr/local/emhttp/plugins/ca.mover.tuning/age_mover: line 73: /tmp/Mover/Mover_Tuning_2023-06-16T145957.log: No such file or directory
Jun 16 14:59:57 Vili move: /usr/local/emhttp/plugins/ca.mover.tuning/age_mover: line 73: /tmp/Mover/Mover_Tuning_2023-06-16T145957.log: No such file or directory
Jun 16 14:59:57 Vili move: /usr/local/emhttp/plugins/ca.mover.tuning/age_mover: line 73: /tmp/Mover/Mover_Tuning_2023-06-16T145957.log: No such file or directory
Jun 16 14:59:57 Vili move: /usr/local/emhttp/plugins/ca.mover.tuning/age_mover: line 73: /tmp/Mover/Mover_Tuning_2023-06-16T145957.log: No such file or directory
Jun 16 14:59:57 Vili move: /usr/local/emhttp/plugins/ca.mover.tuning/age_mover: line 73: /tmp/Mover/Mover_Tuning_2023-06-16T145957.log: No such file or directory
Jun 16 14:59:57 Vili move: /usr/local/emhttp/plugins/ca.mover.tuning/age_mover: line 73: /tmp/Mover/Mover_Tuning_2023-06-16T145957.log: No such file or directory
Jun 16 14:59:57 Vili move: /usr/local/emhttp/plugins/ca.mover.tuning/age_mover: line 73: /tmp/Mover/Mover_Tuning_2023-06-16T145957.log: No such file or directory
Jun 16 14:59:57 Vili move: /usr/local/emhttp/plugins/ca.mover.tuning/age_mover: line 73: /tmp/Mover/Mover_Tuning_2023-06-16T145957.log: No such file or directory
Jun 16 14:59:57 Vili move: /usr/local/emhttp/plugins/ca.mover.tuning/age_mover: line 73: /tmp/Mover/Mover_Tuning_2023-06-16T145957.log: No such file or directory
Jun 16 14:59:57 Vili move: /usr/local/emhttp/plugins/ca.mover.tuning/age_mover: line 73: /tmp/Mover/Mover_Tuning_2023-06-16T145957.log: No such file or directory
Jun 16 14:59:57 Vili move: mover: started
Jun 16 14:59:57 Vili move: find: '/tmp/Mover/*': No such file or directory
Jun 16 14:59:57 Vili move: find: '/tmp/Mover/*': No such file or directory
Jun 16 15:00:03 Vili move: cat: /tmp/Mover/Hard_Mover_Tuning_2023-06-16T145957.list: No such file or directory
Jun 16 15:00:06 Vili move: mover: finished
Jun 16 15:00:06 Vili move: removed '/mnt/user/tools/mover-ignore.txt'

And the content of the forlder :

15:02:22 with root in /mnt/ssd_pool/medias at Vili took 2m 41.5s …
➜ ls -ltr /tmp/Mover
total 1204
-rw-r--r-- 1 root root 244944 Jun 16 15:00 Mover_Tuning_2023-06-16T145957.list
-rw-r--r-- 1 root root  18350 Jun 16 15:00 Custom_Mover_Tuning_2023-06-16T145957.list
-rw-r--r-- 1 root root 228495 Jun 16 15:00 Custom_Mover_Directory_2023-06-16T145957.list
-rw-r--r-- 1 root root      0 Jun 16 15:00 Cache_Tuning_2023-06-16T145957.list
-rw-r--r-- 1 root root      0 Jun 16 15:00 Custom_Cache_Tuning_2023-06-16T145957.list
-rw-r--r-- 1 root root 736823 Jun 16 15:00 Mover_Tuning_2023-06-16T145957.log

Edited June 16, 2023 by foux

foux · June 16, 2023

3 minutes ago, hugenbdd said:

Was this second run working? I believe there is a bug on the first run of Mover after installing. Second and following times it should work.

No, no runs worked (done a few ones)

The subsequent runs have way less errors, but still doesn't move anything :

15:04:01 with root in /mnt/ssd_pool/medias at Vili …
➜ tail -f /var/log/syslog | grep "move:"
Jun 16 15:04:15 Vili move: Log Level: 1
Jun 16 15:04:15 Vili move: mover: started
Jun 16 15:04:20 Vili move: cat: /tmp/Mover/Hard_Mover_Tuning_2023-06-16T150415.list: No such file or directory
Jun 16 15:04:23 Vili move: mover: finished
Jun 16 15:04:23 Vili move: removed '/mnt/user/tools/mover-ignore.txt'

And the hard mover tuning file indeed doesn't exists in the folder :

15:06:40 with root in /mnt/ssd_pool/medias at Vili took 2m 35.6s …
➜ ls -ltr /tmp/Mover
total 2416
-rw-r--r-- 1 root root 244944 Jun 16 15:00 Mover_Tuning_2023-06-16T145957.list
-rw-r--r-- 1 root root  18350 Jun 16 15:00 Custom_Mover_Tuning_2023-06-16T145957.list
-rw-r--r-- 1 root root 228495 Jun 16 15:00 Custom_Mover_Directory_2023-06-16T145957.list
-rw-r--r-- 1 root root      0 Jun 16 15:00 Cache_Tuning_2023-06-16T145957.list
-rw-r--r-- 1 root root      0 Jun 16 15:00 Custom_Cache_Tuning_2023-06-16T145957.list
-rw-r--r-- 1 root root 736823 Jun 16 15:00 Mover_Tuning_2023-06-16T145957.log
-rw-r--r-- 1 root root 246008 Jun 16 15:04 Mover_Tuning_2023-06-16T150415.list
-rw-r--r-- 1 root root  18712 Jun 16 15:04 Custom_Mover_Tuning_2023-06-16T150415.list
-rw-r--r-- 1 root root 229235 Jun 16 15:04 Custom_Mover_Directory_2023-06-16T150415.list
-rw-r--r-- 1 root root      0 Jun 16 15:04 Cache_Tuning_2023-06-16T150415.list
-rw-r--r-- 1 root root      0 Jun 16 15:04 Custom_Cache_Tuning_2023-06-16T150415.list
-rw-r--r-- 1 root root 740502 Jun 16 15:04 Mover_Tuning_2023-06-16T150415.log

Edited June 16, 2023 by foux

hugenbdd · June 16, 2023

8 minutes ago, foux said:

No, no runs worked (done a few ones)

The subsequent runs have way less errors, but still doesn't move anything :

15:04:01 with root in /mnt/ssd_pool/medias at Vili …
➜ tail -f /var/log/syslog | grep "move:"
Jun 16 15:04:15 Vili move: Log Level: 1
Jun 16 15:04:15 Vili move: mover: started
Jun 16 15:04:20 Vili move: cat: /tmp/Mover/Hard_Mover_Tuning_2023-06-16T150415.list: No such file or directory
Jun 16 15:04:23 Vili move: mover: finished
Jun 16 15:04:23 Vili move: removed '/mnt/user/tools/mover-ignore.txt'

And the hard mover tuning file indeed doesn't exists in the folder :

15:06:40 with root in /mnt/ssd_pool/medias at Vili took 2m 35.6s …
➜ ls -ltr /tmp/Mover
total 2416
-rw-r--r-- 1 root root 244944 Jun 16 15:00 Mover_Tuning_2023-06-16T145957.list
-rw-r--r-- 1 root root  18350 Jun 16 15:00 Custom_Mover_Tuning_2023-06-16T145957.list
-rw-r--r-- 1 root root 228495 Jun 16 15:00 Custom_Mover_Directory_2023-06-16T145957.list
-rw-r--r-- 1 root root      0 Jun 16 15:00 Cache_Tuning_2023-06-16T145957.list
-rw-r--r-- 1 root root      0 Jun 16 15:00 Custom_Cache_Tuning_2023-06-16T145957.list
-rw-r--r-- 1 root root 736823 Jun 16 15:00 Mover_Tuning_2023-06-16T145957.log
-rw-r--r-- 1 root root 246008 Jun 16 15:04 Mover_Tuning_2023-06-16T150415.list
-rw-r--r-- 1 root root  18712 Jun 16 15:04 Custom_Mover_Tuning_2023-06-16T150415.list
-rw-r--r-- 1 root root 229235 Jun 16 15:04 Custom_Mover_Directory_2023-06-16T150415.list
-rw-r--r-- 1 root root      0 Jun 16 15:04 Cache_Tuning_2023-06-16T150415.list
-rw-r--r-- 1 root root      0 Jun 16 15:04 Custom_Cache_Tuning_2023-06-16T150415.list
-rw-r--r-- 1 root root 740502 Jun 16 15:04 Mover_Tuning_2023-06-16T150415.log

DM sent. Lets work this out there. Once fixed I can post the solution in the forum.

hugenbdd · June 16, 2023

New release, sorry for the releases back to back.

###2023.06.16
- Fix no empty directories file
- Fix missing echo statement on hardlinks area. (Foux 6-16-2023)

foux · June 16, 2023

Thanks a lot @hugenbdd!

dopeytree · June 16, 2023

Have been thinking about other mover options. Perhaps the easiest would be to have different schedules for mover per share?

That way you could just set the data download share to move every month or 90 days. and other other data to move every day or once a week.

zellleonhart · June 18, 2023

I am new to unraid, just finished my first parity build and trying out the mover tuning plugin. On stable unraid 6.12, with test mode ON, it doesn't seem to log the files matched? If test mode is OFF, it does move successfully and log the filenames that are moved.

The log:

My mover tuning settings:

Mover logging: Enabled
Log when not moving due to rules: Yes
Move Now button follows plug-in filters: Yes
Test Mode: Yes

Edited June 18, 2023 by zellleonhart

hugenbdd · June 18, 2023

1 minute ago, zellleonhart said:
I am new to unraid, just finished my first parity build and trying out the mover tuning plugin. On stable unraid 6.12, with test mode ON, it doesn't seem to log the files matched? If test mode is OFF, it does move the files to array successfully.

The log:

My mover tuning settings:
Mover logging: Enabled
Log when not moving due to rules: Yes
Move Now button follows plug-in filters: Yes
Test Mode: Yes

More details/additional logging info is now under /tmp/Mover

zellleonhart · June 18, 2023

6 minutes ago, hugenbdd said:

More details/additional logging info is now under /tmp/Mover

Thank you!! I found all the logs in /tmp/Mover. It is indeed working fine.

DontWorryScro · June 18, 2023

I am completely up to date on plugins, have the very latest and greatest and yet im still getting tons of:

move: error: move, 392: No such file or directory (2): lstat: /mnt/cache_nvme/*

any idea why this is still happening?

Swarles · June 18, 2023

I can also confirm that on a fresh install the mover does not run correctly I think due to the changes to using /tmp/Mover for log files. It appears the directory is not made to begin with. I tested the following before running the Mover for the first time on a fresh plugin install and then after running the mover got the second result. I'm not sure why this is the case because it should be made immediately on the first age_mover / mvlogger() run....hmmmmm.

Spoiler

root@unraid:~# /tmp/Mover
bash: /tmp/Mover: No such file or directory
root@unraid:~# /tmp/Mover
bash: /tmp/Mover: Is a directory

My first run also resulted in these systemlogs:

Spoiler

Jun 19 01:49:06 unraid emhttpd: shcmd (92): /usr/local/sbin/mover |& logger -t move &
Jun 19 01:49:06 unraid root: Starting Mover
Jun 19 01:49:06 unraid root: ionice -c 2 -n 0 nice -n 0 /usr/local/emhttp/plugins/ca.mover.tuning/age_mover start 0 0 0 '' mp3, jpg, jpeg, .txt '' '' no 5 'yes' ''
Jun 19 01:49:06 unraid move: Log Level: 1
Jun 19 01:49:06 unraid move: /usr/local/emhttp/plugins/ca.mover.tuning/age_mover: line 73: /tmp/Mover/Mover_Tuning_2023-06-19T014906.log: No such file or directory
Jun 19 01:49:06 unraid move: /usr/local/emhttp/plugins/ca.mover.tuning/age_mover: line 73: /tmp/Mover/Mover_Tuning_2023-06-19T014906.log: No such file or directory
Jun 19 01:49:06 unraid move: /usr/local/emhttp/plugins/ca.mover.tuning/age_mover: line 73: /tmp/Mover/Mover_Tuning_2023-06-19T014906.log: No such file or directory
Jun 19 01:49:06 unraid move: /usr/local/emhttp/plugins/ca.mover.tuning/age_mover: line 73: /tmp/Mover/Mover_Tuning_2023-06-19T014906.log: No such file or directory
Jun 19 01:49:06 unraid move: /usr/local/emhttp/plugins/ca.mover.tuning/age_mover: line 73: /tmp/Mover/Mover_Tuning_2023-06-19T014906.log: No such file or directory
Jun 19 01:49:06 unraid move: /usr/local/emhttp/plugins/ca.mover.tuning/age_mover: line 73: /tmp/Mover/Mover_Tuning_2023-06-19T014906.log: No such file or directory
Jun 19 01:49:06 unraid move: /usr/local/emhttp/plugins/ca.mover.tuning/age_mover: line 73: /tmp/Mover/Mover_Tuning_2023-06-19T014906.log: No such file or directory
Jun 19 01:49:06 unraid move: /usr/local/emhttp/plugins/ca.mover.tuning/age_mover: line 73: /tmp/Mover/Mover_Tuning_2023-06-19T014906.log: No such file or directory
Jun 19 01:49:06 unraid move: /usr/local/emhttp/plugins/ca.mover.tuning/age_mover: line 73: /tmp/Mover/Mover_Tuning_2023-06-19T014906.log: No such file or directory
Jun 19 01:49:06 unraid move: /usr/local/emhttp/plugins/ca.mover.tuning/age_mover: line 73: /tmp/Mover/Mover_Tuning_2023-06-19T014906.log: No such file or directory
Jun 19 01:49:06 unraid move: /usr/local/emhttp/plugins/ca.mover.tuning/age_mover: line 73: /tmp/Mover/Mover_Tuning_2023-06-19T014906.log: No such file or directory
Jun 19 01:49:06 unraid move: /usr/local/emhttp/plugins/ca.mover.tuning/age_mover: line 73: /tmp/Mover/Mover_Tuning_2023-06-19T014906.log: No such file or directory
Jun 19 01:49:06 unraid move: mover: started
Jun 19 01:49:06 unraid move: find: '/tmp/Mover/*': No such file or directory
Jun 19 01:49:06 unraid move: find: '/tmp/Mover/*': No such file or directory
Jun 19 01:49:06 unraid move: file: /mnt/disk1/system/docker/docker.img
Jun 19 01:49:06 unraid move: move_object: /mnt/disk1/system/docker/docker.img File exists
Jun 19 01:49:06 unraid move: file: /mnt/disk1/system/libvirt/libvirt.img
Jun 19 01:49:06 unraid move: move_object: /mnt/disk1/system/libvirt/libvirt.img File exists
Jun 19 01:49:06 unraid move: mover: finished

The logs in /tmp/Mover on the first run were as follows:

Spoiler

mvlogger: CACHETHRESH is blank
mvlogger: Share Name Only: Test share
mvlogger: Cache Pool Name: cache 
mvlogger: cache Threshold Pct: 
mvlogger: OVERALL Threshold: 0
mvlogger: Share Path: /mnt/cache/Test share
mvlogger: cache is zfs.
mvlogger: Pool Pct Used: 2 %
mvlogger: DFTPCT LIMIT USED FOR SETTING: 0
mvlogger: Threshold Used: 0
mvlogger: Skipfiletypes string: find "/mnt/cache/Test share" -depth
mvlogger: Share Name Only: isos
mvlogger: Cache Pool Name: cache 
mvlogger: cache Threshold Pct: 
mvlogger: OVERALL Threshold: 0
mvlogger: Share Path: /mnt/cache/isos
mvlogger: cache is zfs.
mvlogger: Pool Pct Used: 2 %
mvlogger: DFTPCT LIMIT USED FOR SETTING: 0
mvlogger: Threshold Used: 0
mvlogger: Skipfiletypes string: find "/mnt/cache/isos" -depth
mvlogger: after customFilelist
mvlogger: After totalsizeFilelist
mvlogger: -----------------------------------------Running Array Files to Cache drives----------------------------------------------
mvlogger: Running to cache drive
mvlogger: Moving files from Array to Cache!
mvlogger: File to Cache filelist created
mvlogger: Cache File List: /tmp/Mover/Cache_Tuning_2023-06-19T014906.list
mvlogger: It's a Directory don't include: /mnt/disk1/system/docker
mvlogger: It's a Directory don't include: /mnt/disk1/system/libvirt
mvlogger: It's a Directory don't include: /mnt/disk1/system
mvlogger: make customArray File List
mvlogger: write first status file after to array
mvlogger: Hard Link Status: false
mvlogger: No /tmp/Mover/Custom_Mover_Directory_2023-06-19T014906.list created as no empty directories
mvlogger: Position:  1     Path: "/mnt/disk1/system/docker/docker.img"   Size: 21474836480
mvlogger: Position:  2     Path: "/mnt/disk1/system/libvirt/libvirt.img"   Size: 1073741824

The second run I had the following systemlogs:

Spoiler

Jun 19 01:55:52 unraid root: Starting Mover
Jun 19 01:55:52 unraid root: ionice -c 2 -n 0 nice -n 0 /usr/local/emhttp/plugins/ca.mover.tuning/age_mover start 0 0 0 '' mp3, jpg, jpeg, .txt '' '' no 5 'yes' ''
Jun 19 01:55:52 unraid move: Log Level: 1
Jun 19 01:55:52 unraid move: mover: started
Jun 19 01:55:52 unraid move: file: /mnt/disk1/system/docker/docker.img
Jun 19 01:55:52 unraid move: move_object: /mnt/disk1/system/docker/docker.img File exists
Jun 19 01:55:52 unraid move: file: /mnt/disk1/system/libvirt/libvirt.img
Jun 19 01:55:52 unraid move: move_object: /mnt/disk1/system/libvirt/libvirt.img File exists
Jun 19 01:55:52 unraid move: mover: finished

And the following results in /tmp/Mover:

Spoiler

mvlogger: *********************************MOVER START*******************************
mvlogger: Age supplied 0
mvlogger: Size supplied
mvlogger: Sparness supplied
mvlogger: No Skipfiles Argument Supplied
mvlogger: SKIP FILE TYPES: mp3,
mvlogger: BEFORESCRIPT: jpg,
mvlogger: AFTERSCRIPT: jpeg,
mvlogger: CTIME Argument: .txt
mvlogger: No Original Mover Threshold Percent Supplied
mvlogger: No Test Mode Argument Supplied
mvlogger: Ignore Hidden Files Argument: no
mvlogger: CACHETHRESH is blank
mvlogger: Share Name Only: Test share
mvlogger: Cache Pool Name: cache 
mvlogger: cache Threshold Pct: 
mvlogger: OVERALL Threshold: 0
mvlogger: Share Path: /mnt/cache/Test share
mvlogger: cache is zfs.
mvlogger: Pool Pct Used: 2 %
mvlogger: DFTPCT LIMIT USED FOR SETTING: 0
mvlogger: Threshold Used: 0
mvlogger: Skipfiletypes string: find "/mnt/cache/Test share" -depth
mvlogger: Share Name Only: isos
mvlogger: Cache Pool Name: cache 
mvlogger: cache Threshold Pct: 
mvlogger: OVERALL Threshold: 0
mvlogger: Share Path: /mnt/cache/isos
mvlogger: cache is zfs.
mvlogger: Pool Pct Used: 2 %
mvlogger: DFTPCT LIMIT USED FOR SETTING: 0
mvlogger: Threshold Used: 0
mvlogger: Skipfiletypes string: find "/mnt/cache/isos" -depth
mvlogger: after customFilelist
mvlogger: After totalsizeFilelist
mvlogger: -----------------------------------------Running Array Files to Cache drives----------------------------------------------
mvlogger: Running to cache drive
mvlogger: Moving files from Array to Cache!
mvlogger: File to Cache filelist created
mvlogger: Cache File List: /tmp/Mover/Cache_Tuning_2023-06-19T015552.list
mvlogger: It's a Directory don't include: /mnt/disk1/system/docker
mvlogger: It's a Directory don't include: /mnt/disk1/system/libvirt
mvlogger: It's a Directory don't include: /mnt/disk1/system
mvlogger: make customArray File List
mvlogger: write first status file after to array
mvlogger: Hard Link Status: false
mvlogger: No /tmp/Mover/Custom_Mover_Directory_2023-06-19T015552.list created as no empty directories
mvlogger: Position:  1     Path: "/mnt/disk1/system/docker/docker.img"   Size: 21474836480
mvlogger: Position:  2     Path: "/mnt/disk1/system/libvirt/libvirt.img"   Size: 1073741824

TO NOTE:
On first run the logs in /tmp/Mover appear to be missing logs at the beginning. On second run, it works fine.

I did more testing and this is the case on every run after the system reboots.

You'll notice that I broke it and didn't actually run it in test mode because using the exclude file types option I had spaces between them. I noticed this because I was going to add an extra variable to this (share name) for my settings-per-share update I am making but was worried that if they had a space it would break it. This could also be an issue for people using a skip files list, before script or after script on a share/folder path that has spaces so I have fixed it. I will push what I have done but you can feel free to reject, or change it

EDIT: Created pull request for whitespace issue fix. Still no idea on the /tmp/Mover file/directory not being created issue.

Edited June 19, 2023 by Swarles
Put code into spoilers to reduce clutter

hugenbdd · June 18, 2023

5 hours ago, DontWorryScro said:

I am completely up to date on plugins, have the very latest and greatest and yet im still getting tons of:

move: error: move, 392: No such file or directory (2): lstat: /mnt/cache_nvme/*

any idea why this is still happening?

I would need to see log files from /tmp/Mover

Also, has the name of the cache been renamed at any point? Sometimes the config file is not updated.

hugenbdd · June 18, 2023

4 hours ago, Swarles said:

I can also confirm that on a fresh install the mover does not run correctly I think due to the changes to using /tmp/Mover for log files. It appears the directory is not made to begin with. I tested the following before running the Mover for the first time on a fresh plugin install and then after running the mover got the second result. I'm not sure why this is the case because it should be made immediately on the first age_mover / mvlogger() run....hmmmmm.

root@unraid:~# /tmp/Mover
bash: /tmp/Mover: No such file or directory
root@unraid:~# /tmp/Mover
bash: /tmp/Mover: Is a directory

My first run also resulted in these systemlogs:

Jun 19 01:49:06 unraid emhttpd: shcmd (92): /usr/local/sbin/mover |& logger -t move &
Jun 19 01:49:06 unraid root: Starting Mover
Jun 19 01:49:06 unraid root: ionice -c 2 -n 0 nice -n 0 /usr/local/emhttp/plugins/ca.mover.tuning/age_mover start 0 0 0 '' mp3, jpg, jpeg, .txt '' '' no 5 'yes' ''
Jun 19 01:49:06 unraid move: Log Level: 1
Jun 19 01:49:06 unraid move: /usr/local/emhttp/plugins/ca.mover.tuning/age_mover: line 73: /tmp/Mover/Mover_Tuning_2023-06-19T014906.log: No such file or directory
Jun 19 01:49:06 unraid move: /usr/local/emhttp/plugins/ca.mover.tuning/age_mover: line 73: /tmp/Mover/Mover_Tuning_2023-06-19T014906.log: No such file or directory
Jun 19 01:49:06 unraid move: /usr/local/emhttp/plugins/ca.mover.tuning/age_mover: line 73: /tmp/Mover/Mover_Tuning_2023-06-19T014906.log: No such file or directory
Jun 19 01:49:06 unraid move: /usr/local/emhttp/plugins/ca.mover.tuning/age_mover: line 73: /tmp/Mover/Mover_Tuning_2023-06-19T014906.log: No such file or directory
Jun 19 01:49:06 unraid move: /usr/local/emhttp/plugins/ca.mover.tuning/age_mover: line 73: /tmp/Mover/Mover_Tuning_2023-06-19T014906.log: No such file or directory
Jun 19 01:49:06 unraid move: /usr/local/emhttp/plugins/ca.mover.tuning/age_mover: line 73: /tmp/Mover/Mover_Tuning_2023-06-19T014906.log: No such file or directory
Jun 19 01:49:06 unraid move: /usr/local/emhttp/plugins/ca.mover.tuning/age_mover: line 73: /tmp/Mover/Mover_Tuning_2023-06-19T014906.log: No such file or directory
Jun 19 01:49:06 unraid move: /usr/local/emhttp/plugins/ca.mover.tuning/age_mover: line 73: /tmp/Mover/Mover_Tuning_2023-06-19T014906.log: No such file or directory
Jun 19 01:49:06 unraid move: /usr/local/emhttp/plugins/ca.mover.tuning/age_mover: line 73: /tmp/Mover/Mover_Tuning_2023-06-19T014906.log: No such file or directory
Jun 19 01:49:06 unraid move: /usr/local/emhttp/plugins/ca.mover.tuning/age_mover: line 73: /tmp/Mover/Mover_Tuning_2023-06-19T014906.log: No such file or directory
Jun 19 01:49:06 unraid move: /usr/local/emhttp/plugins/ca.mover.tuning/age_mover: line 73: /tmp/Mover/Mover_Tuning_2023-06-19T014906.log: No such file or directory
Jun 19 01:49:06 unraid move: /usr/local/emhttp/plugins/ca.mover.tuning/age_mover: line 73: /tmp/Mover/Mover_Tuning_2023-06-19T014906.log: No such file or directory
Jun 19 01:49:06 unraid move: mover: started
Jun 19 01:49:06 unraid move: find: '/tmp/Mover/*': No such file or directory
Jun 19 01:49:06 unraid move: find: '/tmp/Mover/*': No such file or directory
Jun 19 01:49:06 unraid move: file: /mnt/disk1/system/docker/docker.img
Jun 19 01:49:06 unraid move: move_object: /mnt/disk1/system/docker/docker.img File exists
Jun 19 01:49:06 unraid move: file: /mnt/disk1/system/libvirt/libvirt.img
Jun 19 01:49:06 unraid move: move_object: /mnt/disk1/system/libvirt/libvirt.img File exists
Jun 19 01:49:06 unraid move: mover: finished

The logs in /tmp/Mover on the first run were as follows:

mvlogger: CACHETHRESH is blank
mvlogger: Share Name Only: Test share
mvlogger: Cache Pool Name: cache 
mvlogger: cache Threshold Pct: 
mvlogger: OVERALL Threshold: 0
mvlogger: Share Path: /mnt/cache/Test share
mvlogger: cache is zfs.
mvlogger: Pool Pct Used: 2 %
mvlogger: DFTPCT LIMIT USED FOR SETTING: 0
mvlogger: Threshold Used: 0
mvlogger: Skipfiletypes string: find "/mnt/cache/Test share" -depth
mvlogger: Share Name Only: isos
mvlogger: Cache Pool Name: cache 
mvlogger: cache Threshold Pct: 
mvlogger: OVERALL Threshold: 0
mvlogger: Share Path: /mnt/cache/isos
mvlogger: cache is zfs.
mvlogger: Pool Pct Used: 2 %
mvlogger: DFTPCT LIMIT USED FOR SETTING: 0
mvlogger: Threshold Used: 0
mvlogger: Skipfiletypes string: find "/mnt/cache/isos" -depth
mvlogger: after customFilelist
mvlogger: After totalsizeFilelist
mvlogger: -----------------------------------------Running Array Files to Cache drives----------------------------------------------
mvlogger: Running to cache drive
mvlogger: Moving files from Array to Cache!
mvlogger: File to Cache filelist created
mvlogger: Cache File List: /tmp/Mover/Cache_Tuning_2023-06-19T014906.list
mvlogger: It's a Directory don't include: /mnt/disk1/system/docker
mvlogger: It's a Directory don't include: /mnt/disk1/system/libvirt
mvlogger: It's a Directory don't include: /mnt/disk1/system
mvlogger: make customArray File List
mvlogger: write first status file after to array
mvlogger: Hard Link Status: false
mvlogger: No /tmp/Mover/Custom_Mover_Directory_2023-06-19T014906.list created as no empty directories
mvlogger: Position:  1     Path: "/mnt/disk1/system/docker/docker.img"   Size: 21474836480
mvlogger: Position:  2     Path: "/mnt/disk1/system/libvirt/libvirt.img"   Size: 1073741824

The second run I had the following systemlogs:

Jun 19 01:55:52 unraid root: Starting Mover
Jun 19 01:55:52 unraid root: ionice -c 2 -n 0 nice -n 0 /usr/local/emhttp/plugins/ca.mover.tuning/age_mover start 0 0 0 '' mp3, jpg, jpeg, .txt '' '' no 5 'yes' ''
Jun 19 01:55:52 unraid move: Log Level: 1
Jun 19 01:55:52 unraid move: mover: started
Jun 19 01:55:52 unraid move: file: /mnt/disk1/system/docker/docker.img
Jun 19 01:55:52 unraid move: move_object: /mnt/disk1/system/docker/docker.img File exists
Jun 19 01:55:52 unraid move: file: /mnt/disk1/system/libvirt/libvirt.img
Jun 19 01:55:52 unraid move: move_object: /mnt/disk1/system/libvirt/libvirt.img File exists
Jun 19 01:55:52 unraid move: mover: finished

And the following results in /tmp/Mover:

mvlogger: *********************************MOVER START*******************************
mvlogger: Age supplied 0
mvlogger: Size supplied
mvlogger: Sparness supplied
mvlogger: No Skipfiles Argument Supplied
mvlogger: SKIP FILE TYPES: mp3,
mvlogger: BEFORESCRIPT: jpg,
mvlogger: AFTERSCRIPT: jpeg,
mvlogger: CTIME Argument: .txt
mvlogger: No Original Mover Threshold Percent Supplied
mvlogger: No Test Mode Argument Supplied
mvlogger: Ignore Hidden Files Argument: no
mvlogger: CACHETHRESH is blank
mvlogger: Share Name Only: Test share
mvlogger: Cache Pool Name: cache 
mvlogger: cache Threshold Pct: 
mvlogger: OVERALL Threshold: 0
mvlogger: Share Path: /mnt/cache/Test share
mvlogger: cache is zfs.
mvlogger: Pool Pct Used: 2 %
mvlogger: DFTPCT LIMIT USED FOR SETTING: 0
mvlogger: Threshold Used: 0
mvlogger: Skipfiletypes string: find "/mnt/cache/Test share" -depth
mvlogger: Share Name Only: isos
mvlogger: Cache Pool Name: cache 
mvlogger: cache Threshold Pct: 
mvlogger: OVERALL Threshold: 0
mvlogger: Share Path: /mnt/cache/isos
mvlogger: cache is zfs.
mvlogger: Pool Pct Used: 2 %
mvlogger: DFTPCT LIMIT USED FOR SETTING: 0
mvlogger: Threshold Used: 0
mvlogger: Skipfiletypes string: find "/mnt/cache/isos" -depth
mvlogger: after customFilelist
mvlogger: After totalsizeFilelist
mvlogger: -----------------------------------------Running Array Files to Cache drives----------------------------------------------
mvlogger: Running to cache drive
mvlogger: Moving files from Array to Cache!
mvlogger: File to Cache filelist created
mvlogger: Cache File List: /tmp/Mover/Cache_Tuning_2023-06-19T015552.list
mvlogger: It's a Directory don't include: /mnt/disk1/system/docker
mvlogger: It's a Directory don't include: /mnt/disk1/system/libvirt
mvlogger: It's a Directory don't include: /mnt/disk1/system
mvlogger: make customArray File List
mvlogger: write first status file after to array
mvlogger: Hard Link Status: false
mvlogger: No /tmp/Mover/Custom_Mover_Directory_2023-06-19T015552.list created as no empty directories
mvlogger: Position:  1     Path: "/mnt/disk1/system/docker/docker.img"   Size: 21474836480
mvlogger: Position:  2     Path: "/mnt/disk1/system/libvirt/libvirt.img"   Size: 1073741824

TO NOTE:
On first run the logs in /tmp/Mover appear to be missing logs at the beginning. On second run, it works fine.

I did more testing and this is the case on every run after the system reboots.

You'll notice that I broke it and didn't actually run it in test mode because using the exclude file types option I had spaces between them. I noticed this because I was going to add an extra variable to this (share name) for my settings-per-share update I am making but was worried that if they had a space it would break it. This could also be an issue for people using a skip files list, before script or after script on a share/folder path that has spaces so I have fixed it. I will push what I have done but you can feel free to reject, or change it

EDIT: Created pull request for whitespace issue fix. Still no idea on the /tmp/Mover file/directory not being created issue.

Yes, since it's in tmp it will get destroyed on every reboot. I think I just need to create the directory and then sleep for 1 second before it moves on every Mover run.

I'm not sure if I updated the age_mover file on github. I'll try and do that in the next few hours. (It is obviously updated in the tar package)

DontWorryScro · June 20, 2023

I did notice in all my system log error spam of

move: error: move, 392: No such file or directory (2): lstat: /mnt/cache_nvme/*

that it seems this only happens when Mover runs on its nightly schedule. When I run it manually I do not get these missing file/directory messages.

hugenbdd · June 20, 2023

16 minutes ago, DontWorryScro said:
I did notice in all my system log error spam of
move: error: move, 392: No such file or directory (2): lstat: /mnt/cache_nvme/*
that it seems this only happens when Mover runs on its nightly schedule. When I run it manually I do not get these missing file/directory messages.

Can you DM the latest list of log files under /tmp/Mover ?

There should be about 6 files. (example below) The two important ones that I will mainly look at are Mover_Tuning_<DATETIME>.log and Custom_Mover_Tuning_<DATETIME>.list

Example:

-rw-r--r-- 1 root root 8742 Jun 19 10:09 Mover_Tuning_2023-06-19T100928.list
-rw-r--r-- 1 root root 6777 Jun 19 10:09 Custom_Mover_Tuning_2023-06-19T100928.list
-rw-r--r-- 1 root root 2861 Jun 19 10:09 Custom_Mover_Directory_2023-06-19T100928.list
-rw-r--r-- 1 root root 0 Jun 19 10:09 Cache_Tuning_2023-06-19T100928.list
-rw-r--r-- 1 root root 0 Jun 19 10:09 Custom_Cache_Tuning_2023-06-19T100928.list
-rw-r--r-- 1 root root 42798 Jun 19 10:09 Mover_Tuning_2023-06-19T100928.log

Edited June 20, 2023 by hugenbdd

Swarles · June 20, 2023

15 hours ago, DontWorryScro said:
I did notice in all my system log error spam of
move: error: move, 392: No such file or directory (2): lstat: /mnt/cache_nvme/*

Do you happen to have this setting enabled? ---> "Move All from Cache-Yes shares when disk is above a certain percentage:"

If yes, what is your percentage set to? And what is the percentage of storage taken up on your pool (cache_nvme)?

I suspect this may be the source of the error.

16 hours ago, DontWorryScro said:

that it seems this only happens when Mover runs on its nightly schedule. When I run it manually I do not get these missing file/directory messages.

Do you have this setting enabled? --> "Move Now button follows plug-in filters:"

If you do not, the behaviour you are seeing makes sense because (I believe, but correct me if I am wrong) when you run it manually it does not use the plugin mover settings. My suspicion is that this is the result of files attempting to be moved twice but I would like to confirm. I believe harmless, but still annoying.

DontWorryScro · June 21, 2023

13 hours ago, Swarles said:

Do you happen to have this setting enabled? ---> "Move All from Cache-Yes shares when disk is above a certain percentage:"

If yes, what is your percentage set to? And what is the percentage of storage taken up on your pool (cache_nvme)?

I suspect this may be the source of the error.

Do you have this setting enabled? --> "Move Now button follows plug-in filters:"

If you do not, the behaviour you are seeing makes sense because (I believe, but correct me if I am wrong) when you run it manually it does not use the plugin mover settings. My suspicion is that this is the result of files attempting to be moved twice but I would like to confirm. I believe harmless, but still annoying.

image.png.59285d643a79e57b505c45fe364575db.png

at the moment.

yamraid · June 21, 2023

Just wanted to drop in a mention of some strange behaviour I'm seeing using 2023.06.16 and Unraid v6.11.5. I'm seeing that some of the filter options seem to be flipped. Eg, the setting 'Move files off cache based on age?' to 'Yes' results on the options being disabled:

Swarles · June 21, 2023

5 hours ago, DontWorryScro said:

at the moment.

Thanks! This confirms the behaviour.

Because your nvme_cache pool is above 25% used, it will always move all the shares that are (cache--->array) to the array and ignoring other mover tuning settings. I suspect that you have other (non cache-->array) shares that are taking up more than 25% of the nvme_cache pool, so this behaviour will always happen. This results in all files in those shares being listed for a move, but then they are again listed later for a move based on the mover settings (because the mover only runs once at the end). As a result, it gets moved and then later when it tries to move it again, it no longer is in the original directory and hence lstat error. Hugenbd has implemented a check to see if the file/directory exists before attempting the move. The error is harmless but will be fixed in the next release

Swarles · June 21, 2023

3 hours ago, yamraid said:

Just wanted to drop in a mention of some strange behaviour I'm seeing using 2023.06.16 and Unraid v6.11.5. I'm seeing that some of the filter options seem to be flipped. Eg, the setting 'Move files off cache based on age?' to 'Yes' results on the options being disabled:

Hmm strange, unfortunately I can't change my unraid version to check but can you use the following in terminal:

cat /usr/local/emhttp/plugins/ca.mover.tuning/Mover.tuning.page

and show me the following lines (11-21)

Spoiler

$plugin='ca.mover.tuning';
$cfg = parse_plugin_cfg($plugin);
$cronDisabled = ($cfg['force'] == "no") ? "disabled" : "";
$ageDisabled = ($cfg['age'] == "no") ? "disabled" : "";
$sizefDisabled = ($cfg['sizef'] == "no") ? "disabled" : "";
$sparsnessfDisabled = ($cfg['sparsnessf'] == "no") ? "disabled" : "";
$filetypesfDisabled = ($cfg['filetypesf'] == "no") ? "disabled" : "";
$filelistfDisabled = ($cfg['filelistf'] == "no") ? "disabled" : "";
$omoverthreshDisabled = ($cfg['omovercfg'] == "no") ? "disabled" : "";
$om = $cfg['threshold'];
$om = $om + 5;

yamraid · June 21, 2023

Quote

and show me the following lines (11-21)

Here's the output:

Spoiler

$plugin='ca.mover.tuning';
$cfg = parse_plugin_cfg($plugin);
$cronDisabled = ($cfg['force'] == "no") ? "disabled" : "";
$ageDisabled = ($cfg['age'] == "no") ? "disabled" : "";
$sizefDisabled = ($cfg['sizef'] == "no") ? "disabled" : "";
$sparsnessfDisabled = ($cfg['sparsnessf'] == "no") ? "disabled" : "";
$filetypesfDisabled = ($cfg['filetypesf'] == "no") ? "disabled" : "";
$filelistfDisabled = ($cfg['filelistf'] == "no") ? "disabled" : "";
$omoverthreshDisabled = ($cfg['omovercfg'] == "no") ? "disabled" : "";
$om = $cfg['threshold'];
$om = $om + 5;

Edit - so, if I change 'No' to 'Yes', then hit save, the options behave as expected. Here's the outrput again with the days value set to 3:

Spoiler

$plugin='ca.mover.tuning';
$cfg = parse_plugin_cfg($plugin);
$cronDisabled = ($cfg['force'] == "no") ? "disabled" : "";
$ageDisabled = ($cfg['age'] == "no") ? "disabled" : "";
$sizefDisabled = ($cfg['sizef'] == "no") ? "disabled" : "";
$sparsnessfDisabled = ($cfg['sparsnessf'] == "no") ? "disabled" : "";
$filetypesfDisabled = ($cfg['filetypesf'] == "no") ? "disabled" : "";
$filelistfDisabled = ($cfg['filelistf'] == "no") ? "disabled" : "";
$omoverthreshDisabled = ($cfg['omovercfg'] == "no") ? "disabled" : "";
$om = $cfg['threshold'];
$om = $om + 5;

It's...the same?

Edited June 21, 2023 by yamraid

Swarles · June 21, 2023

15 minutes ago, yamraid said:

It's...the same?

Yeah so that's just a script that specifies that if setting "age" is set to "no" then make it disabled. It wont change if you edit the settings but if it was set to "yes" somehow that would explain it. It looks like the script is correct so I am not sure why you are getting that behaviour. Unfortunately I don't know much at all about HTML which the rest of that page uses but I can say the code for that page hasn't changed in 2 years. It might be something to do with your unraid version or another glitch. Do you have any other plugins installed and do they have the same behaviour? (for eg. parity check tuning). I might just suggest uninstalling and then re-installing it and seeing if it still happens.

Edited June 21, 2023 by Swarles

yamraid · June 21, 2023

3 minutes ago, Swarles said:

Yeah so that's just a script that specifies that if setting "age" is set to "no" then make it disabled. It wont change but if it was set to "yes" somehow that would explain it. It looks like the script is correct so I am not sure why you are getting that behaviour. Unfortunately I don't know much at all about HTML which the rest of that page uses but I can say the code for that page hasn't changed in 2 years. It might be something to do with your unraid version or another glitch. Do you have any other plugins installed and do they have the same behaviour? (for eg. parity check tuning). I might just suggest uninstalling and then re-installing it and seeing if it still happens.

Ah ok - thanks for the explanation. I did try an uninstall/reinstall and that didn't have any effect. I've not noticed the behaviour with any of my other plugins, but I'll certainly keep an eye on it. Ultimately, it's now working after saving but I'll see if I can identify any correlating behaviour in other areas. Appreciate your help!

Edited June 21, 2023 by yamraid
Formatting

[Plugin] Mover Tuning

Recommended Posts

Link to comment

Top Posters In This Topic

Popular Days

Top Posters In This Topic

Popular Days

Popular Posts

Squid

hugenbdd

hugenbdd

Posted Images

Link to comment

Link to comment

Link to comment

Link to comment

Link to comment

Link to comment

Link to comment

Link to comment

Link to comment

Link to comment

Link to comment

Link to comment

Link to comment

Link to comment

Link to comment

Link to comment

Link to comment

Link to comment

Link to comment

Link to comment

Link to comment

Link to comment

Link to comment

Link to comment

Join the conversation