[Plugin] Mover Tuning

[ZGief]

@Reynald I'm a programming novice, but looking at this block of code around the error, wouldn't the POOLSIZE variable be created in a context (the if / else) that isn't available below?

 

[Reynald]

50 minutes ago, ZGief said:

@Reynald I'm a programming novice, but looking at this block of code around the error, wouldn't the POOLSIZE variable be created in a context (the if / else) that isn't available below?

 

Yes, you got it !

In your log, the error:

df: /mnt/cache: No such file or directory

is relative to the else.

Now we need to understand why $PRIMARYSTORAGENAME is cache/ instead of cache/domains. Can you please check if domains folder is present in cache?

Edited August 8 by Reynald
grammar, seems Ican speak bash better than english :D

[ZGief]

1 hour ago, Reynald said:

Now we need to understand why $PRIMARYSTORAGENAME is cache/ instead of cache/domains. Can you please check if domains folder is present in cache?

 

I think i understand your question. My interpretation:

 

Looking at lines

412

for SHARECFG in /boot/config/shares/*.cfg; do

465

POOLSIZE=$(df --output=size --block-size=1 /mnt/$PRIMARYSTORAGENAME | tail -n 1)

438

PRIMARYSTORAGENAME=$(grep "shareCachePool=" "$SHARECFG" | cut -d'=' -f 2 | tr -d '"' | tr -d '\r')

 

PRIMARYSTORAGENAME gets its value from SHARECFG (in a loop) looking through the config files in /boot/config/shares

 

my configs in that dir

 

The "data.cfg" is the one I care about. It is the cache SSD that holds the 'data' directory.

 

 

This folder is present in the cache.

 

However, you said "check if the 'domains' folder is in the cache". I don't know anything about a domains folder. That isn't a share i setup and isn't in my shares list in the UI.

 

The doamains cfg is here if that helps. I don't run vms. Not sure how the domains file got here to be honest.

[ZGief]

Ok, pretty sure I solve my issue.

 

TLDR - I only have 3 shares: data, appdata, system. Deleting the cfg files that don't exist fixed it.

 

However, as you can see above there were three .cfg files in the /boot/config/shares folder that don't match up.

 

tmp.cfg - i created a 'tmp' share thinking i needed it, but then deleted it in the UI a few days ago

isos - i think unraid created this automatially to support VMs but i don't use VMs so i deleted this folder form the UI too

domains - no idea where this came from, maybe it was there on initial install of unraid a few weeks ago, probably deleted it in the UI too

 

Anyway, even those these shares no longer exist in the UI, these .cfg files are hanging around. And mover tuner uses them to do its work. You can see that here in the /usr/local/emhttp/plugins/ca.mover.tuning/age_mover file:

 

It is looping through all cfg files in that folder trying to get useful info out of them to set variables and such but the files are empty and worthless causing it to hang when it gets to the part of the script that requires those variables.

 

Possible solution: write a few more checks to make sure the cfg file is not one of these orphaned share configs and skip them if they are.

[alturismo]

4 hours ago, ZGief said:

domains - no idea where this came from, maybe it was there on initial install of unraid a few weeks ago,

exactly

 

4 hours ago, ZGief said:

Possible solution: write a few more checks to make sure the cfg file is not one of these orphaned share configs and skip them if they are.

should have been fixed already a few days ago, i stumbled over the same issue

 

when i remember correctly, it was fixed, i cleaned up my "old" unused shares afterwards

 

to test again now, i created a test share and deleted it from the share tab, it was gone, so all clear if we remove through the webui, now we just remove the root folder from cache, the config persists while the share is gone (as there is no more root folder)

 

root@AlsServerII:~# ls -la /mnt/cache/test_mv/
total 0
drwxrwxrwx  2 nobody users   6 Aug  9 06:23 ./
drwxrwxrwx 13 nobody users 168 Aug  9 06:23 ../
root@AlsServerII:~# rm -r /mnt/cache/test_mv/
root@AlsServerII:~# ls -la /boot/config/shares/
total 48
drw-r--r--  2 root root 4096 Aug  9 06:23 ./
drw-r--r-- 12 root root 4096 Jul 29 06:15 ../
-rw-r--r--  1 root root  445 Nov  3  2023 Daten.cfg
-rw-r--r--  1 root root  445 Nov  3  2023 Dokumente.cfg
-rw-r--r--  1 root root  444 Nov  3  2023 Media.cfg
-rw-r--r--  1 root root  428 Apr  5 23:04 Nextcloud.cfg
-rw-r--r--  1 root root  436 Nov  3  2023 appdata.cfg
-rw-r--r--  1 root root  464 Nov  3  2023 domains.cfg
-rw-r--r--  1 root root  455 Nov  3  2023 isos.cfg
-rw-r--r--  1 root root  445 Nov  3  2023 lxc.cfg
-rw-r--r--  1 root root  457 Nov  3  2023 system.cfg
-rw-r--r--  1 root root  448 Aug  9 06:23 test_mv.cfg
root@AlsServerII:~# ls -la /mnt/cache/tes*
/bin/ls: cannot access '/mnt/cache/tes*': No such file or directory
root@AlsServerII:~#

 

when i now run mover, looks fine to me ...

 

...
Aug  9 06:25:32 AlsServerII move: ------------------------------------------------------------ Processing test_mv share -----------------------------------------------------------
Aug  9 06:25:32 AlsServerII move: Primary storage: cache - used:  55 % (1014.75 GiB)
Aug  9 06:25:32 AlsServerII move: Secondary storage: none
Aug  9 06:25:32 AlsServerII move: Share Information: Name: test_mv - Path: /mnt/cache/test_mv
Aug  9 06:25:32 AlsServerII move: /mnt/cache/test_mv does not exist. Is the share still used? Consider removing /boot/config/shares/test_mv.cfg if not.
Aug  9 06:25:32 AlsServerII move: => Skipping
Aug  9 06:25:32 AlsServerII move: ************************************************************ ANALYSING MOVING ACTIONS ***********************************************************
...

 

so yes, looks already fixed to me, are you using the latest version ?

[zoggy]

On 8/7/2024 at 4:03 PM, Reynald said:

Thanks for your comment. Is it still not moving anything or was it test mode?

Good point for bc. Anyway, as unraid 7 is based on slackware 15, as recent 6.x versions, I don't see a problem (I myself choose to continue using nerdpack as long as repo is alive), but yes, I will definitly think about this point

 

I will move the option on top in second position, that makes sense  .
I will also remove this default when the plugin will be stable enough, I've completly rewritten some part and I consider that it's safer to check in test mode, before going live, at the moment.

It was not, I had given up on your plugin and went back to hugenbdd/original version and did the symlink and its working fine..  so at some point I'm sure I'll try it out again but at this point it didn't seem like it was ready with all the active development and bugs.

[Reynald]

6 hours ago, ZGief said:

Ok, pretty sure I solve my issue.

 

TLDR - I only have 3 shares: data, appdata, system. Deleting the cfg files that don't exist fixed it.

 

However, as you can see above there were three .cfg files in the /boot/config/shares folder that don't match up.

 

tmp.cfg - i created a 'tmp' share thinking i needed it, but then deleted it in the UI a few days ago

isos - i think unraid created this automatially to support VMs but i don't use VMs so i deleted this folder form the UI too

domains - no idea where this came from, maybe it was there on initial install of unraid a few weeks ago, probably deleted it in the UI too

 

Anyway, even those these shares no longer exist in the UI, these .cfg files are hanging around. And mover tuner uses them to do its work. You can see that here in the /usr/local/emhttp/plugins/ca.mover.tuning/age_mover file:

 

It is looping through all cfg files in that folder trying to get useful info out of them to set variables and such but the files are empty and worthless causing it to hang when it gets to the part of the script that requires those variables.

 

Possible solution: write a few more checks to make sure the cfg file is not one of these orphaned share configs and skip them if they are.

Very nice analyse @ZGief, and thanks for complements @alturismo

There is already a check on line 494:

if [ ! -d "$SHAREPATH" ] && [ $SHAREUSECACHE != "no" ]; then
# Do not process this pool if path does not exist
mvlogger "$SHAREPATH does not exist. Is the share still used? Consider removing $SHARECFG if not."
mvlogger "=> Skipping"
continue # Move to the next iteration of the loop

 

However, this adress a deleted or removed share, but deleted or renamed pool is not checked.

 

Is your cachepool still named cache? (that's in your domains.cfg)

You can modify the code like this, around line 438

 

        PRIMARYSTORAGENAME=$(grep "shareCachePool=" "$SHARECFG" | cut -d'=' -f 2 | tr -d '"' | tr -d '\r')
        if [ -z "$PRIMARYSTORAGENAME" ]; then
            PRIMARYSTORAGENAME="user0"
        fi
        # Check if the storage exists
        if [ ! -d "$PRIMARYSTORAGENAME" ]; then
            # Do not process this pool if path does not exist
            mvlogger "$PRIMARYSTORAGENAME does not exist. Is the pool still used? Consider removing $SHARECFG if not."
            mvlogger "=> Skipping"
            continue  # Move to the next iteration of the loop
        fi

 

I'm going to add this in next version

[zoggy]

might be worth while creating your own form post for your version of the plugin mover to do a proper page of whats its features over squids (or hugenbdd even).. also can do proper support link in your plugin and so on. as it can easily get confused between versions since one doesnt upgrade the other they are just basically two same named plugins at the moment..

Edited August 10 by zoggy

[Reynald]

That's a good point. I went a bit far away from original mover tuning plugin and it probably worth it.

There are some specific use case that are not completely working (dirty config files, config without array, or something we don't have time to investigate yet)

I've honestly thought about naming this Mover Tuning V2 or beta, but in the beginning we worked together with Hugenbdd and Swarles.

 

@alturismo what do you think please ? If we decide a split would you help me moving relative posts in a new thread I would create ? Or finally, as 2023 version is not maintained anymore we can stay here ?

I don't mind in any case, my goal is to bring a functional plugin to the community..

Edited August 10 by Reynald

[alturismo]

48 minutes ago, Reynald said:

what do you think please ?

i personally would remain here as this plugin will prolly takeover the "deprecated" anyway sooner or later

 

@Squid what do you say, its your Thread

 

so we wouldnt clutter into 2 seperate Threads in the end ... my personal opinion.

 

52 minutes ago, Reynald said:

If we decide a split would you help me moving relative posts in a new thread I would create ?

sure, just would need some start point  as we have many posts already here.

 

52 minutes ago, Reynald said:

I don't mind in any case, my goal is to bring a functional plugin to the community..

 

me neither any a great 👍for your efforts here.

 

i would wait if @Squid may wants to keep this one rather for ... lets say v1 and moving Threads related to "v2" into a seperate one, or as this one has already all Versions covered  ... stay here.

[Reynald]

On 8/7/2024 at 7:42 PM, zoggy said:

updating to the latest version, it resolves the fatal error but does not move anything...

 

to comment you are showing:

Info: you may optionally install 'bc' from 'NerdTools' to have more precise size value

IMHO: this should not be mentioned in logs, as nerdtools is not really recommended for unraid 7.x nor alive anymore. if you think bc should be included, just included with the plugin...  or offer up an option to enable more precise sizes and blip on there about telling the user to go install bc.

 

Then, your plugin has "Test Mode (dry run):" defaulting to yes.. and as this is buried middle way down in the options it makes it easy to miss (!it got me). I'd recommend moving this to the very top (or bottom like old plugin) or something for better visibility.

I was working on your recommendation until I see that a new version of NerdTools is alive, maintained, and working: gravityfargo/unRAID-NerdTools: Install extra packages for advanced users. (github.com)
I will then use the NerTools mechanism to propose to install bc. I don't want to develop a package manager function in Mover Tuning as it's not the objective (while it's as easy as a wget package, put in extra, and install but I don't want the hassle of managing updates)

Any documented guidance to a cleaner solution will be welcomed

Edited August 10 by Reynald

[Reynald]

2024.08.10:

• Better filtering with ctime=no (R3yn4ld) Shall solve escaping special characters.
• Improved Synchronization (R3yn4ld):
  • Improve synchronization by looking for files on cache first
  • Do not count synchronized files twice (freeing/priming target were half achieved)
  • Optimize Filtering File and Decision loops regarding Rebalance and Synchronize
• Moved test mode on top of Mover Tuning Page
• Add check for primary storage not existing (dust config files)

I think I covered everything and everyone since previous version, except the bc thing

[ZGief]

11 hours ago, Reynald said:

I was working on your recommendation until I see that a new version of NerdTools is alive, maintained, and working: gravityfargo/unRAID-NerdTools: Install extra packages for advanced users. (github.com)

Sweet. How do you know this is maintained? I don't see it in the community apps.

 

Mover tuner question. Does the setting "Script to run before mover (No checks, always runs):" wait for the script provided to complete before running mover?

[motoflyer]

Unraid Version: 6.12.11

Mover Tuning Version: 2024.08.10.1532
Issue: error message when booting Unraid

 

Hello,

 

I just installed the plugin and during booting the server I get this two errors:

 

rm: cannot remove ´/var/run/mover.pid´ : No such file or directory

rm: cannot remove ´/var/run/moversoft.stop´ : No such file or directory

 

Please see enclosed picture.

 

Is this normal or a problem?

[Reynald]

12 hours ago, ZGief said:

Sweet. How do you know this is maintained? I don't see it in the community apps.

Because there are activity I installed it with the URL
 

 

12 hours ago, ZGief said:

Mover tuner question. Does the setting "Script to run before mover (No checks, always runs):" wait for the script provided to complete before running mover?

Yes, and in the logs you should see:

mvlogger "Launching before script: (path of the script)"
mvlogger "Before script finished"

 

[Reynald]

3 hours ago, motoflyer said:

Unraid Version: 6.12.11

Mover Tuning Version: 2024.08.10.1532
Issue: error message when booting Unraid

 

Hello,

 

I just installed the plugin and during booting the server I get this two errors:

 

rm: cannot remove ´/var/run/mover.pid´ : No such file or directory

rm: cannot remove ´/var/run/moversoft.stop´ : No such file or directory

 

Please see enclosed picture.

 

Is this normal or a problem?

Hello,

This is a bug. I added these command in install script to clean leftovers of a previous interrupted execution.
Added some controls for next version.

These two errors do not affect the program's functionality.

[Reynald]

2024.08.11

Allow operation without array if multiple pools (R3yn4ld): Fixed fatal error bug

Fixed error message about mover.pid and softstop file when installing the plugin or booting Unraid

 

@ramjam824 shall be OK for you now

[Nonoss]

Thank's for all your job.

Just one thing. After each plugin update, "Test mode" is forced to Yes, event if in previous settings it was Test mode : No.

 

You decided to force Test mode: Yes after updates ?

[Reynald]

Just now, Nonoss said:

You decided to force Test mode: Yes after updates ?

Yes I did, because I made some major changes in filtering or decision handling. It's stated in the plugin installation window:

  Updating config file
  - Forcing test mode

There shouldn't be any major changes in the next few weeks. The next planned feature is smart caching TV shows, which will optimize storage and playback. We'll introduce this through a beta branch before a full release.

[ramjam824]

11 hours ago, Reynald said:

2024.08.11

Allow operation without array if multiple pools (R3yn4ld): Fixed fatal error bug

Fixed error message about mover.pid and softstop file when installing the plugin or booting Unraid

 

@ramjam824 shall be OK for you now

 

Looks like it's working now.  Will let this run overnight and report back in the morning!  

Thank you @Reynald

[alturismo]

13 hours ago, Reynald said:

2024.08.11

may a question about the bc implementation

 

i see the following message about bc <no>

 

...
Aug 12 05:00:01 AlsServerII root: No install bc argument provided, defaulting to no
Aug 12 05:00:01 AlsServerII root: /usr/local/emhttp/plugins/ca.mover.tuning/age_mover: line 386: =: command not found
Aug 12 05:00:01 AlsServerII root: ***************************************************************** FILTERING FILES ***************************************************************
...

 

now, i have bc installed anyway (i need it for a aspm script)

 

root@AlsServerII:~# bc -v
bc 1.07.1
Copyright 1991-1994, 1997, 1998, 2000, 2004, 2006, 2008, 2012-2017 Free Software Foundation, Inc.
root@AlsServerII:~#

 

so, does this mean you are not using bc in your mover tuning as its not installed through mover tuning ?

 

or is this just ignoring the "install routine" from mover tuning side ?

[Reynald]

2 hours ago, alturismo said:

may a question about the bc implementation

 

i see the following message about bc <no>

...
so, does this mean you are not using bc in your mover tuning as its not installed through mover tuning ?

 

or is this just ignoring the "install routine" from mover tuning side ?

If bc is installed it's used whatever the "install routine" and it's parameter.
I made a typo on line 386, thanks.

I will reduce verbosity in case bc is detected by installation routine. The setting has no effect if bc is already installed.

We know bc is working when we have size with two decimals. If not, it's rounded to the nearest integer (also for TiB which make the information useless)

Edited August 12 by Reynald

[Faceman]

If I enable Rebalance Shares my log gets spammed (to the point of filling the entire log FS) with Rsync throwing errors for files that obviously don't exist. not sure what the normal operation of that function looks like, but it seems like its just trying to move every file from cache to array, whether it actually exists or not.

 

I have also seen it trying to move files from a share on my secondary cache pool that shouldn't (and doesn't) even exist on that pool.

 

I have mover logging disabled, but the rsync commands are very spammy. I'm not sure what the /./ directory means in this case?

 

e.g.

Aug 12 12:55:26 Morpheus root: Moving "/mnt/cache_sata/./Media/Music/John Coltrane/A Love Supreme/A Love Supreme.jpg"  to  /mnt/user0/ 
Aug 12 12:55:26 Morpheus root: rsync: [sender] link_stat "/mnt/cache_sata/Media/Music/John Coltrane/A Love Supreme/A Love Supreme.jpg" failed: No such file or directory (2)
Aug 12 12:55:26 Morpheus root: rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1338) [sender=3.3.0]

 

and I have hundred of megs of it generated every move.

[Reynald]

Hi @Faceman

 

1 hour ago, Faceman said:

I have mover logging disabled, but the rsync commands are very spammy. I'm not sure what the /./ directory means in this case?

 

/./ in rsync is for relative path. From rsync man:

Quote

It is also possible to limit the amount of path information that is sent as implied directories for each path you specify.  With a modern rsync on the sending side (beginning with 2.6.7), you can insert a dot and a slash into the source path, like this:

rsync -avR /foo/./bar/baz.c remote:/tmp/

That would create /tmp/bar/baz.c on the remote machine. (Note that the dot must be followed by a slash, so "/foo/." would not be abbreviated.)

It's also how unbalanced plugin works.

 

1 hour ago, Faceman said:

If I enable Rebalance Shares my log gets spammed (to the point of filling the entire log FS) with Rsync throwing errors for files that obviously don't exist.

That's a bug, it's corrected (see below)

Edited August 12 by Reynald

[Reynald]

2024.08.12.0852

Repair/optimize cache priming (R3yn4ld).

Adding check to bc (un)installation routine

2024.08.12.0023

Add bc (un)install option (R3yn4ld)

Force test mode only on major upgrade, keep on minor (R3yn4ld)

Better cache priming (R3yn4ld)