10 minutes ago, JohnnyGrey said:

I'm trying to invoke the mover with the "Move Now" button after enabling the option to use the plugin version with said button, and am encountering an error message about "/usr/local/sbin/move" not existing, then failing to create the symbolic link because it already exists. Any ideas?

 

Dec 29 15:11:17 Denton emhttpd: shcmd (6066204): /usr/local/sbin/mover start |& logger -t move &
Dec 29 15:11:17 Denton root: Starting Mover
Dec 29 15:11:17 Denton root: Forcing turbo write on
Dec 29 15:11:17 Denton kernel: mdcmd (198): set md_write_method 1
Dec 29 15:11:17 Denton kernel: 
Dec 29 15:11:17 Denton root: ionice -c 2 -n 0 nice -n 0 /usr/local/emhttp/plugins/ca.mover.tuning/age_mover start 3 0 0 '' '' '' '' no '' '' '' 0
Dec 29 15:11:17 Denton move: Log Level: 1
Dec 29 15:11:17 Denton move: /usr/local/sbin/move does not exists, creating soft link
Dec 29 15:11:17 Denton move: ln: failed to create symbolic link '/usr/local/sbin/move': File exists
Dec 29 15:11:17 Denton move: mover: started
Dec 29 15:11:18 Denton move: /usr/local/emhttp/plugins/ca.mover.tuning/age_mover: line 250: /usr/local/sbin/move: No such file or directory

 

The last line repeats many times, presumably for each of the files that match the age requirement.

 

I tried to unlink the /usr/local/sbin/move file and then run the mover again. It successfully creates it, but it seems the link is immediately orphaned. Is this symlink creating incorrectly?

 

 

EDIT:
 

I think I found the error. age_mover seems to want to link /usr/local/bin/move to /usr/local/sbin/move, however the first file does not exist. The only files in /usr/local/bin/ are mover and smart locate. The third item is a directory.

 

Should this instead be mover instead of move?

Edited December 29, 20241 yr by JohnnyGrey

Disregard all my posts. I read back a few pages; I too was using the older version of this plugin. I have since uninstalled Squid's version and installed Reynald's version.

I don't know why, I have 5tb free
 

Dec 31 05:00:31 unRAID move: Moving "/mnt/cache/./plex/Serie TV/Paris Has Fallen/S01E07 - 4K.mkv"  to  /mnt/user0/ 
Dec 31 05:00:31 unRAID shfs: assign_disk: /plex/Serie TV/Paris Has Fallen/.S01E07 - 4K.mkv.nbknyx (28) No space left on device
Dec 31 05:00:31 unRAID move: rsync: [receiver] mkstemp "/mnt/user0/plex/Serie TV/Paris Has Fallen/.S01E07 - 4K.mkv.nbknyx" failed: No space left on device (28)
Dec 31 05:00:35 unRAID move: rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1338) [sender=3.3.0]
Dec 31 05:00:35 unRAID move: 6291 files remaining from caches to array  273GiB
Dec 31 05:00:35 unRAID move: Moving "/mnt/cache/./plex/Serie TV/Paris Has Fallen/S01E08 - 1080p.mkv"  to  /mnt/user0/ 
Dec 31 05:00:35 unRAID shfs: assign_disk: /plex/Serie TV/Paris Has Fallen/.S01E08 - 1080p.mkv.RuQIWm (28) No space left on device
Dec 31 05:00:35 unRAID move: rsync: [receiver] mkstemp "/mnt/user0/plex/Serie TV/Paris Has Fallen/.S01E08 - 1080p.mkv.RuQIWm" failed: No space left on device (28)
Dec 31 05:00:37 unRAID move: rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1338) [sender=3.3.0]
Dec 31 05:00:37 unRAID move: 6290 files remaining from caches to array  271GiB
Dec 31 05:00:37 unRAID move: Moving "/mnt/cache/./plex/Serie TV/Paris Has Fallen/S01E07 - 1080p.mkv"  to  /mnt/user0/ 
Dec 31 05:00:37 unRAID shfs: assign_disk: /plex/Serie TV/Paris Has Fallen/.S01E07 - 1080p.mkv.mTms0m (28) No space left on device
Dec 31 05:00:37 unRAID move: rsync: [receiver] mkstemp "/mnt/user0/plex/Serie TV/Paris Has Fallen/.S01E07 - 1080p.mkv.mTms0m" failed: No space left on device (28)
Dec 31 05:00:38 unRAID move: rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1338) [sender=3.3.0]
Dec 31 05:00:38 unRAID move: 6289 files remaining from caches to array  269GiB

unraid-diagnostics-20250102-0152.zip

Hello,

After moving to Unraid 7 RC2, I switch to the new version of the mover plugin, and I've got an issue with ignoring files.

 

As an exemple, in my ignore list file, I've got the following entry : 

/mnt/ssd_pool/cameras/recordings/2025-01-03/04/garage/00.55.mp4

 

But the mover still moves the file : 

Moving "/mnt/ssd_pool/./cameras/recordings/2025-01-03/04/garage/00.55.mp4"  to  /mnt/user0/

 

Any idea on how to fix this?

 

Thanks

9 minutes ago, foux said:

Any idea on how to fix this?

 

what does the logs may say

 

sample (im just ignoring folders)

Jan  4 05:59:29 AlsServerII move: List Path: /mnt/user/system/mover/mover_tuning_excl.txt:
Jan  4 05:59:29 AlsServerII move:     Ignored files are using 1.2TiB

 

may also take a look at the temp ignoring list which also is mentioned in the logs, unraid terminal, sample

 

cat /tmp/ca.mover.tuning/Filtered_files_2025-01-04T055929.list

 

check your logs for your filename

Thanks for your answer @alturismo

 

No issues on the logs : 

move: List Path: /mnt/user/tools/mover-ignore.txt:
move:     Ignored files are using 1.8GiB

 

And the fils being moved are in the ignored list :

➜ grep "2025-01-03/04/garage/00.55.mp4" /tmp/ca.mover.tuning/Filtered_files_2025-01-04T100234.list
ssd_pool|ssd_pool|cameras|skipped|9999999999|561233107353|275456|0|0|0|/mnt/ssd_pool/cameras/recordings/2025-01-03/04/garage/00.55.mp4
ssd_pool|user0|cameras|yes|1735876868|561233107353|209427|1|1|62146|/mnt/ssd_pool/cameras/recordings/2025-01-03/04/garage/00.55.mp4

 

I guess the issue is that when moving, it adds a supplementary ./ in the path name (/mnt/ssd_pool/cameras/recordings/2025-01-03/04/garage/00.55.mp4 becomes /mnt/ssd_pool/./cameras/recordings/2025-01-03/04/garage/00.55.mp4) but that shouldn't change anything, as it is the same file.

 

Thanks

Edited January 4, 20251 yr by foux

23 minutes ago, foux said:

No issues on the logs : 

ok, then we have to wait for @Reynald to recover from his moving

Even adding the directory (/mnt/ssd_pool/./cameras) to the ignore list doesn't work

@alturismo in the exclude list, did you put the path under /mnt/user or the path under your cache pool?

 

Thanks

1 hour ago, foux said:

did you put the path under /mnt/user or the path under your cache pool?

cache

 

but, was just a test as I'm using my own mover actually 

OK, I've tried everything I can think of : 

 

/mnt/cache_pool/./share, /mnt/cache_pool/share, /mnt/user/share for folders ignore

/mnt/cache_pool/./share/file.txt, /mnt/cache_pool/share/file.txt, /mnt/user/share/file.txt for files ignore

 

None of them are working, so I'm kind of out of ideas now

Just to add my findings as ignore list was working super inconsistently for me. Basically I would see some stuff would be ignored while others would not and there was no rhyme or reason why, as it could be any folder or file. Plus it would vary from 1 day to the next, where a file was not ignored 1 day, would be ignored the next day.

 

The TLDR is the current latest version is bugged for ignore lists if you have a lot of entries in your list. Apart from lowering the amount of entries to ignore, I found no fix bar rewriting the script.

 

The longer version, digging through each Filtered_files_....list and Mover_action_....list, I noticed double entries for ignored files and folders. Essentially 1 saying it will be skipped and another that it will be moved. Going through the age_mover script and adding debugging commands, I got the final find command, it's loooong!

 

The script will add a switch " -not -path \"${skipped_path}\"" or " -not -name \"${skipped_path}\"" for every entry on the ignore list. I have thousands of entries in my list, so no surprise the resultant find command is super chunky. My use case for having a large ignore list is, ctime is super inconsistent on my system, so I wrote a python script that will determine age to move and writes files and folders to the ignore list and then let mover tuning do it's thing.

 

Copying and pasting the full find command into my unraid console displayed the same behaviour of some results being filtered while others not. Turns out find, bash or something else, has, I guess a character limit or maybe just limits amount of switches, that it will handle. When it hits the limit, it truncates and ignores the rest of the find command, so for entries that should be ignored, fall into the truncated section, they are returned in results for what to move.

 

If I limited the number of entries on my ignore list to less than a hundred entries it seems to work fine. I didn't bother trying to work out the exact cut off point though. The code change happened in the 2024-08-15 version and seems to affect all versions after. The previous releases didn't work for me either as they seemed to ignore folders from my ignore list (plus suffered from other bugs).

 

My only fix, was to go ahead and rewrite a large part of the code to use a different method to apply filters for ignored entries. The method I found to work was running a broad find search with all other switches minus the "-not path" and "-not name" elements, write results to a tmp file, read the tmp file and remove entries that match from the ignore list and feed the results to $FILTERFILELIST. Bar some buggy reporting (size left is always wrong), mover now works properly for me and will always ignore files from my list.

 

Note I don't now if this is the best approach but every other thing I tried had some sort of issue. For instance, I only went with writing to tmp files to allow me to debug, when it was working, I tried just writing results to variables in memory, but that ended up being super buggy and inconsistent and at that point I couldn't be motivated to troubleshoot it further so left my version using tmp files.

 

This might also explain why some people have issues with smart caching and etc, as they may be truncated / malformed, if used in combination with a large ignore list. Hopefully this might help someone if a newer version is developed.

 

 

Thanks @Silver226, this is most definitely my issue, as I've got a huge ignore list.

1 hour ago, Silver226 said:

If I limited the number of entries on my ignore list to less than a hundred entries it seems to work fine. I didn't bother trying to work out the exact cut off point though. The code change happened in the 2024-08-15 version and seems to affect all versions after. The previous releases didn't work for me either as they seemed to ignore folders from my ignore list (plus suffered from other bugs).

Confirmed working by limiting the file list. Thanks @Silver226, hope we can get a fix.

59 minutes ago, foux said:

Confirmed working by limiting the file list. Thanks @Silver226, hope we can get a fix.

No problem @foux, at least you know what the cause is now. It'd been driving me crazy for months before I figured out what was going on. Fingers crossed we'll see a proper fix soon!

On 12/24/2024 at 9:55 AM, itimpi said:

Did you make sure that the docker service was disabled while doing this?   Mover will ignore open files so you need to make sure there is nothing that can keep open files you want to be moved.

No I didn't. Thanks very much. If I stop the container the Mover completes moving everything. Now to work out schedule stopping of the docker containers.

 

Thanks again

On 1/4/2025 at 11:13 AM, alturismo said:

ok, then we have to wait for @Reynald to recover from his moving

Hi all,

I'm back  
After my move, I had to rethink my system because I've broke the motherboard (AIO leak) and for some reason, I've been on Truenas scale for a couple of months just to learn and see to see how it works with ZFS (well.. better IMHO, unraid lacks backup and replication integrated tools, but yeah, we can integrate some scripts or plugins blabla bla).
7.0.0 beta 3 changelog made me come back (the tailscale integration)

 

Sorry if I were away for too long.

 

I'm going to catch my backlog and restart maintenance and updates of this plugin next week 

Edited January 5, 20251 yr by Reynald

welcome back.

 

ZFS on unraid has been improving quickly and more of those functions will end up in the UI.

Advanced ZFS users have been running with the separate ZFS plugins or doing that in the terminal for years now, and most of those plugins still work perfectly for people that need them, but we know it will end up part of Unraid stock soon enough.

 

I always found Truenas to be clunky and convoluted compared to UnRaid, I have seen it slowly getting better and I do run it for testing and experimentation on my proxmox box but since I use Unraids main feature of mixed disks with easy expansion for my bulk storage pool I'd never use it as my main OS. even with ZFS Expansion now mostly working, expanding large arrays with multiple VDEVs is still a pain and more expensive to grow than unraid.

 

I think having a medium size ZFS pool sitting on top of the slow but cheap to scale unraid array is the optimal NAS solution for the vast majority of users, and the Mover Tuning plugin is what makes that work...

 

I'm looking forward to the file exclude feature working fully again, it seems broken for some people at the moment and we need that to work to get @alturismo's new media caching plugin working with mover tuning seamlessly as it would need to write to an exclude file or have some other dedicated allocation method to not interfere or have them fight each other.

Edited January 6, 20251 yr by Faceman

Is there a way/mechanism to explicitly include folders to be moved and ignore others for X days? For context I have both a Scrypted and Lancache container which generate lots of small files to their shares and I'd rather move them on a more regular basis and let anything else just pile up more on the drive.

 

Is it more logical to figure out a better way of moving these files with a user script? And if so, does anyone have any prior art of how to do that without ruining the array 😅

Hey there,

 

So it's been a few months since I posted about this and I still can't figure out a good solution: files from my torrent container get skipped, even after I stop seeding and remove them from the torrent app. The mover log shows "expected 2 files, got 1...(*lists file path*) won't be moved to prevent breaking hardlinks." How do I work around this? The only setting I user in mover tuning plugin for is to ignore hidden files and folders and schedule moves. Thank you for the assistance.

I've upgraded to official Unraid 7, learned i had the old version of the plugin, upgraded to this new version.

mover moves file from cache to array just fine but leaves the empty folders on the cache drive. I see the setting to 'clean empty folders' and have tried it set to yes and no with same result.  Am i missing a setting somewhere or it this a known issue?

This version hasn't had a bugfix for a few months, Dev is back and will be working on it again soon.

 

the empty folders aren't going to cause any issues, and they should be able to be cleaned up once the script is fixed.

Hi,

I’m using Unraid 7.0 and my setup doesn’t include an array—just two pools. Everything works perfectly without the Tuner plugin, but as soon as I try to use it, I get an error that says:
Fatal error: /boot/config/share.cfg does not exist, check this.

Has anyone encountered this?

 

I'm having issues in general with the mover.  I didn't have Reynald's installed prior to unraid 7, so I'm not sure if this is a setup issue or a bug with the mover tuning plugin and v7.

 

It seems to be ignoring the share's mover action.  This particular share is setup as such:

 

 

however, the mover seems to be trying to move it from my cache drive to the array:

Jan 12 03:51:27 Dammerung move: Moving "/mnt/cache/./data/workspace/scripts/ffmpeg-2024-11-18-git-970d57988d-full_build/LICENSE" to /mnt/user0/ Jan 12 03:51:28 Dammerung move: 1813 files remaining from caches to array 569GiB

 

In fact, it tried moving my appdata to /mnt/user0 as well until I dropped the array from the secondary storage.  

On 1/4/2025 at 6:17 AM, Silver226 said:

Just to add my findings as ignore list was working super inconsistently for me. Basically I would see some stuff would be ignored while others would not and there was no rhyme or reason why, as it could be any folder or file. Plus it would vary from 1 day to the next, where a file was not ignored 1 day, would be ignored the next day.

 

The TLDR is the current latest version is bugged for ignore lists if you have a lot of entries in your list. Apart from lowering the amount of entries to ignore, I found no fix bar rewriting the script.

 

Excellent find, and seems consistent with what I've been experiencing. Thanks for documenting it, and hopefully we can get a fix in place soon now that Reynald has returned!