5 hours ago, Triiiple said:

I'm running the mover but nothing is moving even though conditions are met and after looking at the attached logs they are empty? The log gets generated after running but doesn't show anything inside.

What next steps can I take to troubleshoot?

 

Thanks!

Debug_Mover_tuning_2025-04-09T163520.log 990 B · 1 download Summary_2025-04-09T163520.txt 115 B · 1 download log.txt 32.95 kB · 2 downloads

Or what other logs can I provide to help give insight?

Have you got a screenshot of your plugin settings, first thing to check and easy to miss a setting and not realise it.

 

 

 

1 minute ago, Faceman said:

Have you got a screenshot of your plugin settings, first thing to check and easy to miss a setting and not realise it.

 

 

I want files older than 15 days when the cache is over 60% to be moved to the array.

12 minutes ago, Triiiple said:

I want files older than 15 days when the cache is over 60% to be moved to the array.

settings looking fine, question left

 

are you sure there are files > 15 days old in /mnt/cache/data

23 minutes ago, alturismo said:

settings looking fine, question left

 

are you sure there are files > 15 days old in /mnt/cache/data

Yes

24 minutes ago, Triiiple said:

Yes

may stat such a file and check results, sample

 

root@AlsServerII:~# stat /mnt/cache/Media/RECORD/.nomedia
  File: /mnt/cache/Media/RECORD/.nomedia
  Size: 0               Blocks: 0          IO Block: 4096   regular empty file
Device: 259,1   Inode: 3226138080  Links: 1
Access: (0777/-rwxrwxrwx)  Uid: (   99/  nobody)   Gid: (  100/   users)
Access: 2024-05-11 19:26:09.949859814 +0200
Modify: 2023-11-16 14:44:37.783891200 +0100
Change: 2024-05-11 19:26:09.950859821 +0200
 Birth: 2024-05-11 19:26:09.949859814 +0200

 

I find sometimes the CTIME gets changed by something or doesn't make sense upon creation sometimes.

9 hours ago, Triiiple said:

What next steps can I take to troubleshoot?

What is shown in list files? 

7 hours ago, Faceman said:

I find sometimes the CTIME gets changed by something or doesn't make sense upon creation sometimes.

I had already checked the CTIME for a file in question, here's an example of one that should have moved:

 

root@Unraid:~# stat "/mnt/user/data/torrents/tv-sonarr/SpongeBob SquarePants (1999) S15E19 Delivery of DOOM (1080p AMZN Webrip x26
5 10bit EAC3 2.0 - Frys) [TAoE].mkv"
  File: /mnt/user/data/torrents/tv-sonarr/SpongeBob SquarePants (1999) S15E19 Delivery of DOOM (1080p AMZN Webrip x265 10bit EAC3 2.0 - Frys) [TAoE].mkv
  Size: 283454264       Blocks: 554130     IO Block: 131072 regular file
Device: 0,104   Inode: 14355223812244000  Links: 3
Access: (0664/-rw-rw-r--)  Uid: (   99/  nobody)   Gid: (  100/   users)
Access: 2025-03-25 12:11:14.852126157 -0500
Modify: 2025-03-25 12:11:31.027970468 -0500
Change: 2025-03-25 12:18:13.518115725 -0500
 Birth: -

 

6 hours ago, Masterwishx said:

What is shown in list files? 

Filtered_files_2025-04-09T163520.list

Here is the file list, this one is not empty. 

Please let me know what else I can provide for more insight

 

Thanks!

Edited April 10, 20251 yr by Triiiple

Here are all the logs I could gather from a new run, changed move files greater than to = 7 days 

Summary_2025-04-10T093216.txtMover_tuning_2025-04-10T093216.logMover_action_2025-04-10T093216.txtFiltered_files_2025-04-10T093216.txtDebug_Mover_tuning_2025-04-10T093216.log

 

 

Edited April 10, 20251 yr by Triiiple

On 4/9/2025 at 1:29 AM, ronia said:

Really?  I don't think that's true.  This is what happens when a file is hardlinked:

 

Apr  7 05:00:08 Dammerung move: *********************************************************** LET THE MOVING SHOW BEGIN ! *********************************************************
Apr  7 05:00:08 Dammerung move: Expected 3 files, got 1. Not moving "/mnt/cache/./data/media/tv2/Silo/Season 02/Silo (2023)-s02e03-NTb-WEBDL-1080p.mkv"  to prevent breaking hardlinks
Apr  7 05:00:08 Dammerung move: Expected 3 files, got 1. Not moving "/mnt/cache/./data/media/tv2/Silo/Season 02/Silo (2023)-s02e01-NTb-WEBDL-1080p.mkv"  to prevent breaking hardlinks
Apr  7 05:00:08 Dammerung move: Expected 3 files, got 1. Not moving "/mnt/cache/./data/media/tv2/Silo/Season 02/Silo (2023)-s02e06-NTb-WEBDL-1080p.mkv"  to prevent breaking hardlinks
Apr  7 05:00:08 Dammerung move: Expected 3 files, got 1. Not moving "/mnt/cache/./data/media/tv2/Silo/Season 02/Silo (2023)-s02e02-NTb-WEBDL-1080p.mkv"  to prevent breaking hardlinks
Apr  7 05:00:08 Dammerung move: Expected 3 files, got 1. Not moving "/mnt/cache/./data/media/tv2/Silo/Season 02/Silo (2023)-s02e04-NTb-WEBDL-1080p.mkv"  to prevent breaking hardlinks
Apr  7 05:00:08 Dammerung move: Expected 3 files, got 1. Not moving "/mnt/cache/./data/media/tv2/Silo/Season 02/Silo (2023)-s02e05-NTb-WEBDL-1080p.mkv"  to prevent breaking hardlinks
Apr  7 05:00:08 Dammerung move: Expected 3 files, got 1. Not moving "/mnt/cache/./data/media/tv2/Silo/Season 02/Silo (2023)-s02e08-NTb-WEBDL-1080p.mkv"  to prevent breaking hardlinks
Apr  7 05:00:08 Dammerung move: Expected 3 files, got 1. Not moving "/mnt/cache/./data/media/tv2/Silo/Season 02/Silo (2023)-s02e07-NTb-WEBDL-1080p.mkv"  to prevent breaking hardlinks
Apr  7 05:00:08 Dammerung move: Expected 3 files, got 1. Not moving "/mnt/cache/./data/media/tv2/Silo/Season 02/Silo (2023)-s02e10-NTb-WEBDL-1080p.mkv"  to prevent breaking hardlinks
Apr  7 05:00:08 Dammerung move: Expected 3 files, got 1. Not moving "/mnt/cache/./data/media/tv2/Silo/Season 02/Silo (2023)-s02e09-NTb-WEBDL-1080p.mkv"  to prevent breaking hardlinks
Apr  7 05:00:08 Dammerung move: ------------------------------------------------------------------- Cleaning up -----------------------------------------------------------------

 

The official mover is apparently able to handle the hardlink and move both the hardlinks and the file to the array.  

 

Again, I don't know if this is true as I don't have the original mover setup and I'm not in a position to do so.  Just something I had heard that I thought was interesting.

I'm experiencing the same thing. My setup relies on these links breaking, as the official mover seems to be able to do.

 

Would it be possible to make this a toggle in the UI? I think the plugin used to break hardlinks up until a little while ago (not sure exactly why it was changed/patched out). I feel like this would be best as an option depending on different setups, and not seen as a bug or issue.

 

Thank you for the work to maintain and update the plugin!

3 hours ago, jv6478 said:

I'm experiencing the same thing. My setup relies on these links breaking, as the official mover seems to be able to do.

 

Would it be possible to make this a toggle in the UI? I think the plugin used to break hardlinks up until a little while ago (not sure exactly why it was changed/patched out). I feel like this would be best as an option depending on different setups, and not seen as a bug or issue.

 

Thank you for the work to maintain and update the plugin!

Actually, what I was expecting to happen was for the hardlinks to be preserved.  If it's working correctly, it SHOULD move all hardlinks and not break them.  I was supposed to test this a few days ago, but I upgraded my cache drive and the mover probably won't be engaged for a couple weeks now.

44 minutes ago, ronia said:

Actually, what I was expecting to happen was for the hardlinks to be preserved.  If it's working correctly, it SHOULD move all hardlinks and not break them.  I was supposed to test this a few days ago, but I upgraded my cache drive and the mover probably won't be engaged for a couple weeks now.

 

Do you mean that it would preserve the original file that was hardlinked before the move? If so, then that's what I meant as well. By breaking the hardlink I was referring to allowing the mover to move the file despite the fact that it has a hardlink. I may have been off with the term though, sorry if so!

 

I'm getting the same messages in my logs as you are in a hardlink scenario, and it sounds like you want it moved but for it to preserve the other links that would be left on cache. I want the same.

2 hours ago, jv6478 said:

 

Do you mean that it would preserve the original file that was hardlinked before the move? If so, then that's what I meant as well. By breaking the hardlink I was referring to allowing the mover to move the file despite the fact that it has a hardlink. I may have been off with the term though, sorry if so!

 

I'm getting the same messages in my logs as you are in a hardlink scenario, and it sounds like you want it moved but for it to preserve the other links that would be left on cache. I want the same.

No I mean suppose you have a file:
/mnt/cache/data/media/myfile.mkv

 

and then you have a hardlinks at:

/mnt/cache/data/torrents/myfile_hardlink.mkv

 

 

The mover SHOULD be able to move both files within the same command such that:

 

/mnt/disk1/data/media/myfile.mkv

/mnt/disk1/data/torrents/myfile_hardlink.mkv

 

this is what the rsync command should be able to do if you --hard-link is specified (which is what the mover plugin uses).

10 hours ago, ronia said:

No I mean suppose you have a file:
/mnt/cache/data/media/myfile.mkv

 

and then you have a hardlinks at:

/mnt/cache/data/torrents/myfile_hardlink.mkv

 

 

The mover SHOULD be able to move both files within the same command such that:

 

/mnt/disk1/data/media/myfile.mkv

/mnt/disk1/data/torrents/myfile_hardlink.mkv

 

this is what the rsync command should be able to do if you --hard-link is specified (which is what the mover plugin uses).

 

Ah okay, I understand. Yes, it should be moving both as you mentioned in that example for sure, as long as the destination is the same file system. It should essentially move one file (as far as amount of data it's moving) and then recreate the links in the same structure on the new drive. This could be a different (but related) problem.

 

In my case, if data is the share, and torrents is the downloads folder, I would have it as /mnt/cache/data/.torrents (hidden) as I don't want torrents to be its own share. I've also selected the option to ignore hidden files and folders. It seems to work as it doesn't descend into those.

 

However, if an *arr app hardlinks from /.torrents to /data/tv/* (which it regularly does) and mover tries to move that file, I'm getting the same error messages saying that "x files expected" ending with not moving the file to prevent breaking hardlinks.

 

If --hard-link is used in the plugin, rsync will want (or need?) to move both files. Whether this causes a skip, or it skips because it simply doesn't "see" the hardlink in the /.torrents folder, I'm not sure. But this creates a "hole" in the plugin options, while trying to force one specific action (move hardlinks all the time) instead of having an option (break hardlinks if I'm okay with it).

 

This situation illustrates the need for it to be an option. It would also have to somehow play nicely with the ignore hidden folders option, as if a hardlink is there, there has to be a way to decide what to do. Perhaps a nested setting; if ignore hidden files and folders is selected AND move all hardlinks is selected, a hardlink found in a hidden folder WILL or WILL NOT be moved (another toggle). The logic around this is just a concept, but it would have to be something along these lines I suppose.

 

Some people will want all hardlinks to be moved, and some people will want them to be broken. I don't think either is right or wrong, but the plugin trying to force all hardlinks to be moved doesn't feel like a solution, rather a choice, that can hopefully made to be our choice somehow.

 

For now, I've customized the mover script to do what I need - i.e. break hardlinks and move the files - and it's working very well although it still calls on the internal move command. I prefer rsync for various reasons, and really hope I can make use of this awesome plugin again sometime soon.

2 hours ago, jv6478 said:

 

Ah okay, I understand. Yes, it should be moving both as you mentioned in that example for sure, as long as the destination is the same file system. It should essentially move one file (as far as amount of data it's moving) and then recreate the links in the same structure on the new drive. This could be a different (but related) problem.

 

In my case, if data is the share, and torrents is the downloads folder, I would have it as /mnt/cache/data/.torrents (hidden) as I don't want torrents to be its own share. I've also selected the option to ignore hidden files and folders. It seems to work as it doesn't descend into those.

 

However, if an *arr app hardlinks from /.torrents to /data/tv/* (which it regularly does) and mover tries to move that file, I'm getting the same error messages saying that "x files expected" ending with not moving the file to prevent breaking hardlinks.

 

If --hard-link is used in the plugin, rsync will want (or need?) to move both files. Whether this causes a skip, or it skips because it simply doesn't "see" the hardlink in the /.torrents folder, I'm not sure. But this creates a "hole" in the plugin options, while trying to force one specific action (move hardlinks all the time) instead of having an option (break hardlinks if I'm okay with it).

 

This situation illustrates the need for it to be an option. It would also have to somehow play nicely with the ignore hidden folders option, as if a hardlink is there, there has to be a way to decide what to do. Perhaps a nested setting; if ignore hidden files and folders is selected AND move all hardlinks is selected, a hardlink found in a hidden folder WILL or WILL NOT be moved (another toggle). The logic around this is just a concept, but it would have to be something along these lines I suppose.

 

Some people will want all hardlinks to be moved, and some people will want them to be broken. I don't think either is right or wrong, but the plugin trying to force all hardlinks to be moved doesn't feel like a solution, rather a choice, that can hopefully made to be our choice somehow.

 

For now, I've customized the mover script to do what I need - i.e. break hardlinks and move the files - and it's working very well although it still calls on the internal move command. I prefer rsync for various reasons, and really hope I can make use of this awesome plugin again sometime soon.

I am still a little confused as to why you wouldn't want the links to move to the the array; however, I think the modifications should be fairly easy if you want to give it a shot:

 

age_mover: processTheMoves() ln1400:

            # Check if list is complete
            NBFILES=$(($(echo "$SYNCED_FILES" | grep -o '" "' | wc -l) + 1))
            if [ $NBFILES != $NBLINKS ]; then
                mvlogger "Expected $NBLINKS files, got $NBFILES. Not moving $SYNCED_FILES to prevent breaking hardlinks"
                continue
            fi

 

I believe you can just short-circuit this entire chunk of script.  I'm pretty sure it's just a check to see if the transfer list contains all of the hard-links.  This is the line that actually causes the file to be skipped, not rsync itself.

 

age_mover: processTheMoves() ln1375:

            #Prepare RSYNC_CMD
            RSYNC_CMD="rsync --archive --xattrs --relative --hard-links"

 

I'm out to lunch on whether --hard-links actually needs to be removed from this line (the rest should remain the same).  The documentation indicates that if the transfer list is incomplete, then it will break those hardlinks.  It isn't clear to me what it means to "break the hardlinks".  To me this means that either the hardlinks will become separate files or that the actual links will become broken and lead to nothing.  The former puzzles me as I am pretty sure that default rsync functionality does exactly this anyways.  The latter scares me and isn't something I would want to try.  

 

Since this is exactly the scenario I would want to avoid, I can't really test this as it would mess up my library, but if you like you could give it a shot.  

Edited April 11, 20251 yr by ronia

On 4/10/2025 at 4:49 PM, Triiiple said:

Here are all the logs I could gather from a new run, changed move files greater than to = 7 days 

 

As you can see you have `keep on primary` for all files and from one file timestamp : 

 

So if you changed to 7days it should work, if they still not moving, try to change Freeing threshold to 40%

5 minutes ago, Masterwishx said:

 

As you can see you have `keep on primary` for all files and from one file timestamp : 

 

So if you changed to 7days it should work, if they still not moving, try to change Freeing threshold to 40%

I did change it to 7 days and still nothing, the logs I posted above are for 7 days. 

Just now, Triiiple said:

the logs I posted above are for 7 days. 

 

try to change Freeing threshold to 40%

2 hours ago, jv6478 said:

This situation illustrates the need for it to be an option.

 

What exactly you want ? to be option to skip this check ? 

 

33 minutes ago, ronia said:

am still a little confused as to why you wouldn't want the links to move to the the array; however, I think the modifications should be fairly easy if you want to give it a shot:

 

age_mover: processTheMoves() ln1400:

            # Check if list is complete
            NBFILES=$(($(echo "$SYNCED_FILES" | grep -o '" "' | wc -l) + 1))
            if [ $NBFILES != $NBLINKS ]; then
                mvlogger "Expected $NBLINKS files, got $NBFILES. Not moving $SYNCED_FILES to prevent breaking hardlinks"
                continue
            fi

 

I believe you can just short-circuit this entire chunk of script.  I'm pretty sure it's just a check to see if the transfer list contains all of the hard-links.  This is the line that actually causes the file to be skipped, not rsync itself.

 

is it good idea at all, to be option to skip the check ?

19 minutes ago, Masterwishx said:

 

try to change Freeing threshold to 40%

I changed it to 30% and it's moving now, but says only 29 GB. Should be moving a lot more to get down to 30%
 

17 hours ago, Triiiple said:

Should be moving a lot more to get down to 30%

 

If you have big files it will not move them. 

It's checking pool size + file size >= freeing theeshold 

Edited April 12, 20251 yr by Masterwishx

1 hour ago, Masterwishx said:

is it good idea at all, to be option to skip the check ?

That check would be necessary if hardlinks are to be moved/preserved together, which would then work in conjunction with --hard-links so only one copy of the file is moved and the links are preserved as they were (they're rebuilt essentially, as @ronia wants basically). Hypothetically, if there were an option in the UI to always move all hardlinks, this check and the --hard-links would have to remain in place. Without --hard-links, rsync would create n copies of the file (where n = number of hardlinks), one for each location of the original hardlinks, which is not ideal and takes up much more space on the destination drive.

 

That said, --hard-links would not prevent the breaking of a hardlink to be moved if it's changing filesystems and has no choice anyway. The key thing is to not build the list in such a way that rsync ignores hardlinked files if someone chooses this (i.e. an option).

 

Further to this...

 

2 hours ago, ronia said:

I am still a little confused as to why you wouldn't want the links to move to the the array; however, I think the modifications should be fairly easy if you want to give it a shot:

 

My setup is as follows (I think many people have similar setups, though it can also just be preference apart from this):

 

1. download directly to /mnt/cache/media/.downloads (media = share)

2. *arr will hardlink completed downloads to /media/* on cache - many downloads are watched within a few days, so it's nice for those to be on cache before it's archived onto the array

3. Mover will move /media from cache to array, but not touch anything in /.downloads (I have set the option to ignore hidden files/folders) so seeding can continue and finish off of cache, as I do not want anything seeding off the array << at this point, it would be detected that this is a hardlink -- move it anyway -- this would break the hardlink, of course, because it would no longer be in /mnt/cache/media/* as it would make a copy of the file on /mnt/user/media/* (array) (or /mnt/user0/media/* I suppose), and then delete the hardlink from /mnt/cache/media/*

4. The file, now the only copy left on cache (in /.downloads), would continue seeding until finished, at which point the download client will remove the original file, leaving no file (or hardlinks) on cache

 

As you can see, in this case, the hardlink must "be broken" for my setup to work, unless I set a delay to always make sure that only files over a certain age (longer than the seeding requirement) be moved. The space on cache would not be reclaimed right away in this case, but it allows the file to be archived when I want, and avoids stale media files staying on cache after the seeding is complete before the next mover runs.

 

By default, rsync will break hardlinks, unless told otherwise. This is the part that would be nice to be user-configurable.

 

Hopefully that didn't complicate things further LOL!

4 hours ago, Masterwishx said:

is it good idea at all, to be option to skip the check ?

I am not sure, but I am a little more sure after hearing @jv6478 reasons for why he wants to break the hardlinks.  Basically, I think he doesn't want the array to spinup due to torrenting.  As highlighted below:

 

1 hour ago, jv6478 said:

4. The file, now the only copy left on cache (in /.downloads), would continue seeding until finished, at which point the download client will remove the original file, leaving no file (or hardlinks) on cache

 

So basically the torrent will continue to seed off of the cache drive until presumably the user hits the tracker's minimum seed requirements and then the cache copy will also be deleted.  So there is an instance in time in which both an array copy and a cache copy will exist.  The cache copy eventually gets deleted on its own, presumably through qbit_manage or manual process.

 

I can see where this would be useful for some people.  I didn't think of this use case as I perma-seed my content for bon.  I'm not a racer so this didn't even occur to me.

Edited April 11, 20251 yr by ronia