Swarles

Yeah, a long time ago I added some functionality to this plugin and had the intention of probably taking it over once hugenbdd was planning on moving on. Eventually I decided I would rewrite the plugin from the ground up because the code was starting to become janky implementation slapped on top of more janky implementations and it was just getting too messy to even consider adding more advanced functionality. I progressed pretty far but graduated and got a full-time job as a robotics engineer and never touched it again...that was about a year ago. Turns out it's really hard to have spare time / effort and energy when you have a full-time job lol. I've got a lot of respect for the people that seem to have a million side projects and somehow balance them with a good work-life balance. I would like to pick up my development again because I had a lot of fun doing it, but I probably won't publish it because I can't commit to maintaining it and there are so many changes for me to go back and read. It's more of a class / object oriented approach so additional functionality can just be added in quite easily. I need to learn how to write tests for it because doing manual testing is so killer and wastes so much time, but having to write tests is also time consuming and not very motivating.

Is the error still appearing after the changes? The change will disappear after a reboot as well just FYI, can you verify the change is still there? Thank you

Mover tuning is essentially just a filter that sits between the mover being started and the mover actually moving files. So when the mover is called either through a schedule or manual run, if mover tuning is installed instead of just sending every file on the cache to the mover and having them all moved, the files will be filtered based on your settings and then sent to the original mover to move them. Mover tuning won't run outside of the mover being invoked. So if you have the schedule set to weekly, it'll only ever invoke the mover weekly (on Sunday in your case) and that's when it will filter files and move them. If you want the behaviour you mentioned where it moves files filtered on a more frequent schedule (set the main schedule to be more frequent) but then moves all files (ignoring tuning) on the sunday, you can use the option "Force move of all files on a schedule". Test mode works by going through all the steps of the mover without actually sending the files to be moved. If there are no files in a .list during test mode there won't be any in the non test mode. If the mover is not working for you, it is likely another issue, but the first thing to check would be all the steps in the FAQ#2. Based on your current settings, nothing will move if the cache your share is on is not above 70% capacity. Can you confirm it's using more than 70% currently?

Mover tuning settings are "disabled" for any Array->Cache moves. Effectively it just uses the default mover without filter and should move everything. It's disabled on the share level for this same reason, customising the settings wouldn't do anything. Regardless, it should still be moving all files associated with that share back on to the cache. The only thing I can suggest trying is disabling or removing mover tuning in the settings to see if it's the thing causing problems. Alternatively you could set the option "Move now button uses mover tuning settings:" to no and then invoke the mover on the main page. Otherwise I'll need some logs to help see what's happening. See FAQ item #3.

Yeah it's on line 370

Awesome! I haven't taken a look through the code but I'm sure this is a feature that many would find useful. Have you done much testing for it? I probably won't be able to take a look for another week or so. I'm not sure which version of age_mover you have taken this snippet from but the error is on line 370 showing in the latest release of the plugin. Yes this is correct. We don't sort the files at all currently. When the user selects the ctime or mtime option this is purely for the sake of determining the "Age" of the file.

You could write a script that runs before the mover to do this. Puts all the paths except the one you want to move into the skiplist file. Unraid doesn't update access time so unless your next cloud does, then no. Yes you can set most settings on a per share basis. It's AND.

You can specify folders that you want to keep on the cache at all times. It does nothing and behaves like the normal mover. Tuning array -> cache is probably a use case for less than 1% of people. But I do acknowledge some may want it. See above. Will only count on cache -> array moves. It won't ever ignore the "Array -> cache" or "cache -> array" ahre settings. Think of mover tuning as a filter that uses the built in mover. It will apply it's filter on "cache -> array" moves but it won't on the other. If you want everything on the cache until it fills up you can set it to "Array -> cache" and it will move stuff to the cache. If you add more files and they can't fit, they will go onto the Array. If you delete files off the Cache and space becomes available, they will move from the array to the cache. I think thats a bad setup though. Otherwise set it to "Cache -> array" think of some filters you want to use to keep only certain files or folders on the cache. Stuff that you think you will access constantly you can make exclude that folder in the mover tuning settings. The rest for example can just be moved. Maybe you only want the mover to move when you reach 50% capacity in the cache. Maybe you only want files that have been on the cache for 1 month already to be moved. Or both.

I think the "TOTALARRAYSIZE += : syntax error" is the same as the one identified back in Jan here. And the problem was identified here. This commit highlights the change but I don't think I ever shared it with you. Hopefully that solves it.

I think usually the "Some or all files unprotected" with the orange icon is related to not having redundancy. So on the array that could be no parity, or a compromised parity. On the cache that could mean you only have one cache drive or just no redundancy. Spaces can cause lots of issues in bash but for the most part the plugin tries to encompass everything in quotes so that they don't cause issues. It's possible that there is something unaccounted for however. If you get the errors again just drop another comment here and we can try figure it out. It's annoying but there's so many things it could be that it's difficult to know for sure what might be causing it.

Oh okay it looks like the folders it can't find are the following: /mnt/cache_ssd/Tech/pfSense/pfSense Backups/April 30 And /mnt/cache_ssd/Tech/pfSense2/pfSense Backups/April 30 This is a peculiar error as it's happening on the share mover not the main mover, the share mover does minimal processing, it basically just sends the share path to the unraid default mover. It seems like you figured out your problem was related to your HBA. Great work! I have no idea how you deduced that or why it would cause issues though. You can delete your diagnostics. If you want me to investigate or the issue hasn't gone let me know, I'll probably need you to confirm that that exact path does exist and if there is anything peculiar about its permissions or properties.

This may sound obvious but have you checked if the file or directory it is trying to move exists? It could be a number of things wrong. Is there any more info you can provide? See the FAQ on how to provide logs.

Unraid itself hasn't really had any big updates/changes recently that impact how the mover works. So it shouldn't be starting to break, but it does appear to be not working for some.

What are all your settings?

Hey sorry for the delay on this, I would very much like to get you up and running. Unfortunately I will need some logs from Mover Tuning which don't get saved with the diagnostics zip. If you follow the steps here in #1 to get the logs that should allow me to figure out exactly what is going wrong Don't worry about doing 3 runs, just the 1 should be fine.

Share override settings take higher priority than the global mover tuning settings. If you disable mover tuning it will also disable the share override settings. Leave everything default in the global "scheduler settings" and the mover will behave has default. Then you can make changes in the individual shares. If you want a share to not use the mover at all, disable the mover in that shares configuration

The short answer is to schedule the mover to run more frequently. If you have a threshold setup it won't do any moves unless that threshold has been reached.

Depending on how many files you have on your array at the moment you could just set it to move from array->cache run the mover then set it back to cache->array and make it use your settings. I'd you have lots of files that'll either take a long time, waste write cycles, or be impossible. So what you should do then is come up with your own find string to find the files you want to move from the array to the cache, eg. find /mnt/disk*/ShareName -depth -size -50M That will search all of the disks that contain the relevant share and find files that are less than (and equal to I think) 50MB. Use ChatGPT to fine tune it to the exact parameters you want if you're not familiar with the find command. It will print out a list of files and folders it finds. Then if you want to manually move them you can do that, or you can ask ChatGPT how to write your own command with eg rsync to do it, or you can temporarily set the mover for that share to array->cache (DON'T RUN THE MOVER NOW) you will then manually invoke the mover by adding the following to end of your find string. | /usr/local/sbin/move That will pipe all the file paths the find string finds into the built in mover which will then see the share is set to move from the array to the cache and it will do the moves for you Don't forget to change your share mover setting back to what you want.

Yeah so at the moment that's not possible with the raw plugin and I totally understand the desire for it. Infact at the moment, there is no mover tuning settings for array->cache moves at all. I have been working on creating a new plugin from the ground up which will bring all the tuning settings to array->cache moves as well even though it probably won't be used by 90% of people. Unfortunately because the mover only really moves stuff one way depending on how the share is configured, what you desire is not easily achievable by manipulating the built in mover. So there may be circumstances where files on the array that your tuning settings would normally keep on the cache will remain on the array. This would be likely to occur if you set up mover tuning after having already established your server file system, or changing the mover tuning settings considerably. What I will think about though, is creating a button that will extract files on the destination drive that match the source drives mover tuning settings (but the opposite), for the purpose of ensuring all files on that share are following the tuned settings. That way you would only need to run it once after making changes to mover tuning. I think this would address your feature request In saying this, the plugin is still a fair way off and this feature will be lower down on my priority list.

Unfortunately your exact requirements are not currently supported. The primary issue is that in point #3 the threshold will take precedence, so if the threshold is not exceeded, files that are older than X days will not be moved. They will only be moved if the threshold has been exceeded AND they are older than X days. Furthermore, the threshold operates on a pool level not a share level, as a share typically does not have a set maximum capacity. So the threshold takes into account the usage of all shares on that pool. You might be able to implement your desired logic by slightly modifying the before script Terebi posted above. Otherwise I recommend setting the threshold to a value that is suitable for all your shares. And then understand that it will only ever move the shares if that threshold is exceeded and then if it is going to move files it will use all your other settings. I'm remaking mover tuning from the ground up and it will support your desired functionality, but it is quite a ways off at the moment.

Interesting, so are you having issues running Mover Tuning entirely? What version of unraid are you using?

In that case you should be able to echo the list of file paths and pipe them into "/usr/local/sbin/move -d".

Yeah unfortunately the way the individual share mover script is implemented it doesn’t use the mover tuning settings. It honestly wouldn’t be the hardest thing to implement so I might do it, but I’m also working on revamping the plug-in from the ground up so that’s where my limited time is being spent at the moment. For now the easiest thing you can do if you want to use the tuning settings but move only 1 share (or a few), would be to put the path for all the other shares in the skip files list. eg. /mnt/cache1/share1 /mnt/cache1/share2 /mnt/cache2/backups etc This will work exactly as expected, albeit quite inefficient as it will still loop through each share but just find no files. Honestly I’m not really sure how the built in mover handles hard-links, but I can tell you that when the mover tuning plugin detects a hard-link in the list of files that should be moved, it sends the entire list of files as a single group to the mover (sbin/move). Otherwise it sends only 1 file path at a time, I didn’t implement this and so I am not sure if this a requirement or not.

See #3 on the FAQ.

I think it would be possible but It's not something I would want to implement in the current mover tuning code base. I am currently in the process of building the mover tuning plugin from the ground up fully in php, using OOP practices and I'll think about the best way to implement evaluating a threshold on a per dataset basis. I'm not sure if I would ship it in the first release but the idea of migrating to OOP would be to make implementing stuff like this easier. For the time being it will remain a whole pool basis unless someone else wants to implement it. In light of that which one do you think would be better to use going forward? zpool get -o value capacity OR df --output=pcent That's weird, can you confirm if that file appears if you perform the following: find '/mnt/cache/movies' -depth -not -path '*/\.*' If not, you might need to confirm what find string is being used in the logs. I couldn't find any code relating to destroying datasets which leads me to believe this is a feature baked into the built in mover. I could possibly add a check to not send datasets to the mover when that setting is enabled. This will definitely be an option when I finish building the new mover tuning plugin.