slimjim181 Posted May 25, 2021 Share Posted May 25, 2021 Context I build custom NASs for artists such as video editors and this feature is important to me as I want a solution built into their workflow where their working projects will have a '.' (dot) at their root directory so that they are not moved to the array. But when they are finished with a project they can remove the dot. Then overnight (assuming the mover is running daily) that project folder will be moved to the array 'archiving' that project. BUG User case As a video editor When I create a folder .shouldNotMove at the top level of my share myshare And this share has its cache settings configured to 'Yes' (files initially on cache then moved to array) And I click the 'move now' button under Settings -> Schedule Expected .shouldNotMove directory remains in /mnt/cache/myshare and the files within .shouldNotMove also remain Actual .shouldNotMove is moved to the array and deleted from /mnt/cache/myshare Developer details Mover script found under `/usr/local/sbin/mover` has a line code comment: # The script is set up so that hidden directories (i.e., directory names beginning with a '.' # character) at the topmost level of a pool or an array disk are not moved. This behavior # can be turned off by uncommenting the following line: # shopt -s dotglob However when I tested this it still moved the hidden directory `.shouldNotMove` at the top of my `/mnt/cache/myshare/.shouldNotMove` to the array. I also sshed into my Unraid server and tailing the log I can see it being moved. If this is no longer a feature this comment will need to be removed from the mover script and I will must modify the find operation in the script myself. --------------------------------------------------- Possibly I am mis understanding something here but I hope this all make sense. NOTE Please don't suggest having a share with cache prefer as that would require a video editor to copy manually between shares. This is what I am trying to avoid! Video editors hate moving files its a massive time sink and I think UnRaid has a great solution here where this script can do it automatically. Thanks Quote Link to comment
Squid Posted May 25, 2021 Share Posted May 25, 2021 12 minutes ago, slimjim181 said: at the topmost level of a pool or an array disk are not moved. 12 minutes ago, slimjim181 said: However when I tested this it still moved the hidden directory `.shouldNotMove` at the top of my `/mnt/cache/myshare/.shouldNotMove` to the array. Your .shouldNotMove folder is not at the topmost level of a pool or disk. It's within a share, and the share is the topmost. IE: only .ShareName would not be moved. Quote Link to comment
slimjim181 Posted May 26, 2021 Author Share Posted May 26, 2021 @Squid thanks for you reply How is a user able to access that directory if its outside a share? what's the point of this feature then? I guess I can just modify the script myself, any pointers with what I am trying to achieve? Quote Link to comment
slimjim181 Posted May 26, 2021 Author Share Posted May 26, 2021 @Squid Your Mover Tuning plugin actually could be helpful, Could I use `Ignore files listed inside of a text file` to exclude a directory(s)? Quote Link to comment
Squid Posted May 26, 2021 Share Posted May 26, 2021 2 hours ago, slimjim181 said: @Squid Your Mover Tuning plugin actually could be helpful, Could I use `Ignore files listed inside of a text file` to exclude a directory(s)? Ask @hugenbdd in the support thread (he's long taken over the plugin) 1 Quote Link to comment
hugenbdd Posted May 26, 2021 Share Posted May 26, 2021 4 hours ago, slimjim181 said: @Squid Your Mover Tuning plugin actually could be helpful, Could I use `Ignore files listed inside of a text file` to exclude a directory(s)? This may work. I have a test mode now that you can try and review the "find" string and all the results (i.e. the files it's sending to the binary mover) in the syslog. This should make it easy to test or find a way to setup the filters like you want. If you test with the move-now button, make sure that the question "Move Now button follows plug-in filters:" is set to Yes. You could also label your directories something specific. i.e. put IGNORE at the end of them or something..... If you run into problems post over here. Quote Link to comment
hugenbdd Posted May 26, 2021 Share Posted May 26, 2021 14 hours ago, slimjim181 said: Your Mover Tuning plugin actually could be helpful, Could I use `Ignore files listed inside of a text file` to exclude a directory(s)? @slimjim181 If you can give me an example "find" command I could probably add it to the plug-in. I did find this link which has an example. https://askubuntu.com/questions/266179/how-to-exclude-ignore-hidden-files-and-directories-in-a-wildcard-embedded-find This version seems to be the easiest for me to implement. $ touch non_hidden_file.txt .hidden_file.txt somethings/.another_hidden_file.txt $ find . -type f -name '*hidden_file*' -and \( -not -name ".*" \) ./non_hidden_file.txt Quote Link to comment
slimjim181 Posted May 29, 2021 Author Share Posted May 29, 2021 @hugenbdd Thanks for your input! Ideally what I would like is the ability for the mover to just move folders or files with some sort of prefix on the front, using the dot might not be idea as that is hidden file. maybe a '--' (double dash) prefix? Let me explain it in user stories as a feature request to your awesome mover plugin: 1 Files/folders by default do not move from cache to array. As a video editor When I first load my project media onto an UnRaid NAS share And put into a project folder eg '20-4-4 B and R promotion' And the share has cache set to 'Yes' And I have enable/configured my Mover to only move files/folders with a '--' prefix Then my entire folder stays on the NAS 2 Files/folders only move when prefixed with '--' As a video editor When I am finished editing my project Then I edit my project folder name and add a prefix So '--20-4-4 B and R promotion' Then the next time the mover runs my project folder is moved from the cache to the array 3 Files/folders move back from array to cache if '--' prefix removed. As a video editor When a client comes back to me after I have finished a video And wants changes (typical client haha) Then I remove the '--' prefix off '--20-4-4 B and R promotion' -> '20-4-4 B and R promotion' Then my project folder is moved from the array back to the cache I hope that makes sense? The end result of all this means video editors can easily manage their working project and archived projects. This is actually a game changed and would give Unraid a great point of difference for video editors over TrueNAS as user do not have this level of control over the SLOG. Much appreciated Quote Link to comment
hugenbdd Posted May 29, 2021 Share Posted May 29, 2021 7 hours ago, slimjim181 said: @hugenbdd Thanks for your input! Ideally what I would like is the ability for the mover to just move folders or files with some sort of prefix on the front, using the dot might not be idea as that is hidden file. maybe a '--' (double dash) prefix? Let me explain it in user stories as a feature request to your awesome mover plugin: 1 Files/folders by default do not move from cache to array. As a video editor When I first load my project media onto an UnRaid NAS share And put into a project folder eg '20-4-4 B and R promotion' And the share has cache set to 'Yes' And I have enable/configured my Mover to only move files/folders with a '--' prefix Then my entire folder stays on the NAS 2 Files/folders only move when prefixed with '--' As a video editor When I am finished editing my project Then I edit my project folder name and add a prefix So '--20-4-4 B and R promotion' Then the next time the mover runs my project folder is moved from the cache to the array 3 Files/folders move back from array to cache if '--' prefix removed. As a video editor When a client comes back to me after I have finished a video And wants changes (typical client haha) Then I remove the '--' prefix off '--20-4-4 B and R promotion' -> '20-4-4 B and R promotion' Then my project folder is moved from the array back to the cache I hope that makes sense? The end result of all this means video editors can easily manage their working project and archived projects. This is actually a game changed and would give Unraid a great point of difference for video editors over TrueNAS as user do not have this level of control over the SLOG. Much appreciated oh wow! that a very specific use case. I think adding an "ignore hidden files" button would fit in the plug-in but not this. I have also only made one modification to the array -> cache code, as that is a very common problem of filling up cache and crashing dockers, etc. My modification allowed for a "test mode" to see what would be moved. You would be better off with a custom script that invokes the mover just for your case. (I have attached an example of code based on my plug-in, but it needs testing and the bugs worked out.) I added | grep -- to the find command. If you use this, you have to make sure that mover isn't running. (Beginning of example code) And that you also kill the PID file once it's done. (Ending of the code) Then just throw it in a cron job to run however often you like. (every 30 minutes, etc) videoeditor_mover Quote Link to comment
slimjim181 Posted May 31, 2021 Author Share Posted May 31, 2021 @hugenbddawesome thank you! I will look into this script, maybe I could built a plugin just for my use case. For the time being I would be keen to see an ignore hidden files and folders built into your plugin, how far away would those changes be from being published? Thanks again Quote Link to comment
hugenbdd Posted June 1, 2021 Share Posted June 1, 2021 On 5/31/2021 at 4:12 AM, slimjim181 said: @hugenbddawesome thank you! I will look into this script, maybe I could built a plugin just for my use case. For the time being I would be keen to see an ignore hidden files and folders built into your plugin, how far away would those changes be from being published? Thanks again A week, maybe two. Depending on if I get some free time this week. Quote Link to comment
hugenbdd Posted June 1, 2021 Share Posted June 1, 2021 On 5/31/2021 at 4:12 AM, slimjim181 said: @hugenbddawesome thank you! I will look into this script, maybe I could built a plugin just for my use case. For the time being I would be keen to see an ignore hidden files and folders built into your plugin, how far away would those changes be from being published? Thanks again I think I can use the options in the find command like this. find /mnt/cache/TV/Processed\ TV/ -not -path '*/\.*' Can you please test a find command but with your path or paths to make sure it's pulling back the files you want, and does not have the hidden files/folders you want to stay? Quote Link to comment
hugenbdd Posted June 4, 2021 Share Posted June 4, 2021 On 5/31/2021 at 4:12 AM, slimjim181 said: @hugenbddawesome thank you! I will look into this script, maybe I could built a plugin just for my use case. For the time being I would be keen to see an ignore hidden files and folders built into your plugin, how far away would those changes be from being published? Thanks again Updated. Please test and let me know if the Ignore Hidden Files setting is working how you expect. With my limited testing it appears to ignore anything with a . in the beginning of the name or file. 2020.06.03 Added size of all files from find string per share to test mode output. Added "Ignore Hidden Files" option. This adds -not -path '/.' to the end of the find command. Requested by slimjim181 to help with video editing folders. Quote Link to comment
slimjim181 Posted June 8, 2021 Author Share Posted June 8, 2021 @hugenbdd Very much appreciate your efforts on this, apologise for the slow response. I've just updated to your latest plugin (thanks for the special mention in the release notes) and have enabled your setting in the mover settings but am still seeing my folder `.doNotMove` being moved from the cache to the array. Possibly I haven't set it up right? Here is a screenshot of my settings Quote Link to comment
hugenbdd Posted June 8, 2021 Share Posted June 8, 2021 4 hours ago, slimjim181 said: @hugenbdd Very much appreciate your efforts on this, apologise for the slow response. I've just updated to your latest plugin (thanks for the special mention in the release notes) and have enabled your setting in the mover settings but am still seeing my folder `.doNotMove` being moved from the cache to the array. Possibly I haven't set it up right? Here is a screenshot of my settings If you are using the Move Now button, you have it set to follow the original unRAID mover settings. (i.e. All files) Else, can you do a few things. 1.) Enable Logging 2.) Move Now button Follows plug-in Filters: Yes 3.) Set Test mode: Yes Then post the syslog output for all of the mover entries. The start will look something like this Jun 8 01:45:01 Tower root: Log Level: 1 Jun 8 01:45:01 Tower root: mvlogger: *********************************MOVER START******************************* Jun 8 01:45:01 Tower root: mvlogger: Age supplied Jun 8 01:45:01 Tower root: mvlogger: Size supplied Jun 8 01:45:01 Tower root: mvlogger: Sparness supplied The end of the mover logging will look something like this Jun 8 02:01:35 Tower root: mover: finished Jun 8 02:01:35 Tower root: mvlogger: No After Script to Run. Quote Link to comment
slimjim181 Posted June 9, 2021 Author Share Posted June 9, 2021 @hugenbdd Aha I missed this setting, yes I was hitting the move now button. 2.) Move Now button Follows plug-in Filters: Yes Yes its working! any files or folders that are prefixed with a dot are ignored THANK YOU Quote Link to comment
Recommended Posts
Join the conversation
You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.