Bug - Mover script moving hidden folders

[slimjim181]

Context

I build custom NASs for artists such as video editors and this feature is important to me as I want a solution built into their workflow where their working projects will have a '.' (dot) at their root directory so that they are not moved to the array. But when they are finished with a project they can remove the dot.  Then overnight (assuming the mover is running daily) that project folder will be moved to the array 'archiving' that project. 

 

BUG User case

As a video editor

When I create a folder .shouldNotMove at the top level of my share myshare

And this share has its cache settings configured to 'Yes'  (files initially on cache then moved to array)

And I click the 'move now' button under Settings -> Schedule

 

Expected

.shouldNotMove directory remains in /mnt/cache/myshare and the files within .shouldNotMove also remain

 

Actual

.shouldNotMove is moved to the array and deleted from /mnt/cache/myshare

 

Developer details

Mover script found under `/usr/local/sbin/mover` has a line code comment:

# The script is set up so that hidden directories (i.e., directory names beginning with a '.'

# character) at the topmost level of a pool or an array disk are not moved.  This behavior

# can be turned off by uncommenting the following line:

# shopt -s dotglob

 

However when I tested this it still moved the hidden directory `.shouldNotMove` at the top of my `/mnt/cache/myshare/.shouldNotMove` to the array.

 

I also sshed into my Unraid server and tailing the log I can see it being moved.

 

If this is no longer a feature this comment will need to be removed from the mover script and I will must modify the find operation in the script myself. 

 

---------------------------------------------------

 

Possibly I am mis understanding something here but I hope this all make sense.

 

NOTE Please don't suggest having a share with cache prefer as that would require a video editor to copy manually between shares.  This is what I am trying to avoid! Video editors hate moving files its a massive time sink and I think UnRaid has a great solution here where this script can do it automatically. 

 

Thanks 

[Squid]

12 minutes ago, slimjim181 said:

at the topmost level of a pool or an array disk are not moved.

 

12 minutes ago, slimjim181 said:

However when I tested this it still moved the hidden directory `.shouldNotMove` at the top of my `/mnt/cache/myshare/.shouldNotMove` to the array.

Your .shouldNotMove folder is not at the topmost level of a pool or disk.  It's within a share, and the share is the topmost.

 

IE: only .ShareName would not be moved.

[slimjim181]

@Squid thanks for you reply

 

How is a user able to access that directory if its outside a share? what's the point of this feature then?

 

I guess I can just modify the script myself, any pointers with what I am trying to achieve?

 

[slimjim181]

@Squid Your Mover Tuning plugin actually could be helpful, 

 

Could I use `Ignore files listed inside of a text file`  to exclude a directory(s)?

[Squid]

2 hours ago, slimjim181 said:

@Squid Your Mover Tuning plugin actually could be helpful, 

 

Could I use `Ignore files listed inside of a text file`  to exclude a directory(s)?

Ask @hugenbdd in the support thread (he's long taken over the plugin)

[hugenbdd]

4 hours ago, slimjim181 said:

@Squid Your Mover Tuning plugin actually could be helpful, 

 

Could I use `Ignore files listed inside of a text file`  to exclude a directory(s)?

This may work.  I have a test mode now that you can try and review the "find" string and all the results (i.e. the files it's sending to the binary mover) in the syslog.  This should make it easy to test or find a way to setup the filters like you want.  If you test with the move-now button, make sure that the question "Move Now button follows plug-in filters:" is set to Yes.  You could also label your directories something specific.  i.e. put IGNORE at the end of them or something.....

 

If you run into problems post over here.  

 

[hugenbdd]

14 hours ago, slimjim181 said:

Your Mover Tuning plugin actually could be helpful, 

 

Could I use `Ignore files listed inside of a text file`  to exclude a directory(s)?

@slimjim181

If you can give me an example "find" command I could probably add it to the plug-in.

 

I did find this link which has an example.

https://askubuntu.com/questions/266179/how-to-exclude-ignore-hidden-files-and-directories-in-a-wildcard-embedded-find

 

This version seems to be the easiest for me to implement.

$ touch non_hidden_file.txt .hidden_file.txt somethings/.another_hidden_file.txt

$ find . -type f -name '*hidden_file*' -and \( -not -name ".*" \)

./non_hidden_file.txt

[slimjim181]

@hugenbdd

 

Thanks for your input!

 

Ideally what I would like is the ability for the mover to just move folders or files with some sort of prefix on the front, using the dot might not be idea as that is hidden file. maybe a '--' (double dash) prefix?

 

Let me explain it in user stories as a feature request to your awesome mover plugin:

 

1 Files/folders by default do not move from cache to array.

As a video editor 

When I first load my project media onto an UnRaid NAS share

And put into a project folder eg '20-4-4 B and R promotion'

And the share has cache set to 'Yes'

And I have enable/configured my Mover to only move files/folders with a '--' prefix

Then my entire folder stays on the NAS

 

2 Files/folders only move when prefixed with '--'

As a video editor 

When I am finished editing my project

Then I edit my project folder name and add a prefix 

So '--20-4-4 B and R promotion'

Then the next time the mover runs my project folder is moved from the cache to the array

 

3 Files/folders move back from array to cache if '--' prefix removed.

As a video editor 

When a client comes back to me after I have finished a video

And wants changes (typical client haha)

Then I remove the '--' prefix off  '--20-4-4 B and R promotion' -> '20-4-4 B and R promotion'

Then my project folder is moved from the array back to the cache

 

I hope that makes sense?

 

The end result of all this means video editors can easily manage their working project and archived projects. This is actually a game changed and would give Unraid a great point of difference for video editors over TrueNAS as user do not have this level of control over the SLOG.

 

Much appreciated

[hugenbdd]

7 hours ago, slimjim181 said:

@hugenbdd

 

Thanks for your input!

 

Ideally what I would like is the ability for the mover to just move folders or files with some sort of prefix on the front, using the dot might not be idea as that is hidden file. maybe a '--' (double dash) prefix?

 

Let me explain it in user stories as a feature request to your awesome mover plugin:

 

1 Files/folders by default do not move from cache to array.

As a video editor 

When I first load my project media onto an UnRaid NAS share

And put into a project folder eg '20-4-4 B and R promotion'

And the share has cache set to 'Yes'

And I have enable/configured my Mover to only move files/folders with a '--' prefix

Then my entire folder stays on the NAS

 

2 Files/folders only move when prefixed with '--'

As a video editor 

When I am finished editing my project

Then I edit my project folder name and add a prefix 

So '--20-4-4 B and R promotion'

Then the next time the mover runs my project folder is moved from the cache to the array

 

3 Files/folders move back from array to cache if '--' prefix removed.

As a video editor 

When a client comes back to me after I have finished a video

And wants changes (typical client haha)

Then I remove the '--' prefix off  '--20-4-4 B and R promotion' -> '20-4-4 B and R promotion'

Then my project folder is moved from the array back to the cache

 

I hope that makes sense?

 

The end result of all this means video editors can easily manage their working project and archived projects. This is actually a game changed and would give Unraid a great point of difference for video editors over TrueNAS as user do not have this level of control over the SLOG.

 

Much appreciated

 

oh wow! that a very specific use case.

I think adding an "ignore hidden files" button would fit in the plug-in but not this.  I have also only made one modification to the array -> cache code, as that is a very common problem of filling up cache and crashing dockers, etc.  My modification allowed for a "test mode" to see what would be moved.

 

You would be better off with a custom script that invokes the mover just for your case. (I have attached an example of code based on my plug-in, but it needs testing and the bugs worked out.)  I added | grep -- to the find command.

 

If you use this, you have to make sure that mover isn't running. (Beginning of example code)  And that you also kill the PID file once it's done. (Ending of the code)

 

Then just throw it in a cron job to run however often you like. (every 30 minutes, etc)

videoeditor_mover

[slimjim181]

@hugenbddawesome thank you! I will look into this script, maybe I could built a plugin just for my use case.

For the time being I would be keen to see an ignore hidden files and folders built into your plugin, how far away would those changes be from being published? 

 

Thanks again

[hugenbdd]

On 5/31/2021 at 4:12 AM, slimjim181 said:

@hugenbddawesome thank you! I will look into this script, maybe I could built a plugin just for my use case.

For the time being I would be keen to see an ignore hidden files and folders built into your plugin, how far away would those changes be from being published? 

 

Thanks again

A week, maybe two.  Depending on if I get some free time this week.

[hugenbdd]

On 5/31/2021 at 4:12 AM, slimjim181 said:

@hugenbddawesome thank you! I will look into this script, maybe I could built a plugin just for my use case.

For the time being I would be keen to see an ignore hidden files and folders built into your plugin, how far away would those changes be from being published? 

 

Thanks again

I think I can use the options in the find command like this.

find /mnt/cache/TV/Processed\ TV/ -not -path '*/\.*'

 

Can you please test a find command but with your path or paths to make sure it's pulling back the files you want, and does not have the hidden files/folders you want to stay?

[hugenbdd]

On 5/31/2021 at 4:12 AM, slimjim181 said:

@hugenbddawesome thank you! I will look into this script, maybe I could built a plugin just for my use case.

For the time being I would be keen to see an ignore hidden files and folders built into your plugin, how far away would those changes be from being published? 

 

Thanks again

Updated.

Please test and let me know if the Ignore Hidden Files setting is working how you expect.  With my limited testing it appears to ignore anything with a . in the beginning of the name or file.

 

2020.06.03

Added size of all files from find string per share to test mode output.

Added "Ignore Hidden Files" option. This adds -not -path '/.' to the end of the find command. Requested by slimjim181 to help with video editing folders.

[slimjim181]

@hugenbdd Very much appreciate your efforts on this, apologise for the slow response.

I've just updated to your latest plugin (thanks for the special mention in the release notes) and have enabled your setting in the mover settings but am still seeing my folder `.doNotMove` being moved from the cache to the array.

 

Possibly I haven't set it up right?

 

Here is a screenshot of my settings

[hugenbdd]

4 hours ago, slimjim181 said:

@hugenbdd Very much appreciate your efforts on this, apologise for the slow response.

I've just updated to your latest plugin (thanks for the special mention in the release notes) and have enabled your setting in the mover settings but am still seeing my folder `.doNotMove` being moved from the cache to the array.

 

Possibly I haven't set it up right?

 

Here is a screenshot of my settings

If you are using the Move Now button, you have it set to follow the original unRAID mover settings.  (i.e. All files)

 

Else, can you do a few things.

1.) Enable Logging

2.) Move Now button Follows plug-in Filters: Yes

3.) Set Test mode: Yes

 

Then post the syslog output for all of the mover entries.   

The start will look something like this

 

Jun  8 01:45:01 Tower root: Log Level: 1
Jun  8 01:45:01 Tower root: mvlogger: *********************************MOVER START*******************************
Jun  8 01:45:01 Tower root: mvlogger: Age supplied
Jun  8 01:45:01 Tower root: mvlogger: Size supplied
Jun  8 01:45:01 Tower root: mvlogger: Sparness supplied

 

The end of the mover logging will look something like this

Jun  8 02:01:35 Tower root: mover: finished
Jun  8 02:01:35 Tower root: mvlogger: No After Script to Run.
 

[slimjim181]

@hugenbdd 
Aha I missed this setting, yes I was hitting the move now button.

2.) Move Now button Follows plug-in Filters: Yes

Yes its working! 

any files or folders that are prefixed with a dot are ignored

 

THANK YOU