Can the mover be set to ignore some folder or user share?


shEiD

Recommended Posts

I want to have rutorrent and nzbget download folders be on the cache drive, because it's an SSD.

 

Now nzbget - this is not a problem, I can set it to cache-Only. Or I could even probably use an additional SSD with unnasigned disks plugin.

I was just now in the process of setting up nzbget, sonarr and everything else and I realized, that:

actually, all of them - nzbget, rutorrent, sonarr, radar and anything else of this kind, needs to be pointed to the exact same user share (eg: /mnt/user/downloads), for them to work perfectly. That means that even nzbget downloads will be on the user share with a setting Use cache disk: Yes. And that is very inconvenient.

 

The rutorrent folder - I need it to be an actual user share with the setting: Use cache disk: Yes. This way rutorrent will download everything into the SSD cache drive, and then I can move those files manually to the pool and keep seeding.

 

I use user shares and split levels to keep my files in a particular order on the data drives. For example - I have movies and tv shows on separate "drive groups", by using separate user shares and Included disk(s) option. Also, I use split levels to keep all of the files belonging to a particular tv show together on the same disk.

 

The user share used by rutorrent is obviously using all the dives in the pool - because I want to seed all sorts of files, from any of the data drives.

In this scenario - if the mover moves the files from rutorrent folder - it has no idea if it's a movie, or an episode, etc, so it will pick any drive from the pool. And that is no help at all, because I will need to actually find those files and move them to a proper disk myself, just now I need to go hunting for them among 28 drives, instead of them all being on a single cache drive.

 

Basically, it's simple torrenting stuff.

 

  1. So can mover somehow be set to ignore some particular user share's files on teh cache drive, and leave them alone?
  2. If not, can mover script be replaced by a "smarter" mover by using a plugin?
  3. Finally, can mover be disabled completely, if want to manage files on the cache drive and the user shares using cache manually?

 

I tried to search for info on this on the forums, but could not find anything recent. Judging form the old discussions, the answers to all 3 would be NO. Which is hard to believe. I hope I misunderstood something or something has changed over last years.

Edited by shEiD
Link to comment

1). This is not an option.   Yours seems rather an unusual requirement so it is unlikely to be added I would think.

2).  Others have replaced the script with their own customised version.   No plugin for this though.

3).  Although you cannot disable it completely via the GUI you could set it to only run monthly.    Not sure if it is possible to manually edit a configuration file to pick a value (e.g. 32) for a day-of-the-month that cannot occur.   Another possibility is to add an entry to your ‘go’ file that renames the mover file which stops it running although it is a bit of a heavy-handed approach as it also then stops you running it manually.

Link to comment

@itimpi Thank you for the answers.

 

It is rather strange, that such an important part as mover script, which actually, imho, is the most important aspect that makes the use of cache drive useful - that it does not have an ability to ignore folders/shares. I mean, you are forced to use user shares either in cache=No or cache=Only, with a "dumb" mover script like that.

 

I am new to unraid, so maybe I haven't discovered a better way to achieve this. But as of now - it seems anyone using unraid and using torrents/usenet on it, would be in same situation as me.

 

What is more baffling, is that mover cannot be easily disabled via a simple setting in a webUI. I seriously do not understand the logic of this decision.

Link to comment

You are asking for a very unusual combination of capabilities!   You normally control mover via the share setting and tell to ignore folders by using the cache=no setting, but you are wanting a strange combination of cache=yes and mover ignoring that setting on the folder.   The vast majority of people want mover to run without any manual intervention and for that the current settings are adequate.   Asking for an option to disable mover via the GUI is reasonable, but beyond that is (I think) beyond what most people want.

 

Have you tried having a setting of cache=only on the ‘downloads’ share and then manually copying the files to a ‘downloads’ folder on particular disks thus bypassing the User Share settings?    The way unRAID works I think these files would still be part of the ‘downloads’ share for read purposes because the cache settings only control writing and not reading.  Whether this is practical in your particular scenario I am not sure.

 

Having said that LimeTech have stated that they intend to enhance the mover tool in the future.    I believe for instance they are thinking of it supporting most (if not all) of the functionality of the unBalance plugin.    You could at least register what you want as a Feature request to see if it can be incorporated into that work.  

Link to comment

Does the putting a . in front of a directory not still work?

 

Here is some talk about disabling the Mover Script. Of course its part of the OS, so keep good notes of what your changing incase you want to restore things.

 

I do exactly what @itimpi Posted about. I use different shares and User.Scripts to automate my moving of files to other shares with either something Automated or via a clickable button. 

 

 

Link to comment

I've just realised I have a solution for you.  I do something similar already just the other way around.

 

I have my appdata set to cache only, but I don't want my background plex transcodes (when syncing to cloud sync or to a device) to be on my cache drive as they can take up a lot of space e.g. a cloud sync run could create 1TB of files that would sit on my cache drive (too small) until uploaded.

 

What I've done to fix this is create a new share called Plex Sync which is set to array only and created a bind mount from the offending appdata folder to the Plex Sync share on the array.  This allows plex docker etc to access the files as though they are still stored in appdata on my cache drive, when in fact they are on my array in Plex Sync.

 

mount --bind /mnt/user/plex_sync/ "/mnt/cache/appdata/plex/Library/Application Support/Plex Media Server/Cache/Transcode/Sync+"

I run this at the startup of the array via the user scripts plugin:

 

#!/bin/bash

mount --bind /mnt/user/plex_sync/ "/mnt/cache/appdata/plex/Library/Application Support/Plex Media Server/Cache/Transcode/Sync+"

 

If you flip this, you can do the same for your scenario:

 


mount --bind /mnt/user/CACHE_ONLY_SHARE/ "/mnt/cache/appdata/DIRECTORY_ON_CACHE_I_DONT_WANT_MOVER_TO_MOVE

i.e create a new 'virtual' share that is set to cache only and then link files from cache shares that you don't want the mover to move

Edited by DZMM
Link to comment
  • 1 year later...

You say it's an unusual request, however, imagine this scenario:

You have a working folder, (downloads, video editing, whatever) You don't want the mover to do anything with this. Share - Cache Only.

You have a storage folder (Movies, Music, Work, whatever) You do want this to use the cache so you can move files from the working folder to it at speed. Share - Cache Yes

 

Problem is when you move a file from Working share to Storage share (from one cached share to another) unRAID actually lifts the file and writes it back to the same cache drive.  From the Cache to the Cache, this takes a lot longer than a disk to disk move as the reads and writes can't happen concurrently.

 

If the Storage share could contain a Cache only folder, then the working folder could exist within the Storage share, not get moved by the Mover and not actually require a lift and shift move when it's moved from one folder to another because it's in the same share.

 

Would save a lot of IOPS.

Link to comment
18 minutes ago, OFark said:

Problem is when you move a file from Working share to Storage share (from one cached share to another) unRAID actually lifts the file and writes it back to the same cache drive.  From the Cache to the Cache, this takes a lot longer than a disk to disk move as the reads and writes can't happen concurrently.

Have you actually tried this?   I thought that in such a scenario a simple rename was used which is close to instantaneous.

Link to comment
52 minutes ago, OFark said:

Problem is when you move a file from Working share to Storage share (from one cached share to another) unRAID actually lifts the file and writes it back to the same cache drive.

17 minutes ago, itimpi said:

Have you actually tried this?   I thought that in such a scenario a simple rename was used which is close to instantaneous.

THIS^

 

If you move from one user share to another, linux sees all the user shares as having the same mount and so it just "renames" the files/folders so they have a different path on the same disk. This behavior often surprises people because something moved from a cache-only share to a cache-no share may stay on cache in a folder for that cache-no share on the cache drive.

 

The workaround is to copy from source to destination followed by a delete from source. Copy forces a new write and so obeys the user share settings.

 

I just tried this again to confirm and it does exactly as I said. I moved a file from a cache-only share to a cache-no share. The move was instantaneous. Checking the actual disks shows that a new folder for the cache-no share exists on cache with the file in it. So it was just "renamed" to a new path on the same disk.

 

Link to comment
15 hours ago, itimpi said:

Have you actually tried this?   I thought that in such a scenario a simple rename was used which is close to instantaneous.

Yes, I have tried this, with a large file. Try it. Move a file from one cached share to another. I moved a 90gb file from my working folder to my storage share and got the attached graph.

 

See this Facebook discussion:

 

Basically, because the shares can have different disk options, unRAID always copies the files to a new location.

49948639_10161052918220411_1935156416394297344_n.jpg

Link to comment
4 hours ago, OFark said:

Yes, I have tried this, with a large file. Try it. Move a file from one cached share to another. I moved a 90gb file from my working folder to my storage share and got the attached graph.

 

See this Facebook discussion:

 

Basically, because the shares can have different disk options, unRAID always copies the files to a new location.

49948639_10161052918220411_1935156416394297344_n.jpg

Did you do this across the network using Windows or something?

 

I did it directly on the server with the result I already posted above.

Link to comment
1 minute ago, OFark said:

That was the Mover.

Mover treats cache separately from the other disks when working with the user shares. It couldn't do its job if it didn't.

 

Regardless, I'm having a little trouble seeing exactly where in your previous post mover is invoked. You were talking about moving something from a cache-only share to a cache-yes share. Mover doesn't move cache-only shares, so I assumed you or some other process was doing the moving.

 

I'm not sure how to attempt to duplicate what you said happened. Maybe you could explain it in a way that included where exactly mover gets involved.

 

 

Link to comment

You are correct the Mover doesn't move things between shares, I was using the unRAID terminal for that file transfer.

 

To replicate, create two shares that use the cache, in any way. With one cache drive; move a large file (several GB's) from one share to another. Make sure you're using the share folder /mnt/user/whatever not direct to the disk e.g /mnt/cache/whatever

Link to comment

There is one bit of your post that isn't clear

41 minutes ago, OFark said:

With one cache drive;

Unraid only has a single cache pool, which can contain multiple drives. But however many drives the pool contains, it is treated as a single entity. I happen to have 2 drives in my cache pool.

 

OK. I moved an 8GB file from a cache-only user share to a cache-yes user share. This is basically the same test I did before except that time I moved to a cache-no share.

 

I got the same result as before. The move was instantaneous.

Link to comment

I haven't tried with two cache drives. But then it still wouldn't be instantaneous.

I don't know what to tell you, I have a 90GB file in one share set to cache only, I moved it via the terminal to the storage share (cache-yes) and it took ages. On inspecting the stats graph I got what you see above. If I do moves inside the share then it's instant as it can just update the file table, but across shares, it's doing a physical move. I was told on Facebook that this is the way unRAID works so it can apply disk rules.

 

I can't even suggest that I might have it configured wrong, as I can't see what you could reconfigure, but it would be damned useful.

 

Maybe 6.7 deals with this problem.

Edited by OFark
Link to comment
6 minutes ago, OFark said:

Maybe 6.7 deals with this problem.

It's not a problem, and the way it worked for me is the way it has always worked.

 

If anything, some would say it's a problem that it does what it did for me, which seems to be what you are proposing it should do.

 

It just "renamed" the path. That actually breaks the user share rules when it results in a folder for that user share being created on a disk that isn't supposed to be part of that user share.

 

That rule breaking is exactly what took place when I did my first test above. I moved from a cache-only share to a cache-no share. Logically you would think that would result in it moving to a disk other than cache. Instead, it just renamed it with a different path on the cache disk and that resulted in the cache disk having a folder for a cache-only share.

 

Link to comment
4 minutes ago, Squid said:

If your command line was something like

mv /mnt/cache/blahblahblah /mnt/user/blahblah

 

Then it will do the copy / delete process.

 

Specifying /mnt/user for both source and destination should just rename it to the new folder

 

That is dangerously close to the "user share copy bug", but would be avoided if the "blah" parts were different from each other as in your example.

 

So, @OFark, is that what you did, a move from a disk to a user share?

Link to comment

I can't remember at this point, however, I have some containers that are set up using user shares. They automate stuff from one share to another, and they're showing signs of the same behaviour I described.

 

I'm going to try moving a large file again in a few hours time, record the steps and see what's going wrong/right. 

Link to comment
21 minutes ago, trurl said:

is that what you did, a move from a disk to a user share?

Forgot to mention. Don't move from a disk to a user share or user share to a disk.

22 minutes ago, trurl said:

That is dangerously close to the "user share copy bug"

If the paths are the same it will try to overwrite the file it is trying to read.

Link to comment

Ok, so I'm lost now. You are correct, from a user share to another user share the move is instantaneous. Yet I've seen it do it before. My automatic file processors take ages to move a file from my working folder to my storage folder, I have no idea why. I've double checked the container template and the paths are shared by user path.  😕

Link to comment

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.

Guest
Reply to this topic...

×   Pasted as rich text.   Restore formatting

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.