Can somebody replicate the following Bug with 6.1.4:

 

1) You need two shares: one set to use cache = yes and the other set to use cache = no

2) Copy a directory with files in it  to the cached share (make sure the mover is not running to move the file yet and is not running for the next steps)

3) ren/move the directory with "mc" from user/cachedshare/directory_with_files to user/non_cachedshare/directory_with_files

 

Result right after step 3: the moved directory is only on the cache drive even though the share it was moved to is set to not use the cache

 

Result after step 3 and after running the mover: The directory/files are skipped and the files remain on the cache drive - even though they reside on a share which is set to not use the cache. Presumably the mover skips the files because it follows the logic that a non-cached share does not need to be moved.

 

Summarizing: Moving files from a cached share to a non-cached share may result in directories/files on the cache drive which are never moved.

 

Solution 1: If the destination of a move is a user share that is not set to use the cache, directly move the files to the array drives instead of renaming it on the cache. (Without breaking the "smart" rename if files are just moved between cached shares)

 

Solution 2: Make the mover always move all files away from cache drive even for non-cached shares. This would keep the shares not-cached after a mover run, if tthe cache drive "accidently" or "intentionally" contains files for this share. This would be a  form of periodic housekeeping.

This is not a bug.

I find it hard to believe that this is not a bug. Could you maybe explain the rational behind this behavior and why you think this is normal?

 

This could lead to situations where through normal use files accumulate on the cache drive that are never moved. This would after a while lead to zero space on the cache drive and degraded performance and even worse to failure of plug-ins that rely on free space on the cache drive. I would have expected that under no circumstance files of a share that is explicitly set to NOT use the cache land on the cache drive (or at least that they do not stay there, see solution 2).....

 

I am curious to why this behavior is not changed - updating the mover as in solution 2 seems not to difficult and would prevent the cache drive from gradually filling up ?!? One could live with current behavior but then it should at least be addressed in the documentation as a known issue, for example by stating that one should run the mover before trying to move files from a cached-share to a non-cached share.

 

 

The only reason you witnessed this behavior is because you accessed unRAID from command line to perform these functions.  If you would have copied the directory with files (or "cut" it) using Windows explorer over SMB and then moved it to another share, this problem would not occur.

 

Using MC is an advanced command line utility and the behaviors of user shares will be a little different as a result.  Linux is doing what it thinks is right.  We do not officially support the use of Linux command line but rather, if advanced users understand how to leverage it, go for it.

To extend on that, there are all sorts of bad operations folks could do using command line that could cause issues, so if you are not familiar with Linux or can't accept that command line interaction with unRAID is "use at your own risk," then the simple answer is "don't do that.". ;-)

One other comment. If you want to force the folder to move to a specific share on a specific disk using MC, you could do that by setting the destination path through /mnt/disk#/share instead of /mnt/user/share.

I find it hard to believe that this is not a bug. Could you maybe explain the rational behind this behavior and why you think this is normal?

 

This could lead to situations where through normal use files accumulate on the cache drive that are never moved. This would after a while lead to zero space on the cache drive and degraded performance and even worse to failure of plug-ins that rely on free space on the cache drive. I would have expected that under no circumstance files of a share that is explicitly set to NOT use the cache land on the cache drive (or at least that they do not stay there, see solution 2).....

 

I am curious to why this behavior is not changed - updating the mover as in solution 2 seems not to difficult and would prevent the cache drive from gradually filling up ?!? One could live with current behavior but then it should at least be addressed in the documentation as a known issue, for example by stating that one should run the mover before trying to move files from a cached-share to a non-cached share.

 

Your solution 2 is how it used to work.  But being able to leave files on the cache disk and not move them to the array is necessary feature.

The only reason you witnessed this behavior is because you accessed unRAID from command line to perform these functions.

 

This would also affect plugins and dockers, right?  I'm thinking of the Krusader docker specifically, but really anything that moves files directly on the server could trigger it.

 

being able to leave files on the cache disk and not move them to the array is necessary feature.

 

Agreed, but setting "Use cache disk" = "only" seems like a better way to accomplish this because it works over network connections as well as the command line.

 

For example, jonp showed a great way to keep certain files on the cache while others stay on the array:

  http://lime-technology.com/forum/index.php?topic=40777.msg385753

and a few of us borrowed that technique for use with Crashplan:

  https://lime-technology.com/forum/index.php?topic=33864.msg393993#msg393993

 

Your solution 2 is how it used to work.

 

In earlier versions of unRAID, people kept running into trouble because they would create an appdata folder on their cache drive without realizing all top level folders automatically become user shares. Because they didn't change the share's "Use cache disk" setting to "only", mover would move the appdata folder to the array and Docker would break.

 

In version 6.1.0, mover was updated to only move shares which were specifically designated with "Use cache disk" = "yes".  This made logical sense and solved the appdata problem, so everyone was happy.

 

But this was before we realized there was a corner case with the "Use cache disk" = "no" setting, where unRAID would sometimes put files on the cache drive anyway.  Knowing that, it seems like we should modify mover again, so that it moves files where the share's "Use cache disk" setting is either "yes" or "no".  This would allow mover to "fix" this corner case and move the files back to the array where the "Use cache disk" setting says they should be.

 

Is this a reasonable solution?

 

Can somebody replicate the following Bug with 6.1.4:

 

1) You need two shares: one set to use cache = yes and the other set to use cache = no

2) Copy a directory with files in it  to the cached share (make sure the mover is not running to move the file yet and is not running for the next steps)

3) ren/move the directory with "mc" from user/cachedshare/directory_with_files to user/non_cachedshare/directory_with_files

 

Result right after step 3: the moved directory is only on the cache drive even though the share it was moved to is set to not use the cache

Ok I see what you are referring to.  The problem is that mc/cp/mv commands all 'see' that /mnt/user/cachedshare and /mnt/user/non_cachedshare are both in the same file system.  Hence they generate "rename" operations instead of 'copy' operations.  I can see where this can be interpreted as a 'bug', and I can also see where this behavior is a desirable 'feature'.  I need to give this more thought.  Curious: what is your use case?

My use case is pretty basic: I have an "incoming" share that uses the cache drive (1TB SSD) for downloading/repairing/unpacking by sabnzbd running on the unraid server. Whenever I like something enough to keep, I move it to my e.g. "movies" share which does not use the cache drive. The reason for not using the cache drive for that share is that I additionally move large files there (20gb+) from other sources and do not want to wear down the ssd for those. Naturally I do not copy these big files over the network if they reside  somewhere on the unraid server (For example on an NTFS drive through "unassigned devices").

 

I think that the easiest and fully adequate solution would be to change the mover script to also move shares to the array that are set to not use the cache. This does NOT mean ALL files should be moved to the array: hidden files (starting with a dot), cache-only shares, files on the root directory of the cached drive need not to be moved. This would allow for files to reside on the cache drive and solve the problem. It would also not induce any additional wear on the ssd cache drive as the files just get renamed and then moved by the mover to the array.

 

Thanks for taking some more time to understand my issue - Maybe bug is a little to harsh, but for me it was totally unexpected behavior.

 

 

My use case is pretty basic: I have an "incoming" share that uses the cache drive (1TB SSD) for downloading/repairing/unpacking by sabnzbd running on the unraid server. Whenever I like something enough to keep, I move it to my e.g. "movies" share which does not use the cache drive. The reason for not using the cache drive for that share is that I additionally move large files there (20gb+) from other sources and do not want to wear down the ssd for those. Naturally I do not copy these big files over the network if they reside  somewhere on the unraid server (For example on an NTFS drive through "unassigned devices").

 

I think that the easiest and fully adequate solution would be to change the mover script to also move shares to the array that are set to not use the cache. This does NOT mean ALL files should be moved to the array: hidden files (starting with a dot), cache-only shares, files on the root directory of the cached drive need not to be moved. This would allow for files to reside on the cache drive and solve the problem. It would also not induce any additional wear on the ssd cache drive as the files just get renamed and then moved by the mover to the array.

 

Thanks for taking some more time to understand my issue - Maybe bug is a little to harsh, but for me it was totally unexpected behavior.

 

Your proposed solution is exactly the way it used to work pre-6.1.  We changed it to support the case where you want to have some files of a share to exist and stay on the cache, and other files of the same share to exist on the array.  For example, say you have a share called 'vdisks'.  Maybe you want some image files in that share to be on an ssd-based cache for speed, but other image files are ok to keep on a slower array hdd.

 

A workaround for you would be to copy to target share under /mnt/user0 instead of /mnt/user.  eg:

 

mv /mnt/user/cachedshare/directory_with_files /mnt/user0/non_cachedshare/directory_with_files

 

This way the 'mv' command (or equivalent 'mc' operation) detects this is a 'move' between two different files systems instead of 'rename'.

 

The /mnt/user0 directory tree is the exact same union of all disks except cache.  This is what the mover uses as the target file system btw.

Copying from mnt/user to mnt/user0 is a good workaround - if source is cached and destination is non-cached. Thinking about the problem why not introduce an additonal setting for the shares

 

1) "Use Cache Disk" has the following settings:

 

Yes

Cache Only

No, but allow files on cache drive

No, mover will move all files to array

 

or alternatively:

2) Keep current "Use Cache Disk" settings

Yes

Cache Only

No

and add an additonal checkbox:

mover ignores share (yes/no)

 

 

Copying from mnt/user to mnt/user0 is a good workaround - if source is cached and destination is non-cached. Thinking about the problem why not introduce an additonal setting for the shares

 

1) "Use Cache Disk" has the following settings:

 

Yes

Cache Only

No, but allow files on cache drive

No, mover will move all files to array

 

or alternatively:

2) Keep current "Use Cache Disk" settings

Yes

Cache Only

No

and add an additonal checkbox:

mover ignores share (yes/no)

I haven't really thought about these additional settings, but it is meaningless unless you take into consideration what the default setting will be. If a user doesn't configure a user share, what should the system assume the use cache setting is?

As only one additonal setting is added, defaults could remain the same as they are now. So for users nothing would change.

 

As only one additonal setting is added, defaults could remain the same as they are now. So for users nothing would change.

The default setting is currently "Use cache disk: No", but you have proposed multiple variations of "Use cache disk: No", which is why I raised the point. It is the default settings that always got users into trouble because they would often create a top level folder, which automatically becomes a user share with default settings. The old mover script skipped "Use cache disk: Only" shares. The current mover script only moves "Use cache disk: Yes" shares. See this thread where these problems were discussed which led to our current implementation of mover.

"use cache disk: no" (the current default) would correspond to "No, but allow files on cache drive". Operation under these settings would be completely identical.

IS it that the mover now skips any top level cache drive directory unless the share is set to "Yes". Why not make it so it only skips shares set to "Only" instead?

IS it that the mover now skips any top level cache drive directory unless the share is set to "Yes". Why not make it so it only skips shares set to "Only" instead?

This is because some Use Cases have been identified where it is useful to be able to explicitly place files on the cache disk that are treated as part of the Use Share for read purposes, but where writes to the User Share always write to the data disks.

 

I personally think it would be useful to make this explicitly a new option in the settings, but that is not currently the case.

IS it that the mover now skips any top level cache drive directory unless the share is set to "Yes". Why not make it so it only skips shares set to "Only" instead?

That is what mover used to do and people kept "accidentally" creating app shares that defaulted to cache-no, so they got moved because they weren't skipped as cache-only. Mover was changed to fix this recurring issue had by many, many users. Please see the link I posted in Reply #14 above. In fact, you posted in that other thread.

LOL, yes I did. And I posted a different way to fix it too back then but it was more complex and might have had other side effects. Changing to "yes" was simpler hence it was done that way.

 

The scripts are in that link you posted. It would be relatively easy for someone to just put a modified copy on their flash drive and then add a command line into the go file that copies it to the ram filesystem every boot.