Share set to not use the cache, but files / folders exist on the cache drive


Recommended Posts

Hi.

I got few times email from Fix Common Problems plugin saying that "Share media set to not use the cache, but files / folders exist on the cache drive", and that is true, I do have share "media" saying No for use cache, and when I check files from krusader I do see media folder on the /mnt/cache folder. 

And it contains some files I moved over the network using Music Brainz picard, it also contains some media that I moved trough Krusader (here I never access shares outside of /mnt/user folder, although I do have mounted entire /mnt folder in Krusader, basically to just control such things as this. Krusader is set per SpaceinvaderOne's video). 

I checked all docker containers, none has access to the media share except Krusader. Which would then be easy to blame, but I do move some files using my windows machine and Music Brainz picard, via network share, and over the network I only have few shares, not the cache drive itself. 

When I run mover, nothing happens, files stay split like that for weeks. 

 

Where should I start looking, what should I check first?

Thanks!

Link to comment

If you have Use Cache=No then mover will never move files from the cache to the array (it needs to be set to “yes” for this to happen).   The question is how they got there?

 

One way is for an app running locally on the server that tries to do a ‘move’ (rather than a copy and then delete) of a file that is on the cache to a different share.  One of the idiosyncrasies of the Linux ‘move’ operation is that Linux can first try a ‘rename’ (which is fast) and only if that fails does it do a copy/delete action.   This can result in a file being moved to a folder that corresponds to a share with cache=No set.

Link to comment

Thanks @itimpi you perfectly described the issue as I do move files from download (on a cached drive) to media share (not on a cached drive).

Now the question is how to solve this issue and have all my shares on their desired locations?


It would be ok if the mover could perhaps fix this automatically, as this doesn't look like an isolated and unlikely scenario to me (it would be even better if this doesn't occur in the first place, but I understand that technology has its limitations). This basically means that if one is using the cache drive, data can end up all over the place and some part of it will not be parity protected.

 

Should this perhaps be elevated to the unraid team via some ticket or something?

Link to comment

Copy from source to destination. That will result in a new file being written at the destination, and that new file will be created according to the settings for that user share.

 

Then delete from the source.

 

That way you get a move that does what you want, copy to destination then delete from source. Instead of a move where Linux is free to decide whether to just rename its path.

 

An alternative would be to make that destination user share cache-yes. Moving from one share to the other would result in the file on cache, but mover would move it to the array at the scheduled time. You would need to be sure to have enough capacity in cache for this.

Link to comment
8 hours ago, dakipro said:

Thanks @itimpi you perfectly described the issue as I do move files from download (on a cached drive) to media share (not on a cached drive).

Now the question is how to solve this issue and have all my shares on their desired locations?


It would be ok if the mover could perhaps fix this automatically, as this doesn't look like an isolated and unlikely scenario to me (it would be even better if this doesn't occur in the first place, but I understand that technology has its limitations). This basically means that if one is using the cache drive, data can end up all over the place and some part of it will not be parity protected.

 

Should this perhaps be elevated to the unraid team via some ticket or something?

This is not something that Limetech can do anything about as it is behaviour built into Linux.    The simple solution is to set the media share to Use Cache=Yes.   Then mover will transfer the files to the array when it runs.    Is there any reason that this will not meet your needs?

 

The values for the Use Cache setting can be little confusing if you have not turned on the Help in the GUI to get a better description of how they work and affect mover.  

Link to comment

Yet another possibility, assuming you are working on the server with the /mnt/... paths.

 

A move from /mnt/user/some-cache-only-share to /mnt/user0/some-cache-no-share appears to do the correct thing, as I would expect. I just tested it in Midnight Commander.

 

/mnt/user is the user shares including cache. /mnt/user0 is the user shares excluding cache. Linux sees /mnt/user and /mnt/user0 as different mounts so doesn't try to rename the path and actually moves the file from cache to array.

 

  • Thanks 1
Link to comment

Turning the cache for this share from no to yes is a valid option, and I will certainly do that, thanks. I will also consider moving data manually just in case (I did this previous time I noticed this)

 

However I do not agree that it is "just a linux thing, nothing to do about it ---> your problem for using linux" as unraid is not just a script I downloaded from the internet and installed on my server. It is a complete system that Limetech (only?) has full control over and knows its internal workings. Unraid is advertised on its home page as "Unraid is an operating system for personal and small business use..." so it is not just a script, but an operating system which I should not know how works internally. It also says here https://unraid.net/product/data-storage-users "Stop worrying about losing your data due to storage device failure. Unraid saves your data even if one device goes bad." Is my data in this scenario protected due to device failure, even though I have it all configured as unraid suggests? 

I could have understood that argument "that is how it is" if I downloaded freenas and installed it on my own, and thus have to know its internals to maintain it properly.

 

This is for me pretty serious issue as my next scenario was to have a "backup-network-share" that is cache only and network shared where all devices would store backup, then to have docker script that would move all that to "backup-non-network-share" that should not be cached nor shared on network as a protection against malware/ransomware.

Now if I am not mistaken this is exactly the scenario where I would have thought that my data is protected, but it is actually not because "Linux is free to decide whether to just rename its path."

 

Again, I understand that it is linux thing, and that it is how it works and that is how the world functions, but I do not agree that it should be ignored by unraid team, even if that means disabling option to not use cache drive if one is present, or by moving the data with the mover even though it is set to No, or at least giving one big red rectangle saying that data might en up somewhere unprotected "because linux". Or perhaps check for this scenario when running parity check, at least giving a link to some page describing the issue, or do anything about it, something... It seems impossible to me that this is a edge case scenario and that it should be that easily accepted as "everyone should know about it, and that it is how it work - linux."

 

I do not feel like I should have to install some community plugin that should protect me from linux itself. What if I didn't install it, I would live in the illusion that my data is protected on the unraid server. (btw thanks for community apps, they are awesome and a main reason I purchased unraid in the first place)

 

Bottom line is that I installed everything as unraid would want me, a parity drive, a data drive and a cache drive. No custom tweaking, no hacking, all by the book. 

And I just moved files from one share to another, for me a real (common) world scenario. 

 

Maybe I am under illusion that data is protected with unraid, as I do not see what I have done wrong here to compromise my data?

  • Upvote 1
Link to comment

Cache pool is something I will certainly implement. But it goes around the issue itself, one should not expect to have data accidentally stored on the cache. 

I did turn on the cached option for the share, and mover did move files as expected on final destination. 

 

However, I have just also checked the help about the cache option, and it says:

"Specify whether new files and directories written on the share can be written onto the Cache disk/pool if present.

No - prohibits new files and subdirectories from being written onto the Cache disk/pool."

 

I would really like to hear what (other?) people from the limetech think about this so I will fill a bug report, my quick search on the forum didn't give relevant result for the issue. 

Link to comment
1 hour ago, dakipro said:

Cache pool is something I will certainly implement. But it goes around the issue itself, one should not expect to have data accidentally stored on the cache. 

I did turn on the cached option for the share, and mover did move files as expected on final destination. 

 

However, I have just also checked the help about the cache option, and it says:

"Specify whether new files and directories written on the share can be written onto the Cache disk/pool if present.

No - prohibits new files and subdirectories from being written onto the Cache disk/pool."

 

I would really like to hear what (other?) people from the limetech think about this so I will fill a bug report, my quick search on the forum didn't give relevant result for the issue. 

I suspect that you were not writing to the share (which would be done via the network), but instead had something running locally that by-passed the network shares and went in at the Linux level.   If you were working at the metwork Level it is definitely a bug.   If it was being done locally (as is often the case with plugins or dockers), then the application needs to handle this case correctly as a copy/delete operation).

Link to comment

That is what I would have expected, if it was some docker doing things directly then it is a my configuration/user error. In this case I am 99% sure that it is done over the network.

 

I downloaded some music files from my online backup using jDownloader inside the docker, where this container has only these two folders mapped /mnt/user/appdata/JDownloader2 and /mnt/user/downloads/completed/

Then on the network, on my windows pc I used CUETools to split files (windows software operating over network) 

I have folders organized in steps: 1toSplit, 2toTag and 3tagged folders, and all problematic files were in the "3tagged" folder, which is where software moves them after splitting. From there MusicBrainzPicard will move them to some other folder under same media share. So it is either CueTools or MusicBrainzPicard that puts files in the 3tagged folder. I never do that manually, especially not on the server itself. 

Network shares are "downloads" with using cache set to "Yes", and "media" with cache set to "No"  (at the moment I changed it to yes, in order to have this working correctly).

 

Only one docker has access to all is /mnt and that is krusader. And I was thinking about it, even if I did move files from /mnt/user/downloads to wrong place like /mnt/user0/media, or perhaps /mnt/disk1/media they would not end up on the cached drive (maybe parity would be broken, but files would not be on the cache).

Only way to I could have done this by mistake would be to use krusader and move each individual file to /mnt/cache/media/music/flac/!NEW/3tagged folder, and in the process of doing that I would have noticed that all these folders were missing and I would have to create them manually "media", "music", "flac", "!NEW", "3tagged". And being a developer myself I would certainly notice so much manual labor and missing folders (that is why I split workflow by folders and I use automated tools, to avoid doing things manually). So pretty sure I would not make such  mistake.

I could have moved folders from download to media/music/flac/!new/1toSplit with krusader, but not on the cache drive, and then certainly not to the 3tagged folder. Perhaps I've done some other step manually somewhere, that at the end got me to this confused state.

 

As I think about it now, I did notice that some files took longer to be tagged and moved to another share, while some of the files were moved instantly. But I thought that has to do with actual file format, as some flac files are not compressed so writing to them is instant (or something like that).

 

And then at the end I have plex in docker who is only binded to /mnt/user/media/ , /transcode/tmp and /mnt/user/appdata/binhex-plexpass and it is seeing files properly as they all appear in /mnt/user/media just fine (whilst some files are actually on the cache drive)

Link to comment
5 minutes ago, dakipro said:

move files from /mnt/user/downloads to wrong place like /mnt/user0/media, or perhaps /mnt/disk1/media they would not end up on the cached drive (maybe parity would be broken

Not a particularly pertinent comment from me, but this would not break parity.

 

 

Link to comment

I just tested this over the network and when I move a file from a cache-only share to a cache-no share, it does a copy from source to destination then delete from source, with the file ending up on the correct disk according to the user share settings. Which is what you would expect since the client machine expects these are different places and it wouldn't be possible to just "rename".

Link to comment

Thanks @trurl , since you have such drives set up, could you test similar thing but using some local (docker?) tool on the server itself?

 

I am thinking now if, even before started moving files on windows over the network, I might have moved them using krusader and somehow they ended up on the cache drive instead (your hypotheses that this might be linux thing of renaming files, when they are moved localy?)

 

Then it would look like that they were, from the network point of view, in the no-cache-media-share but they were actually already on the cache drive, by mistake done earlier in the process?

Link to comment
6 hours ago, dakipro said:

that is how things should work I suppose

I am not really going to argue this point one way or the other. This behavior is understandably surprising to the user, and same goes for the User Share Copy Bug. But both of these problems have very low level origins in Linux that may not get addressed soon if ever.

 

There is a feature request to get a builtin file manager for the webUI. I don't expect much to happen on that front either, but I chimed in on that thread and mentioned these 2 surprises as something that might be considered if such were ever to be developed.

 

Just keep them in mind and work around them. Most people never complain about them. Probably a lot of people never do anything that would make them happen. Probably some that don't complain just aren't aware that it is happening.

Link to comment
  • 3 weeks later...
On 3/3/2019 at 3:10 PM, itimpi said:

One of the idiosyncrasies of the Linux ‘move’ operation is that Linux can first try a ‘rename’ (which is fast) and only if that fails does it do a copy/delete action.

This is not correct.  If source and destination are both in the same physical filesystem, 'mv' simply does a rename, otherwise it does a copy followed by remove.

 

'shfs' is not a physical file system, it's a union.

 

It's hard for me tell what the issue is here, can someone post a 'tldr'?

Link to comment
21 minutes ago, limetech said:

It's hard for me tell what the issue is here, can someone post a 'tldr'?

Imagine this

 

/mnt/user/share1/file1.txt  (this is a cache-only share)

 

From the command prompt, move the file to another share that is set to not use the cache.

 

mv /mnt/user/share1/file.txt /mnt/user/share2/file.txt

 

The file will remain on the cache drive in the share2 folder, in apparent violation of the use cache setting.  IMO, not a big deal as you're basically bypassing the system, but this can happen on occasion depending upon how you've set up docker containers (in particular if for simplicity sake you simply map /mnt to /mnt and nothing else).

 

The "problem" here is that because share2 is set to not use the cache drive, mover will not move the file off of the cache drive to the array.  Unfortunately though @jonp did come up with a neat little use case to leverage this behaviour of mover here which would preclude changing mover to fix this (although, I personally think that mover should always move files to fix this problem, and @jonp be damned ;) )

 

Edited by Squid
Link to comment

Now I am experiencing additional problem with this. Both shares are now set to use cache Yes, but the mover doesn't move the files from cache drive to main drive, thus leaving the file off the parity unprotected, and almost impossible for me to find it out unless I look for it manually (and expect it)

 

Here is short summary:

My system has one cache ssd, one 4tb parity drive and one main 4tb drive.

I have two shares, one downloads, second called media.  Now both are set to use cache disk Yes (after advised on this topic).

I have a krusader docker with this mounted point   /media/   ->  mnt    (this is how I saw it on spaceinvadersone video)

Then I ripped dvd on to the downloads share from my pc, and tried to move it from /root/media/user/downloads/   to /root/media/user/media/movies   using krusader (so this is how krusader maps /mnt/media/user/downloads/ and /mnt/media/user/media/movies). I am very very sure I used exactly these folders after having this issue. 

 

now on the media share I do see my movie just fine.

But if I open root/media/cache/movies from krusader I still see my movie folder there, with files in it. 

if I open root/media/disk1/media/movies/ I do see the movie folder, but NO files in it. 

Mover is set to run every night, but even manually invoking it does not move the files. 

 

I first thought that this might do something with plugin/setting that makes disk sleep, but I now tried multiple times to invoke mover, once it said "moving" and I am checking now from Krusader, file is still on cache (for three days)

 

 

Link to comment
4 minutes ago, dakipro said:

Mover is set to run every night, but even manually invoking it does not move the files. 

Enabling Mover Logging within Settings - Scheduler - Mover Settings then attempting to run mover and finally posting your diagnostics would say why.  But, it has to be noted that if the file is in use it cannot be moved.

Link to comment

Thanks, that did the trick. I first didn't got much in the logs


Mar 22 20:30:03 hpenas emhttpd: req (17): shareMoverSchedule=40+3+*+*+*&shareMoverLogging=yes&cmdStartMover=Move+now&csrf_token=****************
Mar 22 20:30:03 hpenas emhttpd: shcmd (63101): /usr/local/sbin/mover |& logger &
Mar 22 20:30:03 hpenas root: mover: started
Mar 22 20:30:03 hpenas move: move: skip /mnt/cache/media/movies/movie.mp4
Mar 22 20:30:03 hpenas root: mover: finished

 

but then I stopped all the containers and now mover did move the file properly.

I am not sure what could have kept the file in use but that is would resolve itself once containers are restarted I guess/hope. 

 

Could it be that initial issue was because of similar scenario of file being in use?

Link to comment
48 minutes ago, Squid said:

Imagine this

 

/mnt/user/share1/file1.txt  (this is a cache-only share)

 

From the command prompt, move the file to another share that is set to not use the cache.

 


mv /mnt/user/share1/file.txt /mnt/user/share2/file.txt

 

The file will remain on the cache drive in the share2 folder, in apparent violation of the use cache setting.  IMO, not a big deal as you're basically bypassing the system, but this can happen on occasion depending upon how you've set up docker containers (in particular if for simplicity sake you simply map /mnt to /mnt and nothing else).

 

The "problem" here is that because share2 is set to not use the cache drive, mover will not move the file off of the cache drive to the array. 

Yes this is an unfortunate side-effect of how 'mv' command is coded and why 'mover' was written.  The fix for this would result in the 'mv' operation failing in this particular case, which is probably not what we want either.

 

Interesting problem which doesn't come up when moving stuff around via SMB.  What is the thing that is executing the 'mv' command in OP's issue?

Link to comment

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.

Guest
Reply to this topic...

×   Pasted as rich text.   Restore formatting

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.