Cache drive filling up, mover not helping

[sheldon]

Hi Guys,

 

I've been using Unraid for almost 1 year now and loving it. At first it was a bit scary (being a windows guy, running Linux for all my important stuff) and not knowing how to handle a problem when/if it shows up. However by now I feel fairly confident I can handle a good chunk of things on my system. 
BUT there is something I recently came across where I could use some advice:

 

I have a 500GB M2.SSD for my cache drive and for the first time I accidentally filled it up without realizing it (at first). This of course caused all kinds of crazy things to happen. The first thing I noticed was several of my dockers crashed and could not be restarted, then I saw lots of errors in the Unraid system log and realized the only real fix is a reboot (after freeing up some space on the cash drive). After a quick reboot and everything was good again I started poking around what caused my cache drive to fill up in the first place. I found out the problem was partially Midnight-commander or my use of it. When "moving" files around from my share folder running solely on my SSD to any other share that runs exclusively on the array (excluding cache drive use) it doesn't actually move the files. It simply creates a shortcut (there is probably a better Linux acronym for this but coming from Windows that's what it looks like to me) on the array but leaves the "meat" on the SSD cache drive. SO after doing this for a while and not knowing what I unintentionally  did I filled up the SSD yesterday while moving a 100gb file.

 

So after all this here the question: Is there a better tool than Midnight-commander to move files on my unraid without involving my windows system (which takes a lot longer for larger files) or is there an application like the mover that will see those left over files and move them?

I noticed that the "fix common problems" app actually warned me about files on my cache for which there was no official share for but I didn't quite realize what that meant until yesterday. So Unraid/fix common problems did see the problem creeping up but I didn't know what to do with the information until it was too late. However I'm thinking now if unraid/fix common problems can see it happening (slowly or not) isn't there some app or function that will tackle this so I don't have to manually move files from my cache mount to my disk mount? Again the problem was that I moved files from my shared folders not from the actual disk mounts.

 

[trurl]

Some of your description doesn't exactly make sense, but let's skip that for now. What exactly are you trying to accomplish with Midnight Commander? If you are trying to move things from cache to array or from array to cache, why aren't you just letting mover do it for you? There are user share settings that you can manipulate to get this straightened out with mover.

 

Are you trying to move from cache to a user share or from a user share to cache? Cache itself is actually part of the user shares, and you must not mix disks (cache is a disk) with user shares when moving or copying files or you could actually lose data. You should only move user share to user share, or disk to disk.

 

I have to leave for a couple of hours but will check back later.

[trurl]

One last thing before I go.

 

Go to Tools Diagnostics and attach the complete diagnostics zip to your next post.

[sheldon]

Sorry I probably didn't explain this correctly. Bear with me while I try to make sense out of this 

Here some background information and maybe things will be clearer:

 

I have an application/docker that downloads files from the internet. These files are stored on a folder/share that uses only the cache drive, not the array (therefore mover won't touch these files). The reason I need to set it up this way is because of the drive performance. Files stored under the array cannot keep up with my gig fiber connection but files on my cache drive can. So as files are downloaded and when I no longer require them to be accessible on the internet (which can be weeks) I move them manually onto the array for long term storing and where performance is no longer an issue. That's when I use MC to move the files from the shared folder that's marked to use SSD/cache drive only to a shared folder that is marked to use the spinning drives/array only (no SSD).

I tried this over SMB/windows (copy/paste/move) for a while but the performance is sub-par while using MC is much faster as I don't need to involve my workstation as a middle man. 

However as described above MC actually doesn't really move the file(s) it only places "shortcuts" (for a lack of a better word) on the array while leaving the real file on the cache disk. 

 

In terms of how the file structure looks like after I "move" a file via MC:

* Here is where the file shows up (from a global perspective):

/mnt/user/download/file1.mov

* However when looking at where the file has been physically stored it actually is here:

/mnt/cache/download/file1.mov

* Where I was hoping to find the file was here:

/mnt/diskx/download/file1.mov

 

 

tower-diagnostics-20181127-1617.zip

Edited November 27, 2018 by sheldon

[itimpi]

You are running into a ‘quirk’ of the way linux works in that under some circumstances where it thinks source and target share a common mount Point a a ‘copy’ends up being implemented as a rename (which.is fast) thus leaving the files on the same drive.

 

the easiest solution would be to change the shares you want to end up on the array to Use Cache=Yes.     This will cause mover to move the files for such shares from cache to array when it runs.

[sheldon]

That makes sense. I could have thought of that myself  

When I first created it I was afraid of using cache for that folder as I have several very large files and I figured I might manage to fill it up before the mover invokes. However I obviously managed to do that even without it - lol.

 

Sounds like a reasonable solution - I'll do just that. Thanks!

 

[trurl]

Your explanation makes more sense now and you have gotten good advice. I just want to add a some nuance to this bit:

17 minutes ago, sheldon said:

I was afraid of using cache for that folder as I have several very large files and I figured I might manage to fill it up before the mover invokes

A share that is set to cache-yes will overflow to the array when Unraid sees that cache is too full. But, how Unraid decides that cache is too full depends on the setting for Minimum Free in Global Share Settings.

 

Unraid has no way to know how large a file will become when it decides which disk to write it to. If a disk has more than Minimum Free remaining, Unraid can choose the disk, and if the file is too large you will get an error for out-of-space. But, if Unraid sees the disk has less than Minimum Free, it will choose another disk to begin writing the file.

 

Notice that description of Minimum Free talked about disks in general, not just cache. In addition to the Minimum Free setting for cache in Global Share Settings, each user share also has a Minimum Free setting that works similarly. When writing to a user share, if a disk has less than Minimum Free remaining, Unraid will choose a different disk.

 

You should set Minimum Free for cache and for each user share to be larger than the largest file you expect to write to that destination. Then Unraid will choose another disk instead of erroring due to out-of-space.

[trurl]

1 hour ago, sheldon said:

In terms of how the file structure looks like after I "move" a file via MC:

* Here is where the file shows up (from a global perspective):

/mnt/user/download/file1.mov

* However when looking at where the file has been physically stored it actually is here:

/mnt/cache/download/file1.mov

* Where I was hoping to find the file was here:

/mnt/diskx/download/file1.mov

It was your talk of "shortcuts" that had me confused at first, because I knew it wasn't creating hard or soft links. itimpi got to the bottom of it though.

[sheldon]

Well I did warn everybody that I'm a windows guy so my point of view is a bit different than unraid reality  .. But it does sound we are all on the same page now. Thanks again for the follow up and the additional information.