Caught in a cache trap - mover can't move files out of cache

[CaptainTivo]

For a several months, I have had a problem such that my cache has some files that won't move to the array.  I keep putting off doing the work to fix this but I think I really need to fix it now.

 

When I run the mover, I get these types of messages:

Mar 25 10:59:58 Tower shfs: copy_file: /mnt/cache/Videos/TV Shows/The Untouchables/Season 2/The Untouchables - S02E25.m4v /mnt/disk8/Videos/TV Shows/The Untouchables/Season 2/The Untouchables - S02E25.m4v (28) No space left on device

 

I think this is because I set the "Videos" share to "Automatically split only the top three directory levels" and "Season 2" is at the third level. 

(complete description of my directory structure is at the bottom of this post)

 

I think  what happened in that I created a "Season 2" directory in one of the TV shows (The Untouchables) and copied a couple of episodes into it.  Much later, I added more episodes but by that time the disk onto which the "Season 2" directory was created (at level 3) was full.  So, when I copied the files onto the share, the ended up in the cache, but could not be moved to the disk directory because there was not enough free space. The disk is disk8 and it currently has 49.6GB free but the "Minimum free space" for that share is 45GB and the total size of the stranded files (on the cache) is 14.6 GB.  

 

The first thing I tried was to lower the "Minimum free space" number for that share to from 45GB to 8GB, since it then has 49.6-8 = 41.6 free, that should be enough free space to allow the mover to move the files to disk8.  But it still fails with the "No space left on device" error.  I don't understand this.

 

Next, I thought of using "Unbalance" to move the files off of disk8 to a disk with lots of free space.  BUT, there is a caveat to using Unbalance that it needs exclusive access and the cache should be disabled.

What if I did this:

1. open a terminal

2. copy the "Season 2" directory and its contents to a "holding" place, say Season_2_copy again at level 3.  This should be allocated on a different disk with more room, right?  Once that is done, I could simply delete the old "Season 2" directory and rename the copy to "Season 2".  This way the Kodi database will have the same file paths to this season.

 

If you have any other ideas, please let me know.

For reference, here is my current directory structure.  Going forward, I think I can avoid this situation by copying all the episode files into a season directory at the same time.  I think this problem happened because I copied a few files into it and that triggered the "minimum space" rule and then I couldn't add any more files.

 

========================================================

My directory tree looks like this:

Videos

    ------Movies

    ----------DVD

    -------------"A Beautiful Mind (2001)"

    -------------------VIDEO_TS

     -----------------------VIDEO_TS.IFO

     -----------------------VTS_01_0.VOB

.

.

.

    ----------Bluray

    -----TV Shows

    ----------The Untouchables

    ---------------Season 1

    --------------------The Untouchables - S02E01.m4v

    ---------------Season 2

etc.

Each DVD or Bluray is a direct rip (ie. a directory containing My original reasoning for setting the split level to 3 was to insure that my DVD/Bluray rips for each movie would be all on one disk.  If the disk failed, I would only have to re-rip those movies.  For TV Shows, I started out "flat" that is, all episode as single video files in the "named" directory.  Later, I decided it would be better to have "Seaon 1" etc directories in the main directory and the episode video files in the "Season x" subdirectories.

 

 

 

tower-diagnostics-20210325-1405.zip

[JorgeB]

Split level always overrides allocation settings, you need to either lower the split level or delete/move the existing folder(s).

[CaptainTivo]

7 minutes ago, JorgeB said:

Split level always overrides allocation settings, you need to either lower the split level or delete/move the existing folder(s).

I don't understand that.  If originally the directory was allocated onto disk8 because the split level dictated all files should be on that disk and the mover can't move the files from the cache to the disk because the free space on the disk minus the size of the files is less than the minimum space required to be left, then lowering the minimum space required should allow them to be moved, right?   Why is mover getting a "No space left on device" when there is now space enough to contain the files left on the cache?

[JorgeB]

12 hours ago, CaptainTivo said:

The disk is disk8 and it currently has 49.6GB free

Diags show a different thing:

 

Filesystem      Size  Used Avail Use% Mounted on
/dev/md8        3.7T  3.7T   48M 100% /mnt/disk8

 

[CaptainTivo]

3 hours ago, JorgeB said:

Diags show a different thing:

 

Filesystem      Size  Used Avail Use% Mounted on
/dev/md8        3.7T  3.7T   48M 100% /mnt/disk8

 

Doh!  That explains it. I mis-read the free amount on the main page.  Although how it got down to 48M when the minimum free space was set to 48GB is a mystery.

I think I will simply copy some data off disk8 to some other disk, then delete it from disk8 until there is enough free space to contain the files stuck on the cache.  Then run the mover.  That should work, right?

Here is a question , though.  I had never thought about it, but I assumed that when you write a directory to a disk at the split level (in this case split level is set to 3 and I am writing to a directory at level 3), that it would somehow know the size of ALL the data that I am trying to write to that directory and prevent it from being allocated if that would exceed the minimum free space requirement.  But HOW does it do that?  All it really knows is how large each file you are writing is.  You should be able to write files up until the minimum is going to be exceeded and then it would not write the next file.  BUT that would violate the no-split rule.  I'm confused.  Can someone explain, please?

 

Thanks.

 

[JorgeB]

10 minutes ago, CaptainTivo said:

I think I will simply copy some data off disk8 to some other disk, then delete it from disk8 until there is enough free space to contain the files stuck on the cache.  Then run the mover.  That should work, right?

Yes.

 

10 minutes ago, CaptainTivo said:

Here is a question , though.  I had never thought about it, but I assumed that when you write a directory to a disk at the split level (in this case split level is set to 3 and I am writing to a directory at level 3), that it would somehow know the size of ALL the data that I am trying to write to that directory and prevent it from being allocated if that would exceed the minimum free space requirement.  But HOW does it do that?  All it really knows is how large each file you are writing is.  You should be able to write files up until the minimum is going to be exceeded and then it would not write the next file.  BUT that would violate the no-split rule.  I'm confused.  Can someone explain, please?

It's by file, it you have 4 x 20GB files and the minium free space is set to 50GB it will successfully copy the 1st two and then you'll get a not enough space error, if the split level set requires the same disk.

[CaptainTivo]

4 hours ago, JorgeB said:

Yes.

 

It's by file, it you have 4 x 20GB files and the minium free space is set to 50GB it will successfully copy the 1st two and then you'll get a not enough space error, if the split level set requires the same disk.

Interesting.  I have been running Unraid since 2012 and I have never seen this error.  How is the error displayed to the user? Does it just silently error and refuse to copy the file or is there some feedback?

 

As I explained in my original post, the level 3 split was meant to prevent part of a DVD rip (containing a VIDEO_TS directory and the various files that make up a DVD) from being split across multiple disks.  Basically, I just rip the DVD into a folder and copy the movie folder to the Videos share.  If what you say is true, Unraid would simply check to see if it could copy each file (the VTS_XX.VOB files that make up a DVD are never larger than 1GB) without violating the minimum space requirement and then copy it to a disk in the share (per the allocation rules - in my case High Water).  This would virtually guarantee that one of the files copied would exceed the minimum space requirement and it would stop copying files.  I would think this would happen every time I attempted to copy a DVD folder and a disk happened to "fill up" while copying the files, but this has NEVER happened to me.  I simply don't understand how this is supposed to work and yet it seems to have worked just fine for 9 years!  One caveat, all of my work is done from a Windows machine with the Unraid shares mounted via SMB.  That is, I just drag or copy a DVD movie folder into my Videos\Movies\DVD folder. This has always worked and I have never had an error.

 

In fact, now that I think about it, what is point of the split level at all if you can't guarantee that all the files in a directory which is to be copied to a share at that split level will be on a single disk?  The only way I can see this working is if, when the minimum space requirement was exceeded, it "backed out" all the previous copy operations and allocated the folder onto another disk.

 

Can someone please explain the logic to me?  Thanks.

 

[JorgeB]

11 minutes ago, CaptainTivo said:

Does it just silently error and refuse to copy the file or is there some feedback?

If you're copying manually you'll get an IO error, if it's the mover it will fail to move those files and log it in the syslog, if mover logging is enable.

[trurl]

5 hours ago, CaptainTivo said:

how it got down to 48M when the minimum free space was set to 48GB is a mystery

Minimum free doesn't mean don't allow the disk to get less free space than that. It means IF the disk has less space than that, choose another disk.

 

Unraid has no way to know how large a file will become when it chooses a disk for it.

 

For example, if Minimum is 20G and the disk has 25G free, it can be chosen, whether the file you are writing is 15G (succeeds) or 30G (fails).

[CaptainTivo]

7 minutes ago, trurl said:

Minimum free doesn't mean don't allow the disk to get less free space than that. It means IF the disk has less space than that, choose another disk.

 

Unraid has no way to know how large a file will become when it chooses a disk for it.

 

For example, if Minimum is 20G and the disk has 25G free, it can be chosen, whether the file you are writing is 15G (succeeds) or 30G (fails).

This makes sense. BUT, what happens when it is writing a file and it runs out of space on that disk (in your words, Fails)?  I assume it simply chooses another disk and then tries again?  Otherwise, how is the user informed that the write failed?

[trurl]

10 minutes ago, CaptainTivo said:

I assume it simply chooses another disk and then tries again?  Otherwise, how is the user informed that the write failed?

No, it just fails. Up to the application writing the file to inform you of that.

[ChatNoir]

That's is why it is generally suggested to set Minimum free space over the expected maximum file size for this share. As stated in the tooltip.