Cache: Yes - not for rsync?


Recommended Posts

I have an array with plenty of space on it, and a share called `media` set to `cache: yes`. However, when the cache fills up, my script (using rsync to copy to `/mnt/user/media`) generates a disk full error: 

 

No space left on device (28)

 

Yet cache: yes says

 

If there is insufficient space on the Cache disk/pool, then new files and directories are created on the array

 

Why is this?? Do cache rules not work for rsync / bash scripts?

Edited by banterer
Link to comment

I do not use RSYNC but the result would probably change depending of the command you are using.

You should share an exemple of your command.

 

Did you set up a Minimum free space for your cache Pool ?

This should solve your problem 

image.thumb.png.d12f26c4b5abc82b81a893c5af659f6a.png

 

As a general rule, you should not cache a share if you regularly add more data than the cache pool can hold.

Link to comment

The command is:

 

rsync -a /mnt/disks/main/backups/* /mnt/user/backups/

 

I can't find the minimum free space setting, where is it exactly?

 

Re your point on not adding more data than the cache pool can hold - absolutely. This is just the initial copy, future syncs will be incremental. I'm caught in a situation where none of the cache options 'fit'. If I set it to 'no', it will leave data it's already copied on the cache... so I have to set it to yes! Could do with another option which will move from the cache, but not put any further data onto it....

Link to comment
12 minutes ago, banterer said:

'm caught in a situation where none of the cache options 'fit'. If I set it to 'no', it will leave data it's already copied on the cache... so I have to set it to yes! Could do with another option which will move from the cache, but not put any further data onto it....

Just run the mover and it will move data already copied from cache to the array, then set cache to no and do the rest of the copy. When finished set cache to yes if you want. Also be sure to set your split level to split any dir. you can change it after the initial copy if you want. Split level is important because rsync will create all the directories first before it copies any files, so if the split level is set to anything other than to split any directory on the initial copy you will run into the same issue with the disk filling up.

Link to comment

I have split any directory enabled - settings below.

 

I will set the min space to 100G or something like that, when move has finished and I can stop the array. I still don't understand why a disk full error is being generated though, because surely the min space setting shouldn't matter - if it's X% full, or 100% full, further writes should go to the array according to the help / docs?

 

1889594064_Screenshot2022-06-05at11_59_38.thumb.png.b743f3f868f25a38cb4bef3d56cdc5b5.png

Link to comment
52 minutes ago, banterer said:

I still don't understand why a disk full error is being generated though

It might be because of the way rsync is copying directories/files. As I said in my previous post rsync will create the entire directory structure before copying any files. And thus will most likely try to copy all the files into the already created directories. If you do a normal copy I think you will find the cache setting is working as intended. I haven't tested it tho, as I never use the cache feature. I have cache no set on all my shares except the appdata and VM share which is set to only.

Link to comment
1 hour ago, banterer said:

because surely the min space setting shouldn't matter

It does matter, you must set a minimum free space for the share and should also set one for the pool, the one for the share should be set at about twice the size of the largest file you expect to copy to that share.

Link to comment
45 minutes ago, JorgeB said:

It does matter

And now I remembered WHY it matters just to put that out there as well. :P  It's because unraid does not know how big the file you're going to copy is, it only knows how much space is left on the disk. So if the file is bigger than the space left on the disk it will fail if the minimum free space is not set. If unraid sees that there is less than the minimum free space left it will choose another disk IF the split level permits it. If not it will continue to fill the disk until it runs out of space or files are manually moved to another disk to free up space.

 

Edit: Paraphraseing, unraid do know how big your files are. But when creating a file unraid does not know how big it's going to be before it created. And when copying/moving a file you're essentially making a new file just with the same data.

Edited by strike
Link to comment

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.

Guest
Reply to this topic...

×   Pasted as rich text.   Restore formatting

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.