Multiple small files, but togeter > Cache into one folder

[Autchirion]

Hey Guys,

 

I'm running a nextcloud server where I sometimes get the issue that the cache is running full. This occurs, whenever we are uploading Video Files into one folder. Each file is smaller than the cache, but together they are way bigger and the upload happens before the mover is scheduled.

So I expected as soon the Minimum free Space gets hit, the next file will automatically be written to the array. However, this does not happen, at least not if everything is in one folder or e.g. multiple folders are bigger than the cache together (so each user would upload his files into his own folder, but they start uploading before the cache is "full") . This is causing me quite a headdache, and I'm not sure what to do.

 

So, can't I tell unraid to just move the files to the array from now on, even if it would mean to split up the folders over cache and array? In generall, this is possible, if the mover took place, the folder is on the array and now we are adding some more files to this folder.

Alternatively, can't we invoke the mover for this specific cache as soon as free space < min free space?

Are there any other options to prevent file loss for huge writes to my cache? I mean, the files individually are relatively small, we are recording using 4k video max, so it's not like that one file can exceed the 120GB of cache drive I'm using.

 

Thank you in advance,

Autchi

[itimpi]

Are these uploads happening in parallel?   If so this problem can easily occur as the check for free space only happens at the point the file is first created (when it has 0 size), and in such a case one needs to have the free space for the cache pool to be enough to allow for the number of files that could be uploaded in parallel to fit.

 

Have you set the Minimum Free Space setting for both the cache pool and the User Share in question? 

 

It might be a good idea to  post your system's diagnostics zip file in your next post in this thread to get more informed feedback as it will allow us to see the settings your are using.  It is always a good idea to post this if your question might involve us seeing how you have things set up or to look at recent logs.

[Autchirion]

Thank you for your reply!

Sorry, native languate isn't english, so sometimes I still think I made something clear, but obivously didn't. You are right, this happens if the users start the transfer at the same time. Of course this won't work and I should have thought of this, my bad. I will need to increas the Min Free Space zu users * max filesize obviously to cover this usecase as well.

 

Is there any way to invoke the mover when the min free space is hit? This way I could move all the finished files while the new files are written, so I don't have to increase the min free space to a massive size. I checked mover tuning, but I didn't see an option for this.

 

[edit] learned that move tuning allowed exactly what I want, start moving (per share setting) as soon the drive is about to be full.

Edited January 24 by Autchirion