I've tried searching under a few potential subjects, but I can't find what I'm looking for.

 

I'd really like my cache to behave *purely* as temporary storage for transfers (at least for some shares). I'd like to be able to dump my backups from my network quickly (cache speed), but I have ZERO interest in keeping it on cache - I just love my 2.5Gbps transfers.

 

I'm presently using the "Mover Tuning" plugin (which is good), but even then, I can only fudge the type of behavior I'd like and it barely works. 

 

There are a few shares (Documents, Photos, CAD Projects) where I like the behavior of keeping recent files in cache. So that's all good.

 

I feel like what I want was always possible in the past (use cache, but prefer array - or whatever), and now it's gone. Can anyone help? Am I just being particularly low-IQ on this?

Edited September 6, 20241 yr by FlexGunship

Caching behaviour has not changed for a long time now - it is just that some of the terminology changed as described in the section on User Shares in the online documentation accessible via the Manual link at the bottom of the Unraid GUI.   

 

For what you describe I think you want the 'cache' pool as Primary storage, the array as secondary storage and Mover Direction as cache->Array.

18 minutes ago, itimpi said:

Caching behaviour has not changed for a long time now - it is just that some of the terminology changed as described in the section on User Shares in the online documentation accessible via the Manual link at the bottom of the Unraid GUI.   

 

For what you describe I think you want the 'cache' pool as Primary storage, the array as secondary storage and Mover Direction as cache->Array.

Thank you for the response. That's precisely what I have for the shares of interest... but mover looks at every single file and moves nothing. Presumably because they are new. I have a 1TB cache (2x 1TB nvme drives). During heavy traffic, my cache will fill up, and it will stay full for days. Invoking the mover gives me a string of "already exists on the array" and no reduction in cache utilization. 

 

The mover basically runs 24/7 and my cache stays 50%+ full.

 

I have a folder that I copied from another system called "Backup" which is 130GB. I haven't touched it and its presence on the cache drive hasn't diminished in over 24hr. I've been running mover all day and it's just chilling there. Is there a way to change that behavior?

artemis-diagnostics-20240906-1545.zip

A number of things occur to me:

• The Mover Tuning plugin may be causing issues.    You might want to disable and see what happens.
• if a file already exists on the array then mover will not overwrite it with a copy from cache.   In such a case you have to manually decide which copy to keep and delete the other one.
• make sure you have set a Minimum Free Space value for the cache pool to stop it filling up too much.

Interesting. During the SMB transfer operation, my assumption (naive) would be that it would comparing the file list against the one on the array - but maybe it's not?

 

File info.dat exists on the array in share Documents (which uses cache).

 

The same file, an unaltered second copy of info.dat, is moved to the unRAID share Documents via SMB transfer.

 

Even though the share has info.dat, it may still move a second copy to the cache since the operation wouldn't generate a conflict?

 

Then the mover can't move info.dat from cache to array because the operation would necessitate an overwrite?

 

Okay. Thanks again. I was going to setup a cron job that uses rsync to do the samw.thijg, but I suspect it would see the file list of cache contents and not the file list of the array then. 

 

Last Q, can the "minimum free space" argument prevent a transfer? Or does it simply trigger the mover when exceeded?

 

And if it's the second one, is there any harm in keeping the minimum free space arbitrarily high? 850GB on a 1TB cache to force the move operation frequently?

14 hours ago, FlexGunship said:

Even though the share has info.dat, it may still move a second copy to the cache since the operation wouldn't generate a conflict?

Unclear what you mean there. User shares are simply a merged view of the root folders in all the participating array disks and pools. If a file exists in multiple locations that are merged, Unraid can't know which file is "correct", so it only shows one of the files in the user share.

 

For example:

/mnt/disk1/myshare/document1.txt

and

/mnt/pool/myshare/document1.txt

are the same location

/mnt/user/myshare/document1.txt

so only one of the copies is actually shown in the user share.

 

If the mover is set to move files from the pool to the array, it can't, because it would overwrite the file on disk1.

 

14 hours ago, FlexGunship said:

Even though the share has info.dat, it may still move a second copy to the cache since the operation wouldn't generate a conflict?

Files that already exist in one location are overwritten rather than making a second copy, to avoid the issue I outlined above. The primary share location is only applied to new files, existing files are read and modified from their current location.

On 9/7/2024 at 9:49 AM, JonathanM said:

Unclear what you mean there. User shares are simply a merged view of the root folders in all the participating array disks and pools. If a file exists in multiple locations that are merged, Unraid can't know which file is "correct", so it only shows one of the files in the user share.

 

For example:

/mnt/disk1/myshare/document1.txt

and

/mnt/pool/myshare/document1.txt

are the same location

/mnt/user/myshare/document1.txt

so only one of the copies is actually shown in the user share.

 

If the mover is set to move files from the pool to the array, it can't, because it would overwrite the file on disk1.

 

Files that already exist in one location are overwritten rather than making a second copy, to avoid the issue I outlined above. The primary share location is only applied to new files, existing files are read and modified from their current location.

Okay, that's clear apart from one item then. Thank you.

IF: /mnt/disk1/myshare/document1.txt exists THEN /mnt/pool/myshare/document1.txt shouldn't, right? It implies the mover already moved the file.

 

I ended up with a copy in disk1 and pool SOMEHOW. They were different revisions of the same file, but same name. I was able to manually resolve it (but, it invovled a lot of careful manual manipulation). 

 

I wonder if I stupidly did a manual rsync against /mnt/cache/ (/mnt/pool/) one time.

 

This also (mostly) resolved my "always moving, but nothing moved" issue and my cache utilization is much lower now.

 

 

Disk shares and User Shares are handled in two different ways.  In fact, files on one are unknown to the other and this creates a bug issue.    See here:

 

     https://forums.unraid.net/bug-reports/prereleases/60-beta6-user-share-copy-bug-r3007/

 

This issue has never been resolved and still exists today.  The fact that it exists can actually be used.  See here:

 

     https://forums.unraid.net/topic/58374-secure-writing-strategy-for-unraid-server-using-write-once-read-many-mode#comment-572532

 

(As a matter of fact, I am using writing files to the cache as a disk share and later moving the files to the array for the exact reason described in the thread.  If there is a file already on the array in a User Share than the write to the User Share will fail and the deleting of the file from the Disk Share will fail.  I use this information to manually fix the problem logically by deciding which file to delete.)

 

As stated above, Mover will not overwrite an existing file on the array.  Could Mover be rewritten to do what you want?  Probably but then it would have to a series of setup switches to allow it to preform in the new manner as well as the present way. 

 

Another approach would be to write a completely new mover script which would do actually what you want.  (I seem to recall that you are not the first person to want to use the cache in this or a very similar way!)  This would probably be preferable to attempting modify the existing script.  You could request that Unraid do this via a request in the Feature Requests subforum or write it yourself.  If you think this might a really great addition and you don't feel you have the programing chops, you might want to open a topic in the Lounge area, have a discussion about needs and requirements, and see if there isn't someone who could do the coding.