Cache setting - duplicate

[Jan Hoker]

Hey,
I'm not sure if this isn't more feature request. I would like to have option to write and load data from cache but also have them in array in case of backup is needed. For example if I would work on video editing over 10gig ethernet on unRAID server I want to read and write into cache pool. Right now this is possible if I choose Prefer or Only option, but I also want mover every hour/day/week depend on my mover setup, to take these data and copy them on array just for case of cache disk failure. Is this possible in current version of unraid or I should create change request and hope?

[JonathanM]

This is not something that will ever be done, for multiple reasons. The biggest reason is if you make a copy instead of moving, then it's no longer clear which copy should be presented in the user share.

 

You can accomplish what you are trying to do with a user script that periodically copies from your active share to a backup share.

rsync -arv /mnt/user/active/ /mnt/user/backup

or something similar. Set the "backup" share to cache: yes, that way the copy will be done on the cache drive, and the mover will put it on the array.

[Energen]

 

Previous discussion.

 

[Jan Hoker]

21 hours ago, jonathanm said:

This is not something that will ever be done, for multiple reasons. The biggest reason is if you make a copy instead of moving, then it's no longer clear which copy should be presented in the user share.

 

You can accomplish what you are trying to do with a user script that periodically copies from your active share to a backup share.

rsync -arv /mnt/user/active/ /mnt/user/backup

or something similar. Set the "backup" share to cache: yes, that way the copy will be done on the cache drive, and the mover will put it on the array.

Well as SW developer I don't think this is issue. Everything what is in cache is up-to-date, you use for read/write only cached data but you also have these data copied in array and when you need to create more space in cache you will automatically move data which wasn't used for longest time to array to be sure they are up-to-date. In current cache setup when you want boost read speeds with cache and also want to have protected data, you have no option. You can use "prefer" setting but then you have unprotected data, mostly on your cache drive and only when you need more space you are moving some parts to array. Basically I want same function but expanded with backup data in array (which are not used for reading if they are in cache too). Perfect option would be independent on mover scheduler setting for creating backup image every hour for example. At that moment would be risk of lossing data maximum hour and that is fine for most of people. 

[Energen]

You can make your own script like jonathanm said or you can use CA Appdata Backup / Restore to back up the entire /cache mount.  Or you could use any backup programs in docker to make a copy of the /cache drive. 

[JonathanM]

Or, since mover is just a fancy script in itself, there is nothing keeping you from implementing your own script that does exactly what you want since you are a software developer.

 

It's really not that complicated, the worst part is going to be ensuring that the clients see and work with only the latest copy, and not overwriting a more recent version with your script. All the data can be accessed directly from each disk, through /mnt/diskX and /mnt/cache, the user shares are simply the merged content of the root folders on all the disks.

 

The reason it's not been done so far is that there are many other ways to accomplish what you are asking instead of breaking the current behaviour of mover which assumes there are only uniquely named files in a specific path on each disk.

 

Backup routines are plentiful and well thought out, like I said earlier you can copy the data to a second location on a schedule with a one line script.

 

If you are dead set on implementing this exactly as you suggested, you are certainly welcome to experiment with it, you won't "break" Unraid, but you might get some unexpected results. Unraid is extremely flexible, you can do all sorts of things that aren't officially supported. Have a play and see what happens when you put the same file name on a second disk with the same path. Like /mnt/disk1/test/testing.txt and /mnt/cache/test/testing.txt. When you access /mnt/user/test/testing.txt, only one of the files will be there. If you populate the file with content when you create it like "this file is on disk1" and "this file is on cache", you can see what's happening. If you delete the file shown in /mnt/user/test, the second file will magically take its place, making it appear that the deletion didn't work. You can also use this "feature" to hide files you don't want shown in a user share, simply by manually creating a file with the same same on a higher priority disk. Keep in mind that Unraid sees this as an error condition, so don't expect the existing mover script to play well with it, you will need to make your own mover script.

[ronia]

Sorry to necro this thread, but it was the first topic that came up when I googled "what happens if unraid has two copies of the same file in cache and array".

 

In response to JonathanM, the behavior is like this:

 

1. Create a test.txt in /mnt/cache with contents "this is on cache"

2. Create a test.txt in /mnt/disk1 with contents "this is on disk1"

3. Create a test.txt in /mnt/disk2 with contents "this is on disk2"

 

Since I was curious about what would happen (and also a little scared about violating cache coherency), this was the result:

 

1. cat /mnt/user/test.txt -> "this is on cache"

2. delete /mnt/cache/test.txt

3. cat /mnt/user/test.txt -> "this is on disk1"

4. delete /mnt/disk1/test.txt

5. cat /mnt/user/test.txt -> this is on disk2"

 

So it seems like the cache copy is what will be visible to the /mnt/user/ and then each disk's copy is ranked in order of priority based on some kind of inherent priority.  Also, I created the cache copy first.  If the priority was based on creation/modification date, then it would have displayed disk2's copy as it was technically "newest".  This was not the case, so I think you can rule out atime/ctime/mtime.

 

I think this behavior is extremely useful if you're trying to use the cache pool as a real cache and implementing background scripts to intelligently move files between cache and array.  

 

Context: I have stumbled across this post while trying to create my own script that rsyncs the contents of an entire tvshow/season from cache to array if it is currently being watched.  Odds are, if someone watches episode 1, there is a good chance they will watch episode 2 and so on.  My original implementation was to rsync the array copy to the cache, touch it and then delete the array copy (cache coherency) and let the mover move it back after it becomes old again.

 

But actually, now I think I will simply rsync the array copy to cache and leave the array copy.  I will write ANOTHER script to handle "old cache copies" when the cache becomes full.  The benefits of this are:

 

1. The array disk is only written once.

2. If a user watches episodes 1 and 2 and then watches episodes 3 and 4 the next day, the array disk is spun up twice.  If all episodes are copied to the cache, the disk is spun up once to read the entire season once, but then spun down and left untouched regardless of rewatching episodes, watching only partial episodes at a time, etc. 

Edited March 24 by ronia

[JonathanM]

5 hours ago, ronia said:

In response to JonathanM, the behavior is like this:

I'm well aware of the behaviour, I was formulating an easy way to demonstrate it for someone who wanted to discover how it worked, sounds like you've got it figured out.

 

Just be sure you keep in mind the write behaviour when using the user share, so you don't accidentally overwrite a newer version of something with an older version, and lose your changes when you "clean up"

[trurl]

For completeness, you would also want to determine which file was "visible" when it existed on more pools than just the one named "cache".