Long time UnRaid user who has been out of the loop for a few years now. I was wondering if this feature might be possible in the future, or if anyone else thinks it even makes sense to have.

 

My main use-case that I think this feature would be useful for is streaming videos with Plex. What I was thinking is that if I start to stream a video, lets say "Breaking Bad Season 1 Episode 1", it's very likely that I'll be watching all the episodes in "Season 1" soon, and eventually (but much later) the episodes in the rest of the series. It would be nice if I could configure UnRaid to watch for access in certain directories, and tell it that if file/video is accessed, that it should copy everything in the directory or parent/sub directory (i.e. either everything in the same season or same series, depending on how much I want to cache at a time) over to the Cache Drive. This way I'm spinning up the data drives less often, but dont have to worry about the losing the data if the Cache Drive dies. This is particularly useful when I have multiple people watching different things throughout the entire day.

 

Of course, this could possibly fill up the Cache Drive fairly quickly, so it'd be nice to have a few different options to limit and/or delete data cached in this way, such as:

• Reserve a maximum of X amount of space for data to be cached in this way.
  
• Only cache the directory if the data in it has been accessed X times over Y days
  
• Only cache the directory if accessing data in it causes the disk to spin up X times over Y days.
  
• Delete data cached in this way from the Cache Drive after X days of inactivity in the folder for that particular season/series/directory
  
• If the maximum amount of space has been reached when a new request for data to be cached has been received, delete the folder(s) that are least frequently accessed until enough space has been made.
  
• Etc.
  

 

This feature may also be useful for purposes other than video streaming, I just framed the idea around that since that's my personal use-case for it.

This seems like a real edge case. How is the OS supposed to know what is a TV show episode versus the tax return from 2007 you looked at in your directory of all your tax returns? You'd be asking the OS to be doing data analytics that I don't think a NAS OS should be doing. This sounds like something better suited for a stand alone application.

 

I think you're worrying too much about spinning disks up though. If accessing a TV episode, to use your example, caused all the disks to spin up, versus the disk the episode is on, it might be more worth considering. But disk 1 and disk 3 spinning up every day for a few hours for a few weeks because you're watching Breaking Bad and your roommate is watching Homeland is not going to prematurely kill those disks.

I'm confused why you even need to do this unless you use an ssd for a cache disk or you other disks are very slow.  A cache disk only speeds things up when copying to unraid no from.  Unless you team network connection your network is the bottleneck.

 

Sent from my SCH-I545 using Tapatalk

 

 

The most general way to explain the feature would be to say: "If a particular file (of any type) is accessed in a directory, then ALL files (regardless of type) in the (parent/sub) directory should be cached". The OS doesn't need to know or care about the file type, I just stuck with the example of video files since that's my personal use-case. Presumably, I keep my data in an organized manner such that my video files and tax-return files aren't in the same directory. To use your example, if in addition to my video files I also had a separate folder that contained nothing but tax returns, and I accessed my tax return from 2007, it should cache ALL the file in that folder, since it's likely I'll need to look at those as well. And of course, the caching would work best if your directory structure is such that the closer files are to each other in the directory structure, the more related they are to each other and the more likely that accessing one file means you'll need/want to access other nearby files. But if my directory structure was a mess or files were organized in some weird way, then you'd probably be right that my idea wouldn't make much sense.

This still sounds like something better suited to a standalone application. Maybe one of the plugin developers could do it.

 

I'll reiterate though that you seem to be overly concerned about disks in the array being spun up.

The feature you are requesting is very common on higher end SAN appliances that tier data usually on a block level.

 

I would suggest a sensible starting point that is much easier to justify would be to allow data to exist on the cache disk and on the array.

 

i.e. the mover script copies the most recently uploaded data to the array on a rolling basis

 

whilst this doesnt give you what you want directly it is a much simpler use case to justify and would be the first step anyway (and in my opinion genuine useful anway)

The feature you are requesting is very common on higher end SAN appliances that tier data usually on a block level.

 

I would suggest a sensible starting point that is much easier to justify would be to allow data to exist on the cache disk and on the array.

 

i.e. the mover script copies the most recently uploaded data to the array on a rolling basis

 

whilst this doesnt give you what you want directly it is a much simpler use case to justify and would be the first step anyway (and in my opinion genuine useful anway)

 

Ah, interesting! I figured my suggestion was nothing new, but had no idea where or how often it would be used in practice.

 

So if I understand correctly, the way the cache drive currently works is: New data is first written on the cache drive, then after some period of time the data is moved to the array AND deleted from the cache drive.

 

Your suggestion is: New data is first written on the cache drive, then after some period of time the data is COPIED to the array BUT ALSO remains on the cache drive. The copy on the cache drive being the copy that the user actually accesses while it exists on the cache drive, so the disks in the array dont need to spin up. I presume that once the cache drive nears it's maximum capacity, it starts to make room by deleting files in order of oldest uploads to newest, or something like that, but the user can continue to access the data from the array.

 

And my idea is basically the reverse of that: Data is copied FROM the array back TO the cache drive (so that the data exists in both places). And everything else would be basically the same as your suggestion.

 

Your suggestion is definitely something I'd like to see as well! Doesn't cover quite all my use cases, but it does over like 70%-80%, and probably would be useful for some of my other uses as well.

 

I'll reiterate though that you seem to be overly concerned about disks in the array being spun up.

 

I know you're probably right  . I don't mean to come off as "paranoid" about spinning up the array, it's not stopping me from using my array to it's fullest or fretting over it every time I throw on an episode. I just figured it doesn't hurt to be just a little "concerned" about it to get the most bang for my buck, and this idea seemed like the best way do it.

 

This seemS overly complex and may adversely affect performance

• If you are the only user then you have the overhead of copying data to the cache drive.
  
• if there are multiple users then they may be contending for access to the cache drive.
  

In SAN type solutions that were mentioned this type of approach can be beneficial despite its added complexity because different parts of the system have very different access spees to different parts of the total storage.  In a home system like unRAID the benefits would be marginal (at best) and the increased complexity would increase the chance of things going wrong.

Not really sure how doing this is getting you the best bang for your buck. You should only have as many disks in your array as allows you to keep a spare of your biggest disk anyway so if you can't afford to keep a spare disk you have too many disks in your array.

 

Also you're not spinning up the array. You're spinning up a single disk. If you're not using cache_dirs so playing an episode only spins up the disk that episode resides on you should start. But disks are going to die from time to time. It's a fact of life. Disks are rated in MTBF (mean time between fail) and not spin ups for a reason. Your disk may be spun down but it's still powered on.

Straight up this concept will gain you nothing that you can't already do using correct split-levels.  Make sure an entire season, or even an entire show, is forced to stay on a single drive and you won't have to worry about spinning up another drive when moving to the next episode or even the next season.

 

For multi-users watching two shows from two drives you aren't saving yourself either because you will still have to first spin up both drives to copy the data off them and to the cache.  Sure, full speed writes followed by a spin-down means less up-time for the two array drives, but it is the same amount of reads.  You're just not doing yourself any favors with this.

 

Honestly at some point their is such a thing as over concerned in so much as you are trying to add complexity in the code (which adds risk) to gain literally nothing objectively measurable.