Utilize cache for most used files


18 posts in this topic Last Reply

Recommended Posts

Hi,

 

I'm wondering if there is any plan to utilize cache disk for holding most requested files, I think it will significantly decrease the spin-up operations on array disks and will give better performance for media streaming.

 

Thanks in advance.

Link to post

The unRaid OS is installed in RAM. Nothing faster. Utilizing an SSD cache for Dockers and VMs results is extremely good performance for them and their databases and supporting data, as well as for transcodes (which can be configured to run in RAM if the SSD isn't fast enough.

 

Most of what is left would be media. Which if you're like me is watched once and then will typically sit idle for a while.

 

Please give an example of a performance improvement you are envisioning.

 

 

Link to post
The unRaid OS is installed in RAM. Nothing faster. Utilizing an SSD cache for Dockers and VMs results is extremely good performance for them and their databases and supporting data, as well as for transcodes (which can be configured to run in RAM if the SSD isn't fast enough.
 
Most of what is left would be media. Which if you're like me is watched once and then will typically sit idle for a while.
 
Please give an example of a performance improvement you are envisioning.
 
 


With kids, they constantly watch the same shows over and over. Take 4 or 5 movies they watch and move them to cache so the disk it is normally could stay spun down. Possibly have a plugin that would watch the active files and durations and move them to cache automatically.
Link to post

Wouldn't really work; he'd have to move the most frequently watched stuff manually, and might not have an idea what this is likely to be ahead of time. 

 

I confess, I thought the unRAID cache would do this prior to using it. I'm not sure how feasible a feature it is, but it would be nice to be able to section off 2-300GB of the cache drive/pool to store frequently used stuff.

 

One use-case that does come to mind: Steam games. Yes you can manually move them, but really, that's a pain in the nuts.

(I also know games don't benefit much from SSD vs. HDD, but when have PC users been shied away from something because of diminishing returns? ;))

Link to post
17 hours ago, alaa said:

will give better performance for media streaming.

Even the slowest hard drives now available can saturate gigabit speeds on the network.  Unless you're watching a raw file (zero compression - movie would be in the TB size range), you're not going to have any issues.  Even with multiple streams simultaneously from the same drive.

17 hours ago, alaa said:

I think it will significantly decrease the spin-up operations on array disks

If this is a pain and the 10 seconds after selecting a movie to watch before it starts, you can either set the drives to not spin down, or utilize spin up groups so that say all the drives that contain TV shows will spin up at the same time so that the delay is only on the first show.

Link to post

So basically there are features available that would enable you to optimize your configuration to avoid spinups. But at the same time, many here believe that you should not be concerned about array disks spinning up when needed to serve media. That's what this platform is all about!

Link to post

I believe it will maximize hard disks lifespan since read operations tend to cause tear and wear for normal hard disk, however it not the same case with SSD's since its lifespan only impacted by write operations.

 

 

Link to post
How about creating a new cache-only share. Move the files into this; then add this share as one of your media locations in kodi or plex, or whatever you use.


This is the method I plan to use once my larger cache drive hits my door. Just manually move stuff as needed.
Link to post

Did this about a year ago.  I had a large 1TB SSD in the array, and moved my small files to it.  I have since removed the 1TB SSD to another system.  I haven't seen much difference in any application that might have used it.  (i.e. Plex)

 

What would really be nice, is if we could get the mover script that runs to take a date or "days old" to move files.  That way it would work with your normal cache and only move file older than XX days old off.  Keeping all the recent files in the cache.  Basically a "smart" mover.

 

 

Link to post

SSDs have a limited lifetime, although in recent years that lifetime has been getting longer. But I just ran a performance test on my new SSD vs my hold one, and the old one has definitely slowed down under heavy use. The wear level is down to about 50%, but the performance on the beginning of the SSD has definitely degraded. I would not be moving a lot of media to and from the SSD on a frequent basis.

 

I think repeated spinups and downs is worse than having a disk spin longer. I've read its not so much the spinning that is bad, but the thermal cycles. So its better for a disk to run constantly a little too hot, than go from cool to warm frequently. For that reason, I have my spindown timer set at 5 hours. That usually means that a disk will only spin up once a day (if at all), rather than several times a day if I had it set a 1 hour. I've had very good luck (knock on wood) with drive longevity.

Link to post

I honestly don't think you need a cache drive to play media files...particularly for the kids who aren't going to 'appreciate' any amazing solution you come up with for them!

 

Here's what I've done which might help you.

 

1. Use Script to run mover at a threshold rather than on a schedule

 

 

I use this script to run mover when my SSD cache pool gets to 80% capacity rather than automatically overnight.  This means that my latest shows usually are on the cache drive, which means they launch immediately rather than after a few seconds if the array drive is spun down.  You could put the kids files in a prefer cache setting so they are never moved

 

2. Setup Split Levels correctly

 

My TV shows are setup:

 

\media\tv shows\kids\show name\season xx

\media\tv shows\adults\show name\season xx

 

I've set my media share to split level 4, so that for any given show a given season is always on the same disk.  So, if an episode isn't on my cache pool then if the disk isn't spun up, the first episode will take up to approx 3-5 seconds to load, but the next episode loads in under a second as the disk is already spun up

 

I honestly think it's a waste of a cache drive to hold a whole video library where you only select a new file usually every 30, 45 or 60 minutes.  For music, it might be a different story but again with the right split levels, you can limit any potential delays while the next track is accessed.  My music is stored \media\music\kids\artist\album so again each album is on one disk to limit access times.  I think random play delay issues in kodi are alleviated by using crossfade, which gets the next track 'ready' before the current track has finished.

 

Link to post
On 25/06/2017 at 6:19 AM, digiblur said:

 


With kids, they constantly watch the same shows over and over. Take 4 or 5 movies they watch and move them to cache so the disk it is normally could stay spun down. Possibly have a plugin that would watch the active files and durations and move them to cache automatically.

 

I wouldn't waste my cache drive space for my kid's movies. I need it for my VMS etc! I would just tell them they just have to wait a few seconds for the drive to spin up. Kids today would have hated VCRs having to rewind tapes !! ;-)

Link to post

Or to wind the old projector, feed the film, roll out the movie screen to watch a Popeye cartoon on 8mm for 2 minutes. I remember someone doing that for my birthday parties when I was a kiddie.

Edited by hernandito
Link to post
On 6/28/2017 at 10:51 AM, gridrunner said:

I wouldn't waste my cache drive space for my kid's movies. I need it for my VMS etc! I would just tell them they just have to wait a few seconds for the drive to spin up. Kids today would have hated VCRs having to rewind tapes !! ;-)

 

ROFLMAO!

Link to post
On 6/28/2017 at 8:14 AM, DZMM said:

I honestly don't think you need a cache drive to play media files...particularly for the kids who aren't going to 'appreciate' any amazing solution you come up with for them!

 

Here's what I've done which might help you.

 

1. Use Script to run mover at a threshold rather than on a schedule

 

 

I use this script to run mover when my SSD cache pool gets to 80% capacity rather than automatically overnight.  This means that my latest shows usually are on the cache drive, which means they launch immediately rather than after a few seconds if the array drive is spun down.  You could put the kids files in a prefer cache setting so they are never moved

 

2. Setup Split Levels correctly

 

My TV shows are setup:

 

\media\tv shows\kids\show name\season xx

\media\tv shows\adults\show name\season xx

 

I've set my media share to split level 4, so that for any given show a given season is always on the same disk.  So, if an episode isn't on my cache pool then if the disk isn't spun up, the first episode will take up to approx 3-5 seconds to load, but the next episode loads in under a second as the disk is already spun up

 

I honestly think it's a waste of a cache drive to hold a whole video library where you only select a new file usually every 30, 45 or 60 minutes.  For music, it might be a different story but again with the right split levels, you can limit any potential delays while the next track is accessed.  My music is stored \media\music\kids\artist\album so again each album is on one disk to limit access times.  I think random play delay issues in kodi are alleviated by using crossfade, which gets the next track 'ready' before the current track has finished.

 

3

 

Hi, I'd like to use this but I'm not quite sure where I can find the share.cfg file, and where I should put the php file you made. Can you please clarify? Thanks!

 
Link to post
On 7/3/2017 at 7:13 PM, vmunich said:

 

Hi, I'd like to use this but I'm not quite sure where I can find the share.cfg file, and where I should put the php file you made. Can you please clarify? Thanks!

 

 link above gives the best info on how to create a custom script.  I think the plugin itself allows you to paste code in now so you don't have to use a file manager.

 

share.cfg is in the config folder on your flash drive.  Look for a like that says something like:

 

shareMoverSchedule="0 4 * * *"
 

 

and change it to:

shareMoverSchedule="#0 4 * * *"
Link to post

I actually have a very similar issue to OP. I solved this issue by installing a small external HDD on the media player TV they watch with their most used movies on it. 

 

Another way I envision this working is for you to create a share on your SDD via unassigned devices, copying your most used movies over to it manually and then adding that share to your library so it just always plays from that disk. It'll require you to manually manage this, but it would accomplish what you want. 

 

Lastly if you use Plex, you can sync optimized copies using plex web to individual players (provided they have storage) and set it to always play from sync copies first. 

 

It's not what you asked for, which I think was a bilateral cache, but you can still get the functionality you need much faster than waiting for LT to even consider adding this.

Link to post

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.

Guest
Reply to this topic...

×   Pasted as rich text.   Restore formatting

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.