Cache - Last Recently Used


Recommended Posts

Greetings!

I'd like to ask if it's possible to implement in near future that I can work exclusively off of SSD and files are backed up let's say once every hour. But if the SSD gets filled, last recently used files will be removed and new files will take it's place.
I'm working with 3D simulations and this is one of several occasions where I can take advantage of even several RAID 0 nvme SSDs. 

Sometimes one frame has more than 2GB which gives you almost 50GB/s of data for 24fps scene. Of course such a speed is another topic of limitations but it's just simply impossible to read from HDD's ( not only the read speed is limiting, but also Random access speed).

 

I currently have 12TB of HDD (2x4TB + 1x4TB parity) and regular SSD, planing to upgrade to 2TB nvme and 40Gbps networking.

Is there any solution to this? I really like unRAID, would be nice if I can use it not only for my personal (travel photos, movies), but also professional use.

 

Thanks in advance :)

Marek

Edited by Marek
Link to comment
  • 2 weeks later...
On 5/7/2020 at 4:22 AM, Marek said:

Greetings!

I'd like to ask if it's possible to implement in near future that I can work exclusively off of SSD and files are backed up let's say once every hour. But if the SSD gets filled, last recently used files will be removed and new files will take it's place.
I'm working with 3D simulations and this is one of several occasions where I can take advantage of even several RAID 0 nvme SSDs. 

Sometimes one frame has more than 2GB which gives you almost 50GB/s of data for 24fps scene. Of course such a speed is another topic of limitations but it's just simply impossible to read from HDD's ( not only the read speed is limiting, but also Random access speed).

 

I currently have 12TB of HDD (2x4TB + 1x4TB parity) and regular SSD, planing to upgrade to 2TB nvme and 40Gbps networking.

Is there any solution to this? I really like unRAID, would be nice if I can use it not only for my personal (travel photos, movies), but also professional use.

 

Thanks in advance :)

Marek

Some point in the future? Probably. Near-future? Doesn't sound likely. Check out this Q&A with LimeTech's CEO from Aug 2019:

 

https://unraid.net/blog/unraid-14th-birthday-blog

 

Quote

Related, I want to build in more intelligence to the ‘mover’ to auto-promote share data based on usage and storage device characteristics available to a server (or group of servers).

 

Link to comment
  • 1 month later...

I was actually about to request the same feature.

 

In my use scenario, my server would have a lot of reads during the day, and maybe a handful of writes (as to when I move files to the server) and most of which aren't big enough to get my small SSD to half use. My cache doesn't see much use outside of that as I'm running the VM's on another SSD (as most people here).


I would like to be able to select a folder and its sub-folders to be cached, maybe even the same disks that I've already have cache active on would work. (Allow exclusion folders to be selected? Auto exclude if (folder size greater than X limit OR files/folder + current cached size > limit/available space? IDK). If I play a song from an album located in one of the selected cache folders, I would like that the song and all of the other songs in the folder and sub-folders to be copied to the cache and then played from there. After an specified amount of time, 3? days, if they have not been accessed (file opened) then the cached files should be simply deleted from the SSD. This should just be an update part of mover, the files would still live in the Array. If I add a new file to this folder, it would just work as it does now, except that as I just recently "Accessed" the folder (to add the song), the mover would only copy it to the array at the normal specified time currently in the settings but leave it in the SSD. Another option to add to this would be a space limit to allow for this feature, perhaps a certain %.

This would make better use of the cache IMO. Keep the drives from spinning up more and therefore life longer, maybe. A great performance boost to recently accessed files while keeping them safe. Lower I/O access usage. Allow for better and/or un-interrupted use of recent files/projects during a parity-check.

Edited by XiuzSu
Typo/update.
Link to comment

+1

 

Adding "Automatic" as a cache option to populate it algorithmically based on access frequency and last-access-time would be great.

 

It'd even make sense as the default because that matches new users' expectations.

 

There should be a default threshold capacity (maybe configurable) or a "reserved" size because you don't want the cache drive 99% full.

  • Like 1
Link to comment
  • 10 months later...

Just going to drop in here and say that I'd like to have some sort of "smart cache" option as well.

 

In my case, we're using a UnRaid server as a NAS for our office. Multiple people are working on the files from the server at the same time.

What I'd like the "smart cache" option to do (and what I already thought "prefer" cache option did..) is:

 

1. Prefer to keep as much as possible within the cache

2. Once a threshold is met (as you can set up the limit within CA Mover Tuning), the mover would prioritize the files that have been most recently accessed, and everything else would be moved to array

3. If a file within the array is accessed, the next time the mover is activated, it'd move said file back to the cache

 

Excuse my english. It's late here, and english is not my native language.

And sorry for digging up such an old topic.

Link to comment
  • 2 weeks later...
  • 6 months later...

Votes for:

 

1. Algorithmic caching of files, perhaps saying, use 50% of an SSD for this. A FIFO. Older files get aged out and written to the array. Drobos have a cache 'accelerator' bay that presumably does this. Hybrid drives have this. I.e., the Windows OS probably lives on the SSD portion of the drive as it gets read and written the most.

 

2. A certain % of fullness where the mover triggers vs. being time based.

 

3. Autobackup to array of VMs etc. that are on the cache drive. And should really rename this function since this isn't really caching? It's just fast storage.

 

4. Perhaps an asynchronous mirror of the cache drive. I.e., cache drive is priority read/write on the mirror then the SSD gets sync'd to it's mirror as speed allows. Example: I could have a 1TB SSD and a 1TB 5400 drive in the array that is a mirror of the SSD. No problem as long as the drive can catch up eventually.

 

 

 

 

 

 

  • Like 1
Link to comment

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.

Guest
Reply to this topic...

×   Pasted as rich text.   Restore formatting

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.