Hi all, so say my unraid server is like the following:

Parity1: 6TB HDD

Disk1: 6TB HDD
Disk2: 2TB SSD

Disk3: 2TB SSD

 

Cache1: 2TB SSD

Cache2: 2TB SSD

 

What I'm looking to achieve is to have most recent files write to cache for speed, then once a week or so move those files to the array but only to the SSDs, and after a month, move the files from Array SSDs to Array HDDs. This way, when reading the files, I get lots of speed, and as files age out, they'll be stored on longer term cheaper storage. Essentially creating a Performance Tier, and a Capacity Tier.

 

As far as I know, read speeds wouldn't be affected, so read speeds from Array SSDs should get good performance.

 

If I set the share to exclude the slower disk, it'll never write to it. Even as it ages out. Perhaps, I can script it to move from disk to disk? But if the Share is set to exclude that disk, it won't see it, correct? 

 

Any suggestions would be helpful.

This can probably be done via a BASH shell script that you would have to write.  To learn how, try googling something like  how to write a BASH script that your script would copy 'aged' files to the other 'side'.  

 

By the way, It is my impression that using SDD's as array drives can be a long term performance issue as you can't run the TRIM operation on them.  In any case, you could setup your 2-2TB SSD cache drives as a cache pool with data protection.  Of course, you would only have 2TB of cache storage in that case.  But it would give you the speed you are looking for along with data protection. 

Edited August 17, 20196 yr by Frank1940

22 hours ago, Frank1940 said:

This can probably be done via a BASH shell script that you would have to write.  To learn how, try googling something like  how to write a BASH script that your script would copy 'aged' files to the other 'side'.  

 

By the way, It is my impression that using SDD's as array drives can be a long term performance issue as you can't run the TRIM operation on them.  In any case, you could setup your 2-2TB SSD cache drives as a cache pool with data protection.  Of course, you would only have 2TB of cache storage in that case.  But it would give you the speed you are looking for along with data protection. 

Hi, thanks for the response. As for SSDs on the array, they can be always on and have extremely fast performance for millions of tiny files. We're using this for an NVR and the video files tiny and stiched together on the NVR side. The benefit is huge when compared to HDD for random reads.

 

As for cache. I'm already doing the 2x2TB in a raid1 for redundancy. But if I add all SSDs to the cache in raid10 I lose half. 

 

I think long term I'm fine with SSDs in the array. We'll replace as performance is impacted. Or remove from array, run a trim and return. 

 

The bash is likely the best place to do this but was hoping for a simpler approach.  The issue I have is the Disk Exclusion on the share. If I exclude the disk, then moving old files there, the share won't see it. If I include the HDD in the share, it'll try writing to that disk if I do fill up or high water. 

If I understand what you are doing, I would suggest this approach.  Setup the HD share (Call it --Long_Term) and restrict it to the HD's.  Setup the SSD's share (Call --Short_Term) and restrict it to the SSD's.  By using the shell script, you can easily tailor it to copy from the Short_Term share to the Long_Term share.  You just have to determine at what age, you want to make the move from one to the other to minimize having to go the long term storage side of things for work-in-progress files. 

Edited August 18, 20196 yr by Frank1940

11 hours ago, Frank1940 said:

If I understand what you are doing, I would suggest this approach.  Setup the HD share (Call it --Long_Term) and restrict it to the HD's.  Setup the SSD's share (Call --Short_Term) and restrict it to the SSD's.  By using the shell script, you can easily tailor it to copy from the Short_Term share to the Long_Term share.  You just have to determine at what age, you want to make the move from one to the other to minimize having to go the long term storage side of things for work-in-progress files. 

Hi Frank1940, appreciate your responses to this. Great idea, but that means I can't pass a single share to a docker or VM and have unraid handle the capacity&performance tier operations. It's pretty simple logic in the disk fill allocation. Similar to how there's the cache "prefer, yes, only" there can be disk groups "slower, faster" or HDD, SSD. I think with larger 4TB SSDs becoming much cheaper, and HDD sort of getting left behind at some point, it only makes sense to accommodate SSD Array users.

 

You can also submit a 'Feature Request' in this sub-forum.  It might be time to revisit this  feature again.  

 

https://forums.unraid.net/forum/53-feature-requests/

I'm a newbie to unRaid, but I thought there was a "mover" function that did just this. I too wanted to make use of SSD, 7200RPM HDD and 5400 RPM HDDs.

There is a "Mover' function that is a part of Unraid.  When it is invoked, it moves everything from the cache to the array.

 

In my quick scan of the the previous posts in this thread, what I think the OP wanted was to move only those files on the cache drive that had not had 'activity' for a certain period of time.  This was  not then (or currently is) a feature of Unraid. 

 

However, you do have the option of setting up Pools today where you could use multiple high speed SSD's to configure a large capacity high speed file system.  Moving the files after they had 'aged' is still an issue if that is what you need. 

 

I have no doubt that a BASH script could be written that would do this and there is a plugin called "Users Scripts" that will help.  (It appears to  me that this plugin will even allow you to schedule when the script will run.)

Edited May 3, 20242 yr by Frank1940

Just so I understand. If I have a "bunch-o-disks" (speeds from SSD, 5400RPM to 7200RPM) I'd have to manually write a script to move files from one physical disk to the other disks based on last usage to the slowed and possible higher capacity disks to simulate my own personal tiered storage.

If I do that what happens to the files in the file system themselves? My issue would be is that I use jdupes to hard link all duplicate files to one inode to save on storage. Wouldn't this moving of files about completely invalidate the inode map of when files are physically located to where the file system thinks they are?

Again I am a newb at unraid, but I'm just trying to mentally understand how unraid lays out its file systems when compared to all the disks physically in the system.

I thought unraid was working kind of like Drobo did, but that doesn't seem to be the case.

 

thank you