[6.8.3] "Smart" episodes distribution across multiple drives

[Januszmirek]

Hello everyone, one of my first posts here.

I was almost pushing the Buy button for a new rig designed to be fuelled by xpenology when by chance I discovered unraid and absolutely dropped my jaw how great that system is.

I found out about unraid when I was reading a post on a different forum where a user was describing cool configuration he was able to pull out from unraid - mainly for his Plex library. This is pretty much the same usability I want to achieve but don't know how to configure. Unfortunately that topic on that forum was already closed and my PMs to that user are left unanswered. So here's the idea:

 

1 x nvme disk used for cache

3 x hhd used for single array (disk1, disk2, disk3) + 1 hdd for parity

 

Issue no 1:

In accordance with what that user was saying he was able to configure his unraid to achieve this:

Say we have a share called "TV" in which there are multiple subfolders with title of "TV Show Title", then obviously these tv show folders would have 'Season' subfolders with multiple Episodes inside. In order to optimise the way files are being read from the array all downloads are primarily downloaded to cache disk first. This is easy - I assume you only set cache drive to Prefer mode for "TV" share. Once the space on cache is full here comes the fun part. From what that user was saying the array is configured to have every other Episode form a single Season to be copied across different disk in array to avoid situation where a popular show is being for example accessed from the same drive by multiple users at the same times, whilst other disks are in idle. So in reality say you have S01E01, S01E02, S01E03, S01E04, S01E05, S01E06, and so on.

The idea is to force unraid to copy from cache S01E01 to disk1, S01E02 to disk2, S01E03 to disk3, S01E04 to disk1, S01E05 to disk2, S01E06 to disk3, and so on. In this way each Season is spread evenly on all disks - and not entire Season per a single disk. Can this be achieved? And if so, what setting do I need to change to achieve this for TV share? And yes, I read the 'Split level' description but still don't get how different options actually work.

 

Issue no 2:

Cache disk configuration. Again the idea is to keep the files mostly accessed by other members of family in cache - to ease the hdd spinning. In the aforementioned post the user was saying that all movies and tv shows are being downloaded to a cache and then once the cache is filled up to 80% the oldest file is being moved to the array - again each movie to different disk - similarly with TV shows idea above. My approach is a bit different - don't know whether this is even possible. I'd like to be able to set condition of moving file to the array not based on the date of file was created but date when file was last accessed. Why? I have for example movies downloaded 3 months ago which are often accessed and I have a movie 1 month old that nobody ever accessed and would like to send this 'unpopular' file to array first even though it is newer than more 'popular' older file. Again is this even possible? If so how do I set this up? By the way is it possible to limit cache disk utilisation to maximum say 80%?

 

And final question, if I understand correctly cache should work two ways, if a file on the array is frequently used, shouldn't it moved back to cache as well? If so, where can I read more on what determines what is classed as 'frequent' use and if this can be in anyway controlled by user or is it automatically handled by unraid with no steer possible from user.

 

Many thanks for help.

 

 

[Squid]

26 minutes ago, Januszmirek said:

The idea is to force unraid to copy from cache S01E01 to disk1, S01E02 to disk2, S01E03 to disk3, S01E04 to disk1, S01E05 to disk2, S01E06 to disk3, and so on. In this way each Season is spread evenly on all disks - and not entire Season per a single disk. Can this be achieved? And if so, what setting do I need to change to achieve this for TV share?

Effectively the way that you do that is to set the allocation method for the shares to be "Most Free".  There's no real way to do it otherwise, but with "Most Free", the system will wind up balancing itself out with the episode files all stored on different disks.

 

26 minutes ago, Januszmirek said:

I'd like to be able to set condition of moving file to the array not based on the date of file was created but date when file was last accessed.

Last access time is disabled by default in unRaid (and many other OS's), as keeping track of that massively slows down file I/O.  But, you can look at the Mover Tuning Plugin (in Apps) which will let you selectively move based upon the "age" of the file (creation date)

 

26 minutes ago, Januszmirek said:

if a file on the array is frequently used, shouldn't it moved back to cache as well?

No.  The point of the cache drive is to speed up writes, not reads.  It was really necessary way back when drive transfer rates on spinners were pathetic compared to now.  Nowadays, hard drives can easily keep up with multiple simultaneous reads on the same file (streaming the same file to multiple devices)  Many people (including myself) don't even bother with using the cache drive as a "cache drive".  Rather they set all the shares to not use it, and relegate the cache drive to being an "application" drive -> data for docker applications, VM's etc

[Januszmirek]

Quote

Effectively the way that you do that is to set the allocation method for the shares to be "Most Free".  There's no real way to do it otherwise, but with "Most Free", the system will wind up balancing itself out with the episode files all stored on different disks.

Great, thanks for this.

Quote

Last access time is disabled by default in unRaid (and many other OS's), as keeping track of that massively slows down file I/O.  But, you can look at the Mover Tuning Plugin (in Apps) which will let you selectively move based upon the "age" of the file (creation date)

I don't get how would daily check on a cache drive for date accessed information 'massively' impair the i/o speeds. As long as there is a script to scan cache drive once a night and create a 'record' of files last access dates before the Mover is launched. Then that could trigger Mover which would move first those files which haven't been accessed for ages. I don't mean constant monitoring of files access dates - agree that could impact the performance. But if the 'file access date check' could be somehow tied with Mover it could achieve exactly what my goal is.

 

Quote

No.  The point of the cache drive is to speed up writes, not reads.  It was really necessary way back when drive transfer rates on spinners were pathetic compared to now.  Nowadays, hard drives can easily keep up with multiple simultaneous reads on the same file (streaming the same file to multiple devices)

Same file yes. But the scenario where 5 different files are accessed from same drive at the same time is a another story. Hence my query about spread of files across all drives. But I get it now, cache is only for writes, not reads. If there is any plugin that would help optimising reading performance by moving files from array back to cache, please let me know.

 

And once again many thanks for help on the above:)