Utilizing SSD as a smart read cache

[jpeltoniemi]

Hello all!

 

I recently discovered unRAID through Linus Tech Tips and am currently preparing my drives for it. 12TB of files is getting a bit too much for Greyhole.

From what I've read, unRAID doesn't utilize cache drives for accelerating often accessed files, which still is better to what I have now, that being no cache at all, but I was wondering if it could be even better.

 

I'm not too familiar with unRAID internals yet, but AFAIK, it uses Samba to serve files. Wouldn't it be possible to to monitor which files are requested, either by hooking in to Samba(https://www.samba.org/samba/docs/current/man-html/vfs_full_audit.8.html) or to kernel(https://linux-audit.com/monitor-file-access-by-linux-processes/) and pass all the file requests to a cache server to process.

 

Being a js developer, my first though was to run a Node.js instance with either Redis or MongoDB to store access to each file. The cache server would periodically check which files are accessed most often and copy them to cache, as well as remove less accessed files from the cache. The cache server could perhaps even be configured to look at currently accessed file names and pre-emptively cache potentially next-in-sequence files, or something like that. Ideally caching strategies could be pluggable. 

 

This kind of read cache would be great for application data(Steam libraries) and project files, which aren't always worth optimizing individually and can't all be stored on a stack of SSDs(unless you're made of cash). I'm sure there's something I'm missing, but if it isn't anything crucial and something like this is actually doable, my questions are as follows:

 

• Is there any plan to implement such functionality in unRAID natively / is there a timeline?
• Despite finding no good answers on Google, does a plugin for this already exist?
• If I were to take a shot at this, is there something I might've missed, underestimated or misunderstood? Anyone to consult when interfacing with unRAID?
• If I manage to wrap something up, any volunteers to help with the UI?

 

Of course, if I have missed something (crucial or not), please tell me.

 

Thanks

 

[JonathanM]

1 hour ago, jpeltoniemi said:

Anyone to consult when interfacing with unRAID?

I recommend emailing @limetech directly, [email protected]

[jpeltoniemi]

Is Limetech not active on the forums? I mean, sure, for extremely detailed topics that would be my first choice, but I would maybe ask on the forums for lighter information that may be common knowledge among tinkerers. If possible, I'd like to avoid bothering Limetech directly with such things.

 

The following question(s), which I forgot to ask in the initial post, is something I suppose could be common knowledge:

 

How does unRAID determine if a file from cache should be served instead of a copy in the array? Would it be possible to either interface with unRAID caching, in order to tell the system about new cached files, or does everything work automagically, i.e. copy files to appropriate path in cache and they become available?

 

This is maybe going a bit off topic, but I was wondering what would happen if the mover script tries to move a file from cache that is either being read or being written to? And what would happen if a file is written to cache while it's already being read from the array?

 

Sorry if my questions are a bit basic. I'm still preparing my drives and can't actually try to find these things out myself, and the both the Wiki and Google produce either answers that are too vague or nothing at all.

[SSD]

Actually the array is a pretty poor choice for running apps or games. Normally users are utilizing the cache drive (often used more as an.application drive / virtual disk image) for such purposes. The array tends to hold large media files with few if any interactive writes.

 

UnRaid also supports unassigned disks, meaning not in the array or cache. They can be spinners or SSDs and used for various purposes that typically involve a lot of interactive disk writes.

[trurl]

The way unRAID allows spanning of disks is called "User Shares". Basically, each top level folder on cache or array with the same name is a user share with the same name as the folder name. For example, if you have a top level folder named "media" on cache, disk1, and disk2, there is a user share named "media" which is the union of the contents of the /media folder on each of these disks.

 

Each user share has settings which control how writes to the user share are done. Any reads from a user share includes all disks with that folder regardless of those write settings.

 

The original purpose of cache was to increase write speed to the user shares. Each write to the parity array requires reads and writes so that parity can be calculated and maintained realtime. In more recent versions of unRAID with dockers and VMs, cache is also often used as the primary storage for user shares that are frequently read.

 

Each user share has settings which control how it uses cache. Those settings are detailed here: https://lime-technology.com/forums/topic/46802-faq-for-unraid-v6/?page=2#comment-537383

 

 

Mover mentioned at that link is a builtin script which runs on a user specified schedule, and moves user share contents to/from cache.

 

There is nothing in any of this that provides for read-caching data on some sort of "as needed" basis. There are just user shares that do or do not get moved to/from cache on a schedule.

[jpeltoniemi]

49 minutes ago, SSD said:

Actually the array is a pretty poor choice for running apps or games. Normally users are utilizing the cache drive (often used more as an.application drive / virtual disk image) for such purposes. The array tends to hold large media files with few if any interactive writes.

 

UnRaid also supports unassigned disks, meaning not in the array or cache. They can be spinners or SSDs and used for various purposes that typically involve a lot of interactive disk writes.

I've seen someone mention that games or apps shouldn't be run from the array. I'd like to know why, though. Does it have something to do with access times, possible data corruption, or something else? I'm genuinely interested in the underlying reasons and my thirst for knowledge can't be satisfied with just "it's a bad idea". If it's just about read performance, this topic is kind of about tackling just that. Many apps and games rarely write into their own directory but instead use AppData and My Documents, so the apps in array would mostly just be read.

 

I know about unassigned disks and I've considered the feature for various purposes, but in the end I'd really like just set it and forget it. I got into disk pooling and unRAID in order to avoid having to think which drive should I put my stuff on today. While I can't avoid unassigned disks for certain specific tasks, I'd like to try to get maximum conveniency and benefits from the array.

 

48 minutes ago, trurl said:

The way unRAID allows spanning of disks is called "User Shares". Basically, each top level folder on cache or array with the same name is a user share with the same name as the folder name. For example, if you have a top level folder named "media" on cache, disk1, and disk2, there is a user share named "media" which is the union of the contents of the /media folder on each of these disks.

 

Each user share has settings which control how writes to the user share are done. Any reads from a user share includes all disks with that folder regardless of those write settings.

 

The original purpose of cache was to increase write speed to the user shares. Each write to the parity array requires reads and writes so that parity can be calculated and maintained realtime. In more recent versions of unRAID with dockers and VMs, cache is also often used as the primary storage for user shares that are frequently read.

 

Each user share has settings which control how it uses cache. Those settings are detailed here: https://lime-technology.com/forums/topic/46802-faq-for-unraid-v6/?page=2#comment-537383

 

 

Mover mentioned at that link is a builtin script which runs on a user specified schedule, and moves user share contents to/from cache.

 

There is nothing in any of this that provides for read-caching data on some sort of "as needed" basis. There are just user shares that do or do not get moved to/from cache on a schedule.

Sorry, I must have not been clear enough. I want to try, if feasible, create a background service that monitors the most used files and puts them on the cache drive for faster access in the future. In theory this should be doable, but I need to learn more about the implementation and possible caveats, which is why I made this topic.

 

I already know that cache is overlaid on top of the array in the share. What I really wanted to know, how the overlaying is done internally, like how exactly does the server list files from the respective path in the array and the cache as one, how does it check for cache while determining which file to serve, and how does it handle conflicts when reading and writing the same file.

Edited February 26, 2018 by jpeltoniemi

[trurl]

20 minutes ago, jpeltoniemi said:

Sorry, I must have not been clear enough.

You were perfectly clear. I was just describing things as they are now in case there was something there you didn't already know.

 

I don't know the low-level details, I think it has something to do with fuse and shfs. Maybe someone else can help.

[trurl]

You might also browse the Feature Requests subforum. There have been requests for similar functionality over the years. For example: https://lime-technology.com/forums/topic/32790-accelerator-drives/

 

[SSD]

@jpeltoniemi -

 

Writes to the parity protected array are pretty slow. And involve accesses to both the disk involved and the parity disk. Each is required to do a block read followed by a block write of the same block. This means a full revolution of the slower of parity and the data disk. It is a decided write penalty.

 

As @trurl mentioned, there is no existing feature in unRAID to leverage your elegant read-caching paradigm. I can't say with certainty if a duplicate file existed in the same user share on cache as on a disk in the array, that unRAID would use the one on cache or not.

[trurl]

And another reason I thought it might be helpful to explain user shares to you:

4 hours ago, jpeltoniemi said:

I recently discovered unRAID through Linus Tech Tips

Many people whose first post starts like this aren't even aware that unRAID is a NAS. They just think it would be cool to allow multiple gamers to play against each other on one machine.

[jpeltoniemi]

Just now, trurl said:

And another reason I thought it might be helpful to explain user shares to you:

Many people whose first post starts like this aren't even aware that unRAID is a NAS. They just think it would be cool to allow multiple gamers to play against each other on one machine.

A reasonable assumption  

I'm a programmer and manage a 6 hdd home server, so I'd like to think I'm not completely incompetent when it comes to basics of NAS or unRAID. Gotta admit though, at first I was going to try having my primary machine run unRAID with virtualized GPU, but I later found out my CPU doesn't support vt-d. Whoops.

 

About your link... I'm currently reading it through. So far I've seen only one post by Limetech, not addressing the original topic. The problem with old threads is that they're often outdated, and sooner or later either die or change topic, which seemed to be the case with this one. I'll try to look for more, but I'm not feeling very optimistic. I guess I'll just have to try to get as far as I can by myself and ask questions when I actually encounter a problem.

[trurl]

2 hours ago, jpeltoniemi said:

Is Limetech not active on the forums?

They typically read release threads and might pop in on some others threads on occasion but they won't read every thread. Even I don't read every thread!

[jpeltoniemi]

1 minute ago, trurl said:

They typically read release threads and might pop in on some others threads on occasion but they won't read every thread. Even I don't read every thread!

Of course, that much is expected. I meant it like would it be reasonable to expect some kind of reply from the developers, if pinged, in the next week or so, or if the thread stays relevant? I'd very much like to avoid emailing Limetech directly and instead keep the discussion public, so others can take part in the discussion. This thread is getting a bit sidetracked right now, so I think we'd best get back on topic from here on

 

I'm probably gonna set up unRAID this evening, when I get my drives prepared. After that I'll look into the mechanics of caching and try to wrap up some more concrete questions. If that goes nowhere, I'll just have to contact Limetech directly.

[itimpi]

Regarding one of your questions - unRAID never expects a file to exist on both cache and array simultaneously.    If you force such a scenario by manually copying files  then one copy will end up hiding the other in a User Share and you do not have any control over which copy you are accessing.    You can also get confusing behavior where you delete a file and nothing appears to happen as you have now ‘uncovered’ the other file with the same name that was previously being hidden.

[primeval_god]

1 hour ago, jpeltoniemi said:

Of course, that much is expected. I meant it like would it be reasonable to expect some kind of reply from the developers, if pinged, in the next week or so, or if the thread stays relevant?

Sorry one last off topic, just an FYI since it sounds like you are fairly new here. In case you were not already aware, Limetech is an incredibly small company; incredible particularly in regards to the product they have created and the community they have fostered here on the forums. Unfortunately due to their size, and despite the best intentions, communication from them here on the forums tends to fall by the wayside, especially on topics that fall lower on the priority scale. Luckily the community engagement here is fantastic, one of the best features of unRAID in my opinion. 

[jpeltoniemi]

2 hours ago, itimpi said:

Regarding one of your questions - unRAID never expects a file to exist on both cache and array simultaneously. 

I'm not sure I understand. Say I copy movie.mp4 to the server. It ends up in cache until the mover script saves it to the array. So far, so good. Now, if I notice this was the wrong version of movie.mp4, be it incomplete or has errors, what then? If a file can not exist on both cache and array simultaneously, doesn't it mean that the file in the array has to be removed, or that cache will be skipped and movie.mp4 will be written straight to the array much slower?

 

I always thought that the array and cache may contain the same file, and in that case the cache file is always used until it is moved into the array, replacing the older file version.

1 hour ago, primeval_god said:

Sorry one last off topic, just an FYI since it sounds like you are fairly new here. In case you were not already aware, Limetech is an incredibly small company; incredible particularly in regards to the product they have created and the community they have fostered here on the forums. Unfortunately due to their size, and despite the best intentions, communication from them here on the forums tends to fall by the wayside, especially on topics that fall lower on the priority scale. Luckily the community engagement here is fantastic, one of the best features of unRAID in my opinion. 

Just registered to start this thread. Thanks for the info. I guessed as much that Limetech is a small company - partly why I'm posting here and not bothering them directly just yet  Having been a small time entrepreneur I understand how much work simple client relations can be.

[itimpi]

20 minutes ago, jpeltoniemi said:

'm not sure I understand. Say I copy movie.mp4 to the server. It ends up in cache until the mover script saves it to the array. So far, so good. Now, if I notice this was the wrong version of movie.mp4, be it incomplete or has errors, what then? If a file can not exist on both cache and array simultaneously, doesn't it mean that the file in the array has to be removed, or that cache will be skipped and movie.mp4 will be written straight to the array much slower?

If you copy a file in the ordinary way via a User share then unRAID will ensure that only one copy of the file exists.   If it is on the cache it will not be on the array and vice versa. 

 

it is possible to circumvent the normal unRAID handling by going in at the raw Linux level and manually copying the same file to more than one location.  In such a scenario there is a predictive order in which copy shows up via the User Shares.   I suspect a cache copy takes top priority but I am reasonably certain this is more an accident of the implementation rather than something you should rely on.    Also if you create such a scenario manually then when mover runs and moves a file from the cache to the array then the cache copy is certainly the one you end up with.

[limetech]

9 hours ago, jpeltoniemi said:

Is there any plan to implement such functionality in unRAID natively / is there a timeline?

 

Sure we've thought about promoting/demoting files from array to cache and cache to array on demand, for the purpose of speeding up I/O for games.  No timeline.

[jpeltoniemi]

20 hours ago, itimpi said:

If you copy a file in the ordinary way via a User share then unRAID will ensure that only one copy of the file exists.   If it is on the cache it will not be on the array and vice versa. 

Thanks for the clarification. Certainly not how I imagined it would work. Can someone tell me what are the advantages of having a rule of one copy of a file in array and cache vs what seems to be actually happening, i.e. some kind of layered file system with the cache drive as the top layer? So far I haven't been able to come up with any good reason to spend resources enforcing the rule, when to a client it's all the same if the file is served from either the array or cache, as long as it's the most recent version.

21 hours ago, itimpi said:

I suspect a cache copy takes top priority but I am reasonably certain this is more an accident of the implementation rather than something you should rely on.

This accident of implementation is partly what I don't get. This should be more or less how layered file systems work and I see no reason to actively enforce any no-duplicates rule.

 

@limetech Thanks for responding. I have a couple of questions cooking for you, but I'll get more comfortable with unRAID first so I wouldn't waste your time with anything too dumb. However, if you have the time, feel free to comment on the current discussion about how cache works. Also, two quick things: 1) Can I potentially break something now or in the future, if I disabled the mover script(and handle flushing of the cache myself) and put whatever files I want to be sped up in the cache? 2) Is the feature still just a thought because you have more important things to focus on, or might there be some technical difficulties?

 

I'm not very fluent with English, so it might seem I'm complaining why things don't work the way I want them to, when, as always, I actually aim to understand the implementation down to the specifics while explaining how I currently (mis)understand them, so others may correct me where I'm wrong and fill the blanks where I'm at least partly on right path 

 

(3TB moved to the array, 7TB to go. So far pretty happy with unRAID! )

[primeval_god]

8 minutes ago, jpeltoniemi said:

Thanks for the clarification. Certainly not how I imagined it would work. Can someone tell me what are the advantages of having a rule of one copy of a file in array and cache vs what seems to be actually happening, i.e. some kind of layered file system with the cache drive as the top layer?

I couldnt say for sure, because i am no next to nothing about how the filesystem actually works, but my guess would be that it is less of a rule and more a natural extension of how the file various disk file systems are fused/layered together. One thing to note, and anyone else please correct me if i am wrong, is that the cache system works only on complete files. Once a file has been moved (by the mover) from the cache to the array, any future writes to that file bypass the cache drive and go directly to the parity protected array. Unless the file is deleted and then recreated of course.

[JorgeB]

42 minutes ago, jpeltoniemi said:

1) Can I potentially break something now or in the future, if I disabled the mover script(and handle flushing of the cache myself) and put whatever files I want to be sped up in the cache?

You don't need to disable the mover, just set that or those shares to cache "only" and the mover won't touch them.

[JonathanM]

There is no rule that forbids multiple copies of the file to exist. For example...

/mnt/disk1/share1/folder1/file.txt and

/mnt/disk2/share1/folder1/file.txt both are mapped to

/mnt/user/share1/folder1/file.txt

If you edit /mnt/user/share1/folder1/file.txt, only /mnt/disk1/share1/folder1/file.txt is changed, and the copy on disk2 is still the original file. That can create some serious confusion, so generally you don't want to create that situation on purpose, but there is no rule that forbids it. The user share system concatenates all the root folders with the same name on the disks into a single share folder, so you can access multiple disks under one folder tree. However, when the user share system encounters a naming collision like I laid out, only the first instance found is shown in the user share. The cache disk isn't special in regards to the user share system, it participates as an equal to the array disks.

 

The question that hasn't been answered here, and only because nobody has come forward after trying it to see, is whether /mnt/cache is parsed for user shares before /mnt/diskX, and thus would take precedence, or parsed last in which case the the diskX copy would be shown in the user share tree.

 

It would only take a minute or two to find out, it's not complex.

 

Your questions seem to be trying to parse out a much more complex system than actually is in use. It's really just as simple as I just laid out, it's not really a "layered" file system.

[trurl]

While something like your idea has been a feature request, it may not have seen much demand because people have been able to use the existing features for their specific purposes. You can already have user shares that remain on cache, and you can have redundancy of the cache pool. Normal HDD read speed in unRAID is just that, only write speed is affected by parity. And normal HDD read speed is usually enough for 1GB ethernet.

 

For example, I have several user shares that stay on cache (cache-only or prefer). All my other user shares don't use cache for speeding up writes (cache-no). These all write directly to the array because they are mostly unattended downloads and backups, so write speed is unimportant.

 

My user shares that remain on cache are for applications such as dockers, but I also have files that are accessed frequently (music for network players and pictures for screensavers). I keep these on SSD cache so no HDDs have to spin up. And my DVR recording goes here also since I may or may not want to keep these and I can write them while reading others that have already been recorded.

 

And there is the ability to mount drives (and network shares) outside the array. I mostly use this feature for creating backups that I keep offsite, but it is also handy for copying files from other sources.

 

Of course others will have different ways they use these built-in features.

[jpeltoniemi]

5 minutes ago, jonathanm said:

There is no rule that forbids multiple copies of the file to exist. For example...

...

The cache disk isn't special in regards to the user share system, it participates as an equal to the array disks.

 

The question that hasn't been answered here, and only because nobody has come forward after trying it to see, is whether /mnt/cache is parsed for user shares before /mnt/diskX, and thus would take precedence, or parsed last in which case the the diskX copy would be shown in the user share tree.

This will save me a lot of time when I start testing the cache. Thanks!

 

10 minutes ago, jonathanm said:

Your questions seem to be trying to parse out a much more complex system than actually is in use. It's really just as simple as I just laid out, it's not really a "layered" file system.

I had a hunch there might not be in a real layered file system, but with the information I've gathered that was the least unsafe assumption I could make. Glad you cleared that up. I do hope someone here could tell me what the process of  taking multiple drives and exposing them as a single merged share consists of. I suppose most work here is done using 3rd party software and perhaps some custom scripts. So, if you know what software is used or paths to any relevant scripts or configs, please tell me so I can focus my attention on them and stop speculating. So far I've managed to find out that shfs is used somehow, not much else.

 

[trurl]

3 minutes ago, jpeltoniemi said:

I suppose most work here is done using 3rd party software and perhaps some custom scripts.

Limetech is of course the one with the definitive answers here.

 

As you have probably already discovered, this is slackware linux. As far as I know there isn't really any "3rd party" except open source. The unRAID specific code is proprietary. I think this includes both the parity and user share implementation.