jpeltoniemi

I was just beginning to think that shfs in unRAID maybe isn't what can be found on Google. Good, now I can stop trying to make sense how that shfs fits into this equasion I'll forgive myself with this one since with Greyhole you actually mount storage pools using CIFS. Nice! I guess this means I can put whatever I want in cache and it'll just work, as long as I make sure that mover won't touch them. There shouldn't be a problem even if I disable mover and write my own. TBH the docs could be better. If I may suggest a quick improvement, just a list of essential keywords, config files and scripts would be a big help in getting started with tuning unRAID. That said, I have to commend your involvement with the community. Kind of makes me want to use my last money to buy a license right now instead of waiting for payday

Yeah. I'm mostly used to Debian-based distros but luckily Slackware is not nightmarishly different from Debian. 3rd party, as in not in-house code. Hopefully with documentation somewhere. I'd assume(like that's done me any good so far) that parity is Limetech's own implementation, but not so sure about user shares, as software like UnionFS exist. Maybe they didn't fit unRAID's requirements or maybe Limetech want to reinvent the wheel just for fun(I do this all the time). Let's hope someone has more accurate information

This will save me a lot of time when I start testing the cache. Thanks! I had a hunch there might not be in a real layered file system, but with the information I've gathered that was the least unsafe assumption I could make. Glad you cleared that up. I do hope someone here could tell me what the process of taking multiple drives and exposing them as a single merged share consists of. I suppose most work here is done using 3rd party software and perhaps some custom scripts. So, if you know what software is used or paths to any relevant scripts or configs, please tell me so I can focus my attention on them and stop speculating. So far I've managed to find out that shfs is used somehow, not much else.

Thanks for the clarification. Certainly not how I imagined it would work. Can someone tell me what are the advantages of having a rule of one copy of a file in array and cache vs what seems to be actually happening, i.e. some kind of layered file system with the cache drive as the top layer? So far I haven't been able to come up with any good reason to spend resources enforcing the rule, when to a client it's all the same if the file is served from either the array or cache, as long as it's the most recent version. This accident of implementation is partly what I don't get. This should be more or less how layered file systems work and I see no reason to actively enforce any no-duplicates rule. @limetech Thanks for responding. I have a couple of questions cooking for you, but I'll get more comfortable with unRAID first so I wouldn't waste your time with anything too dumb. However, if you have the time, feel free to comment on the current discussion about how cache works. Also, two quick things: 1) Can I potentially break something now or in the future, if I disabled the mover script(and handle flushing of the cache myself) and put whatever files I want to be sped up in the cache? 2) Is the feature still just a thought because you have more important things to focus on, or might there be some technical difficulties? I'm not very fluent with English, so it might seem I'm complaining why things don't work the way I want them to, when, as always, I actually aim to understand the implementation down to the specifics while explaining how I currently (mis)understand them, so others may correct me where I'm wrong and fill the blanks where I'm at least partly on right path (3TB moved to the array, 7TB to go. So far pretty happy with unRAID! )

I'm not sure I understand. Say I copy movie.mp4 to the server. It ends up in cache until the mover script saves it to the array. So far, so good. Now, if I notice this was the wrong version of movie.mp4, be it incomplete or has errors, what then? If a file can not exist on both cache and array simultaneously, doesn't it mean that the file in the array has to be removed, or that cache will be skipped and movie.mp4 will be written straight to the array much slower? I always thought that the array and cache may contain the same file, and in that case the cache file is always used until it is moved into the array, replacing the older file version. Just registered to start this thread. Thanks for the info. I guessed as much that Limetech is a small company - partly why I'm posting here and not bothering them directly just yet Having been a small time entrepreneur I understand how much work simple client relations can be.

Of course, that much is expected. I meant it like would it be reasonable to expect some kind of reply from the developers, if pinged, in the next week or so, or if the thread stays relevant? I'd very much like to avoid emailing Limetech directly and instead keep the discussion public, so others can take part in the discussion. This thread is getting a bit sidetracked right now, so I think we'd best get back on topic from here on I'm probably gonna set up unRAID this evening, when I get my drives prepared. After that I'll look into the mechanics of caching and try to wrap up some more concrete questions. If that goes nowhere, I'll just have to contact Limetech directly.

A reasonable assumption I'm a programmer and manage a 6 hdd home server, so I'd like to think I'm not completely incompetent when it comes to basics of NAS or unRAID. Gotta admit though, at first I was going to try having my primary machine run unRAID with virtualized GPU, but I later found out my CPU doesn't support vt-d. Whoops. About your link... I'm currently reading it through. So far I've seen only one post by Limetech, not addressing the original topic. The problem with old threads is that they're often outdated, and sooner or later either die or change topic, which seemed to be the case with this one. I'll try to look for more, but I'm not feeling very optimistic. I guess I'll just have to try to get as far as I can by myself and ask questions when I actually encounter a problem.

I've seen someone mention that games or apps shouldn't be run from the array. I'd like to know why, though. Does it have something to do with access times, possible data corruption, or something else? I'm genuinely interested in the underlying reasons and my thirst for knowledge can't be satisfied with just "it's a bad idea". If it's just about read performance, this topic is kind of about tackling just that. Many apps and games rarely write into their own directory but instead use AppData and My Documents, so the apps in array would mostly just be read. I know about unassigned disks and I've considered the feature for various purposes, but in the end I'd really like just set it and forget it. I got into disk pooling and unRAID in order to avoid having to think which drive should I put my stuff on today. While I can't avoid unassigned disks for certain specific tasks, I'd like to try to get maximum conveniency and benefits from the array. Sorry, I must have not been clear enough. I want to try, if feasible, create a background service that monitors the most used files and puts them on the cache drive for faster access in the future. In theory this should be doable, but I need to learn more about the implementation and possible caveats, which is why I made this topic. I already know that cache is overlaid on top of the array in the share. What I really wanted to know, how the overlaying is done internally, like how exactly does the server list files from the respective path in the array and the cache as one, how does it check for cache while determining which file to serve, and how does it handle conflicts when reading and writing the same file.

Is Limetech not active on the forums? I mean, sure, for extremely detailed topics that would be my first choice, but I would maybe ask on the forums for lighter information that may be common knowledge among tinkerers. If possible, I'd like to avoid bothering Limetech directly with such things. The following question(s), which I forgot to ask in the initial post, is something I suppose could be common knowledge: How does unRAID determine if a file from cache should be served instead of a copy in the array? Would it be possible to either interface with unRAID caching, in order to tell the system about new cached files, or does everything work automagically, i.e. copy files to appropriate path in cache and they become available? This is maybe going a bit off topic, but I was wondering what would happen if the mover script tries to move a file from cache that is either being read or being written to? And what would happen if a file is written to cache while it's already being read from the array? Sorry if my questions are a bit basic. I'm still preparing my drives and can't actually try to find these things out myself, and the both the Wiki and Google produce either answers that are too vague or nothing at all.

Hello all! I recently discovered unRAID through Linus Tech Tips and am currently preparing my drives for it. 12TB of files is getting a bit too much for Greyhole. From what I've read, unRAID doesn't utilize cache drives for accelerating often accessed files, which still is better to what I have now, that being no cache at all, but I was wondering if it could be even better. I'm not too familiar with unRAID internals yet, but AFAIK, it uses Samba to serve files. Wouldn't it be possible to to monitor which files are requested, either by hooking in to Samba(https://www.samba.org/samba/docs/current/man-html/vfs_full_audit.8.html) or to kernel(https://linux-audit.com/monitor-file-access-by-linux-processes/) and pass all the file requests to a cache server to process. Being a js developer, my first though was to run a Node.js instance with either Redis or MongoDB to store access to each file. The cache server would periodically check which files are accessed most often and copy them to cache, as well as remove less accessed files from the cache. The cache server could perhaps even be configured to look at currently accessed file names and pre-emptively cache potentially next-in-sequence files, or something like that. Ideally caching strategies could be pluggable. This kind of read cache would be great for application data(Steam libraries) and project files, which aren't always worth optimizing individually and can't all be stored on a stack of SSDs(unless you're made of cash). I'm sure there's something I'm missing, but if it isn't anything crucial and something like this is actually doable, my questions are as follows: Is there any plan to implement such functionality in unRAID natively / is there a timeline? Despite finding no good answers on Google, does a plugin for this already exist? If I were to take a shot at this, is there something I might've missed, underestimated or misunderstood? Anyone to consult when interfacing with unRAID? If I manage to wrap something up, any volunteers to help with the UI? Of course, if I have missed something (crucial or not), please tell me. Thanks