July 13, 20178 yr What it'd do: maintain a catalog of all the content on the server - files, folder, shares etc. The catalog would be built on initial install. After that it would hook into the unRaid filesystem and get updated automatically on any file modification/access, without needing to run a separate script on schedule. Thus it'd be like updating parity in realtime vs doing it on schedule. Features it'd provide to the user - - instant search of any file/folder on the server: would list the locations (disk, share) where its located, without having to search - if a disk goes offline, can generate a report of all missing content and all affected shares/folders - various stats such as most used, age, size map, distribution - possibly a realtime view of shares/disks with file access lighting up sections The key to all this is automatic updates of the underlying catalog by hooking into the file system, which I believe is shfs. IMO this would be a very powerful addon. I don't know if APIs exist to enable this, I've never written a plugin, but am interested.
July 19, 20178 yr Love the idea - since besides all the benefits you already mentioned, it could also replace the cache dir plugins, as all queries short of the actual files can be served from that catalog (like browsing the directory tree of shares without spinning up any array drives). The cache drive would be a candidate to store that catalog, and as that is typically a SSD drive this would be very fast as well.
July 20, 20178 yr Author Thought no one had seen this. Can I get some input from the regulars/unRaid authors if this is possible, and what it'd take. Yes it'd be more powerful than cache_dirs. Also this kind of info is very well suited to be kept in memory for instant access. The more I think of this the more uses I can come up with - e.g. you keep a history of the catalog and you have an instant view of how your server usage has grown over time.
July 21, 20178 yr I do not think this is that practical as it is likely to involve some fundamental changes within Linux core code to be that useful. UnRAID tries to avoid that sort of change.
July 21, 20178 yr On 7/19/2017 at 3:52 AM, Videodr0me said: Love the idea - since besides all the benefits you already mentioned, it could also replace the cache dir plugins, as all queries short of the actual files can be served from that catalog (like browsing the directory tree of shares without spinning up any array drives). Thats not feasible when dealing with external clients. Only custom tailored queries could be directed to the catalog. Clients like windows systems wouldnt be able to be switched.
July 22, 20178 yr Author On 7/21/2017 at 3:50 AM, itimpi said: I do not think this is that practical as it is likely to involve some fundamental changes within Linux core code to be that useful. UnRAID tries to avoid that sort of change. How is this a core change? In fact it'd be no change at all if the right api hooks exist. I looked into this a bit more and inotify seems to provide this info, and its part of slackware. Are there any unRaid authors on this forum or is it just users?
July 23, 20178 yr 1 hour ago, MrCrispy said: How is this a core change? In fact it'd be no change at all if the right api hooks exist. I looked into this a bit more and inotify seems to provide this info, and its part of slackware. Are there any unRaid authors on this forum or is it just users? How is it not a core change? Kindly point out all the technical kernel hooks you feel are straight forward. Inotify does nothing for intercepting reads of files and replacing the contents with different data.That is what you're asking for, replacing core filesystem directory handling with something else. The only possible place it might be put in place is the user filesystem "shfs". For all other accesses outside of /mnt/usr/, it would require making major fundamental changes to intercept and replace accesses for reads and shadowing writes. At this point you're replacing the existing user filesystem with your own. I have worked in Linux and have done Kernel development way back in the day. Yes, it is a core fundamental change.
July 23, 20178 yr Author Sorry I am not a Linux expert, let alone a kernel dev, and I read a bit more about inotify and take it back. It seemed at first glance to provide a callback for every file operation but its only for specific folders. I don't know if Linux itself has this capability. e.g. NTFS has a journal - https://en.wikipedia.org/wiki/USN_Journal - which can be easily queried by apps and is used for this kind of functionality. Its used by many programs for things like versioning, file history etc. I believe OSX had similar features which were used by Time Machine, and the new APFS is probably much more sophisticated for this feature. This seems like a pretty basic Linux feature - e.g. how do iotop/lsof etc operate? I am not asking for something to intercept file read/write calls, examine the data etc. That would be a very low level filesystem driver. Just a way to get notified when a file is created/moved/deleted - basically whenever its metadata has changed. which is a LOT less frequent operation.
July 23, 20178 yr How would something like qnaps qsirch using the tf-idf algorithm work with unRAID? Do you think something like this could be made for unRAID? It would be very useful to be able to have a fast search.
July 23, 20178 yr Actually Unraid already uses the FUSE filesystem as it's main core component. And one of the reasons to have a filesystem in userspace is to have flexibility in how to handle it's operations. @MrCrispy: You might look here https://github.com/libfuse/libfuse. Especially the passthrough example might give you a good starting point: https://github.com/libfuse/libfuse/blob/master/example/passthrough.c But I guess you will need some support by the unraid dev team to make this work - it is far from a trivial undertaking.
September 22, 20178 yr On 19.7.2017 at 9:52 AM, Videodr0me said: Love the idea - since besides all the benefits you already mentioned, it could also replace the cache dir plugins, as all queries short of the actual files can be served from that catalog (like browsing the directory tree of shares without spinning up any array drives). The cache drive would be a candidate to store that catalog, and as that is typically a SSD drive this would be very fast as well. Just curious... does SSD involve cloud computing? Edited September 22, 20178 yr by carolyn6
Archived
This topic is now archived and is closed to further replies.