September 25, 200817 yr Wget is not in the standard unRAID distro (last time i looked). We need something else (curl, an awk script) to do it, and it has to be something in the stock unRAID distro. With that, unMenu could include package management. .... and a few select kernel modules enabled, such as hardware monitoring and SYS V IPC. Config files are all symlinked back to the flash so they are persistent. It has a larger memory footprint now, and I am trying to reduce it by symlinking large libraries and such back to a supplementary location on the flash. I've quoted this conversation because in review, there are probably a few packages and kernel features that should be re-enabled. In effect, maybe we need to make a request for additional packages to be part of the standard unRAID distribution. I've always felt that curl or wget should be part of the distro. Curl probably being better because there are libraries that other programs can use. I also think ctorrent would be worthwhile if lime-tech wants to save bandwidth and use the community to help distribute the product. From conversations on the board I read ethtool, lspci, lsof. >>> and a few select kernel modules enabled, such as hardware monitoring and SYS V IPC. And I agree here SVS V IPC and kernel modules for hardware sensing (Even if they are just compiled, left off the distribution but made available as a separate addon package). In conclusion I see Kernel modules to enable. SYS V IPC, Hardware Sensors. PAE? CPU FREQ? Packages or tools to add to the stock unRAID. ethtool, lspci, lsof, curl (or wget), Packages for consideration ssh (over telnet), as it will give us ssh login, ssh command line spawning, sftp, and scp. I find it cumbersome to be on my unRAID system and not be able to push files off it.
September 25, 200817 yr I think connectivity/networking should be prioritized and that for all platforms not just windows or linux. Avahi should be available in the stock unRaid. Many would argue for afp as well, even if it is mac specific. Roland
September 25, 200817 yr Author what's avahi?? never used it, have not even heard of it, nor have my colleagues. UPDATE: Seems to be available as a third party add on http://sotirov-bg.net/slackpack/download.cgi?id=232
September 25, 200817 yr I want a built in web server... I currently use Apache, but it is a bit bulky. Does anyone have a suggestion for a light-weight http server? I know of several, but I'm interested in people who have ACTUAL EXPERIENCE with a particular one.
September 25, 200817 yr Author I've used lighthttp, It's not as bulky as apache, and is pretty capable. I installed it for the SCGI interface so I could have a web front end to rtorrent. (of which I should be releasing the binaries soon). I've been able to compile webfs in a static mode so it was one simple binary (for static content only). To add more to this, I've found a number of very simple http servers with basic CGI interfaces for serving content. It's the whole CGI/PHP thing that starts to make the webserver heavy. I would suggest a review of lighttpd, There are many appliances that use it My friend uses it in a webserver farm to serve static content. Comedy central has a thanksgiving contest every year. They use lighttpd for the static content (due to speed and lightweight) and apache for the dynamic content.
September 25, 200817 yr I use lighthttpd as well in a production environment. Its not the smallest out there but dev is strong and its WAY lighter than apache.
September 25, 200817 yr Author I'll add a prior request which was to include/compile the pc speaker driver for auditory alerts in unRAID.
September 25, 200817 yr what's avahi?? never used it, have not even heard of it, nor have my colleagues. UPDATE: Seems to be available as a third party add on http://sotirov-bg.net/slackpack/download.cgi?id=232 there are various dependencies for avahi to run and I was not able to get it working. Roland
September 26, 200817 yr Static content is nice, but CGI via PHP is a must-have ... there are many apps I use, such as apcupsd, nzbget, etc., that have php-based web interfaces. A number of media serving packages also need a web server w/php-cgi. That said, I was able to configure lighttpd to use php-cgi (fast-CGI) using socket communications.
September 26, 200817 yr Author I think it best to consider bare and supplementary. bare being bare binaries that we feel are a must, for example certain kernel modules. supplementary being items we can add ourself but would be nice to have installed already. Perl, php, python, lighthttpd being of the supplementary type. I would put screen and hddtemp in this because we can compile and add it ourself via a package load during boot. Kernel being ======== cpuidle/power/freq support. (not sure of the detail here) S3 support. SYSV IPC FS Support (what is missing from what I saw, modules fo rmost popular do exist). Sensor support. pcskpr ALSA Support. PAE? Tools being ========= lspci lsof beep (for speaker control). latest smartctl some form of off host mail sending tool lhddtemp (I'll add this but I think smartctl is enough to get what is needed) I see ssh as being important to help login, move data and automate remote commands. >> Static content is nice, but CGI via PHP is a must-have ... I agree, yet, lightttpd support, can go both ways, but I really think it's supplementary. At the very least being able to browse our hard drives via browser is a good thing. php could be added. just by the installpkg command. Each scripting language adds more weight to the initramfs. Might be worth devising an install to a different root, then some form of symlink back to root (i.e unless we could get unioning to work well).
September 26, 200817 yr You are forgetting dependencies... if you want lm-sensors, you need perl. PHP can be lightweight if you leave out the bloat modules, like MySQL access. Plus, if you think unmenu is nice, the same concept done in PHP in a real browser would leave it in the dust. Consider this, unmenu is universally useful because awk is in the unRAID stock distro. If php and a web server were in the stock, you'd have the same universality with much, much, more capability. As for filesystems, ext and ex3 support are missing.
September 26, 200817 yr Author You are forgetting dependencies... if you want lm-sensors, you need perl. I was aware that perl was needed, but it's only for the initial detect. /usr/bin/sensors is a binary (at least on my system). Plus, if you think unmenu is nice, the same concept done in PHP in a real browser would leave it in the dust. I do agree here. PHP can be lightweight if you leave out the bloat modules, like MySQL access. I guess my goal was to try and gather the bare minimum so we can present a list to Tom and enable the rest of us to build further. Without some bare minimum kernel support, some enhancements become much more complex in a build and support role. For example apache requires the sysvipc as previously mentioned. If emhttp did at least cgi, then we would be able to use that to build upon using either perl, php, python which can be added on demand by the user. With PHP and it's bloat modules, do we want lime-tech to continually maintain a customized scaled down version of PHP? My thought was with if there was too much layered addons, it may slow down releases. I could be just thinking about this tooo much I figure the first go round is bare minimum kernel support and some tools.
September 26, 200817 yr With PHP and it's bloat modules, do we want lime-tech to continually maintain a customized scaled down version of PHP? No need to... it is already packaged that way. This isn't a production web-server... the web functions are utilitary, so no need to hop on minor PHP updates every week.
September 26, 200817 yr Plus, if you think unmenu is nice, the same concept done in PHP in a real browser would leave it in the dust. <sniff> my feelings are hurt... the unmenu server is a prototyping tool. No way it will compete with a full fledged web-server, but it is very easy to create an add-on, and that was the goal... Yes, we can write an equivalent "php" interface, and in fact, most of the code is almost exactly the same... although php does insist on the semi-colons at the end of every statement, so the hard part. figuring out how to access the data we want to see is done. Converting to a different language, a mechanical task for anyone interested. Consider this, unmenu is universally useful because awk is in the unRAID stock distro. If php and a web server were in the stock, you'd have the same universality with much, much, more capability. other than multi-threading, and the ability to return huge binary objects, not so sure. Apache still needs to deal with the fact that we don't usually have a swap partition. Yes, it would be easier if "awk"had a true "<include>" capability... but we are doing pretty well without it. What do you have in mind with the "Much, Much more capability" you referred to? Now, don't get me wrong, there are lots of packages that pre-exist who use "php" . We need the ability to parse "php" to handle them. I do not intend to write a ""php" interpreter in awk... (but I did not intend to write a web-server either ) Many of us have original unRAID servers... with 512 Meg of ram... got to keep in mind the basic distribution must live on the existing flash drive, with the existing ".key" file from Tom. In my case it is a 1Gi drive, but many still have 256Meg and 512 Meg flash drives running "Pro" and "Plus" versions of unRAID. We need to be able to handle them... Maybe if we boot off of another flash drive labeled "Bubba" and have the key file on the "UNRAID" labeled drive... </sniff> Ok, now I'm over it. ;D I'd love a full Slackware distribution, with development tools, compilers, etc... but it ain't gonna fit on a small USB flash drive. Joe L.
October 12, 200817 yr What do you have in mind with the "Much, Much more capability" you referred to? Joe, I know you love awk, but you have to admit it is a kludge as a web application development platform and substitute for a web server. I wouldn't touch it except for a few situations where it is useful in shell scripts or a management back-door. Development of plugins for unRAID in awk is a compelled choice.... like one of those old tests where they tell you that you are stranded on a deserted island with only a handful of items, and you have to figure out what you can do with them. Not unlike when Bogart bent the prop shaft in The African Queen or after the explosion and fire on Apollo 13. You make do with what you have rather than giving up, but if you were back in civilization, you'd be crazy to do the same things. Of course, I used to use archie and gopher, and didn't see much advantage to Mosaic... I'm sure someone can rewrite existing applications like wTorrent, Slimserver, etc., in awk ... but if you have a plain old web server, like lighttpd, with php, perl, etc., you can drop in all kinds of already-written applications. You can't to auth with awk. Awk is not running as a web server... it is just responding to socket requests by creating dynamic text output.... not serving up html files. It doesn't even serve up images. Again, on a deserted island, you make do with what you have..... Many people use unRAID as a media server, and a lot of the media management applications need ... wait for it... a web server and one of the big-3 cgi languages (php, perl, python). For serving up bandwidth-hungry hidef files, nothing is faster than HTTP for streaming. It comes down to using the right tool for the job. I've coded projects in over 20 different languages, and yet in a pinch, I've used Lotus as a word processor because no word processor was available. You can download files using native awk... so why use awk as a bootstrap to download wget, and then use wget? Because wget is a much better tool for the job. None of this diminishes the value of awk, or the exceptional application that unMenu is. But a lightweight web server (list at http://en.wikipedia.org/wiki/Tiny_web_servers) and a judicious install of scripting languages, makes a much more powerful and versatile platform.
October 12, 200817 yr Author Agreed, it seems unRAID is attracting a more intensive user that wants the electricity paid for, to serve more. The cost will be memory (and maintenance).
October 12, 200817 yr Development of plugins for unRAID in awk is a compelled choice.... like one of those old tests where they tell you that you are stranded on a deserted island with only a handful of items, and you have to figure out what you can do with them. Not unlike when Bogart bent the prop shaft in The African Queen or after the explosion and fire on Apollo 13. You make do with what you have rather than giving up, but if you were back in civilization, you'd be crazy to do the same things. Of all the posts from this great community that I can remember, this one definitely gets my vote as the most poetic! Moreover, the point is extremely valid.
October 12, 200817 yr What do you have in mind with the "Much, Much more capability" you referred to? Joe, I know you love awk, but you have to admit it is a kludge as a web application development platform and substitute for a web server. I wouldn't touch it except for a few situations where it is useful in shell scripts or a management back-door. you apparently did not understand what I had written in my prior post. I think of unMENU is a prototyping tool. Some of the features developed there will eventually work their way into the stock unraid package, most will not. It does not pretend to be a full featured web-server, but it sure comes pretty close for most "prototyping" needs. My comments were all tongue-in-cheek. None of this diminishes the value of awk, or the exceptional application that unMenu is. But a lightweight web server (list at http://en.wikipedia.org/wiki/Tiny_web_servers) and a judicious install of scripting languages, makes a much more powerful and versatile platform. I won't argue that a full-fledged web-server isn't better. When we get one as part of the stock unraid, I'll be the first to use it. If I have a personal need, or an application that must run on top of a true multi-threaded web-server, I'll have Apache, or something that will handle the needs quickly installed on my server. I have the skills to install it, configure it, and write applications for it. Unfortunately, asking a newbie to write a "php" page to perform a simple management task on their server is beyond their level of programming skill. For some, editing a script is a major task... When we provided a simple "plug-in" capable architecture, all kinds of ideas were voiced. That will continue. The ideas will build on each other. The biggest advance that has occurred in the unmenu.awk project is not an awk-script, but the ideas on how the unRAID user-interface can be improved. Obvious to me is we need a plug-in/cgi interface to a web-server. If we have one, it will be used in a multitude of ways. It is also obvious the interface must be workable by an appliance operator. Most users are NOT unix experts... heck, most users are just interested in storing their media collections. I can program in any language... it is not that I love "awk," but rather, it is available and very powerful. I wish it could be multi-threaded, but I can't have everything. I wish I could set error traps, but it can't. I'm having fun seeing how much I can get it to do. As I said, I'm a bit unusual... I can work in hardware, firmware, or software. As often as not, I'm asked to work and lead 4GL projects where I've never seen the syntax. My very first "web" project had over 350 active content web-pages and replaced two mainframe applications dealing with workforce management and staffing. (It used MS-IIS, and "asp" as its back end. Javascript and an early predecessor to "ajax" for true interactivity with the web-server. ) Yup... no experience with "asp" or javascript, or even much htm.... and no "awk." I instead wrote a "shell script" to "write" "asp" and SQL... yes, it wrote complete working web-pages in skeletal form, with affiliated PL/SQL packages. In 5 minutes my code generator could get a complete add/update web-page page running, with comments, including one that said "business logic goes here" I frequently have no choice on the language for a project. Usually, it is a decision made by some upper management person who has never even seen the inside of a server. So, to use "awk" to build a web-server was an interesting challenge. Interestingly, when I started this project, I did not know that "bash" also had networking built-in. I can give it a try at some point. It might be even more versatile.. It does have multi-threading, and traps, and signals, about the only thing missing is associative arrays. Yes, I'm crazy... I could have had a true web-server if I was "back in civilization" But... unraid is exactly like being on that deserted island, with a tiny toolbox., but the ability to order any tool we would want from a catalog. I'm sure I would have been very much like the "Professor" on Giligan's Island. he made anything out of a couple of coconuts and a few odds and ends salvaged from the "minnow" Wait until you see the new "Package manager" plug-in in action. I've merged in the download code and it is nearly ready. I need to add "md5" checksums, and a bit better error handling on dead download links. I now have it downloading, installing, and creating/removing "auto-install" scripts, so the package can be installed automatically on every reboot. It is verifying the versions of application packages, and presenting appropriate options. It will greatly simplify the task of installing one of the lightweight servers you mentioned earlier. Remember, unmenu.awk is the rapid development environment prototype tool... not the end-all to every need. Clearly it does not serve all your needs or bubbaraid would not exist.. All I expect unmenu.awk to do is to provide a utility or two currently missing from unRAID and give Tom some ideas on what he might include in the stock unRAId product. It has greatly exceed my initial expectations. As you said, there are many packages that expect the existence of a full fledged web-browser. We will need one to accommodate them. BubbaRaid is one solution. I've not yet installed BubbaRaid, but only because you have indicated it is not compatible with the current "beta" version of unRAID. Once you catch up to the current release I'll install it too. I'm quite happy with the "limited" toolset at this point... I'm targeting ideas, not specific syntax or programming tools. The user of "BubbaRaid" is more likely the opposite. They are not targeting ideas, but interested in specific applications that are dependent on an infrastructure BubbaRaid can supply, but stock unRAId cannot. Both have their place. Joe L.
October 12, 200817 yr I think we stand more chance of Tom including a small set of extra tools for us if we can keep them short and focussed akak as much bang for the buck as possible: Web server: speaks for itself. So much can be done with this. I would suggest lighthttpd as the best option between size and flexibilty. Alot of tiny webserver projects dont get maintained, lighthttpd does. Wget: opens up possibility of trivial extra addons without any fancy footwork Perl: uber power, simple to use, simple to learn language with countless scripts already in existence. Ok so im saying what has been said before but if we can get these 3 things we can do so so so so much. If we had these 3 things tomorrow, lets face it we would be releasing scripts within days. Essentially what im saying is we as a group stay focused and ask for these 3 things and only these 3 things this now. We have more chance if we do this.
October 12, 200817 yr NAS, keep size in mind. Perl is HUGE.... over 40 MB... I had to split it out of BubbaRaid because it was so large (now it lives on the flash, not the initramfs) PHP is not huge, but it ain't small either (12MB) Python is 65M ... I gave up on it. Now I'm sure I can trim all of these some... I live w/o man pages on unRAID since they are easily (and arguably better) available on the web.... I just need to spend the time to optimize them.
October 12, 200817 yr Joe, I agree completely with your sentiments. One thing, don't get hung up on "multi-threading" as many powerful web servers, like Lighttpd, are not multi-threaded. Lighttpd runs some of the most heavily loaded web servers in the world, such as YouTube. Although it is an excellent file server, I was not satisfied with stock unRAID... I saw it as a criminally underutilized box, wasting uptime in the face of other "always on" boxes I have for other tasks. My goal was to eliminate all the other boxes, and collapse to one. That goal required a custom kernel, which in turn required modifying the initramfs. Since I had to roll up my sleeves and do it, it opened up a lot of opportunity to do other things, and I ended up just running unRAID on a complete Slackware development system booting from HD and not flash, and I still do for my production system. There are some things that simply can't run on unRAID w/o kernel mods... once I decided to bute the bullet and do what it took to run a custom kernel and initramfs, I was, in essence, deciding to haul the boat back to civilization so I had a full shipyard at my disposal for upgrading it. I'm excited about the prospect of an overlay file system or a pivot_root in the future. I also see a capability for unraid to bootstrap itself into an application platform, like this: - keep the initramfs small - after the array is started, the user can select an array drive for the "application server" to be installed on - a file-based filesystem is then be downloaded and stored on the array drive or cache drive - on subsequent boots, the initramfs will either - pivot_root to the file-based filesystem... - or it could just be an additional mounted filesystem (but that would require a lot of customization or symlinks) - or an overlay filesystem (not yet stable or in the main linux kernel) The PopcornHour works on this model .... it performs its core functions (media player) right out of the box. If you want the NZBGet, torrents, web server, etc., you install a hard drive, and navigate to the menu option to install the NMT Apps, the filesystem image is downloaded, and then you have an application platform booting.
October 12, 200817 yr Author NAS, keep size in mind. Perl is HUGE.... over 40 MB... I had to split it out of BubbaRaid because it was so large (now it lives on the flash, not the initramfs) PHP is not huge, but it ain't small either (12MB) I know perl can be large. I've been considering the addition of just miniperl in the stock unRAID. It's just the perl interpreter without all of the suport packages. Allot can still be done with it, opening files, parsing, etc, etc. It would only be 1-2MB. bash-3.1# ls -l --si /boot/custom/bin/miniperl -rwxr-xr-x 1 root root 1.2M Aug 22 14:45 /boot/custom/bin/miniperl I think perl can get down to about 32M if you delete al the man and doc files. There also seems to be a number of binaries in usr/bin that probably would not be of use. bash-3.1# du -hsx / 178M / Executing install script for perl-5.8.8-i486-6... bash-3.1# du -hsx / 215M / bash-3.1# du -hsx /tmp/perl 38M /tmp/perl bash-3.1# rm -rf /tmp/perl/usr/lib/man bash-3.1# du -hsx /tmp/perl 33M /tmp/perl bash-3.1# rm -rf /tmp/perl/usr/doc bash-3.1# du -hsx /tmp/perl 32M /tmp/perl I bet if this were on a cramfs it might be better. Not sure if it's worth the effort. bash-3.1# du -hsx /tmp/perl/usr/lib/perl5/5.8.8 28M /tmp/perl/usr/lib/perl5/5.8.8 As far as putting the interpreted languages on the flash. Good idea.
October 12, 200817 yr Author I'm excited about the prospect of an overlay file system or a pivot_root in the future. I also see a capability for unraid to bootstrap itself into an application platform, like this: - keep the initramfs small - after the array is started, the user can select an array drive for the "application server" to be installed on - a file-based filesystem is then be downloaded and stored on the array drive or cache drive - on subsequent boots, the initramfs will either - pivot_root to the file-based filesystem... - or it could just be an additional mounted filesystem (but that would require a lot of customization or symlinks) - or an overlay filesystem (not yet stable or in the main linux kernel) The PopcornHour works on this model .... it performs its core functions (media player) right out of the box. If you want the NZBGet, torrents, web server, etc., you install a hard drive, and navigate to the menu option to install the NMT Apps, the filesystem image is downloaded, and then you have an application platform booting. Agreed 100%, this is why I was exploring the unionfs and autofs along side the GIGABYTE I-RAM disk. What I did learn is you cannot pivit_root off the initramfs (incase you want to play). You have to use the busybox switchroot instead. pivitroot works with initrd, switchroot works with initramfs. It took me many hours trying to figure this out when olaying with the autofs module.
October 12, 200817 yr Yes, Busybox is also how the PCH does it.... you used to could to a pivot-root with older unRAID before Tom switched to an initramfs from initrd. I prefer a separate FS, not overlayed.... this is safer and more self-contained... however it is more work on the app packager, since you usually can't just run installpkg... you have to compile apps from sources with config options to live in a place other than their usual home. Perl is fairly self contained... you can symlink the /usr/lib/perl5 (where 31 of the 32 MB lives) to a different FS
Archived
This topic is now archived and is closed to further replies.