May 7, 200818 yr Before i suggest this let me say this is to simplify things for n00b users and to allow one more unRAID standard file (namely go) to stay standard. Rather than suggest users edit their "go" file I think there is a big advantage to modifying the standard go script to look for and run other standard scripts from a standard directory location. Now this may not seem like a big deal but essentially what we have now is a very basic plugin system for startup. Forum members can suggest enhancements (like the read performance one) and instead of saying things like "open up go and edit it... adding these lines"... we would just say download this file and place it in the plugin directory. Thought would need to be given to order of running but thats a hurdle not a show stopper. Also once we have this its not a huge leap to say teh community could do a auto go script plugin installer. Thoughs?
May 7, 200818 yr I've suggested this a number of times. In fact I already have my own frame work to do this. Here's my go script #!/bin/bash # Start the Management Utility echo trace > /proc/mdcmd;sleep 1 /usr/local/sbin/emhttp & fromdos < /boot/config/rc.local/rc.local_startup | sh Here is my script to run the plugin directory of scripts root@unraid:/boot/config# cat /boot/config/rc.local/rc.local_startup #!/bin/bash logger -trc.local_startup -plocal7.info -is "Initiating Local Custom Startup." for script in /boot/config/rc.local/* do scriptbase=${script##*/} # Strip pathname if [ $scriptbase = "rc.local_startup" ] ;then continue; fi if [ $scriptbase = "rc.local_shutdown" ] ;then continue; fi ( echo "Processing $script" fromdos < $script | sh ) 2>&1 | logger -t$scriptbase -plocal7.info -is # Old way not used. # fromdos < $script > /tmp/$scriptbase # chmod u+x /tmp/$scriptbase # /tmp/$scriptbase # rm /tmp/$scriptbase done # Setup shutdown script RCFILE=rc.local_shutdown if ! grep /boot/config/rc.local/$RCFILE /etc/rc.d/$RCFILE > /dev/null 2>&1 then echo /boot/config/rc.local/$RCFILE >> /etc/rc.d/$RCFILE fi if [ ! -x /etc/rc.d/$RCFILE ] then chmod u+x /etc/rc.d/$RCFILE fi logger -trc.local_startup -plocal7.info -is "Local Custom Startup Complete." Here's a directory of all my plugin scripts Notice how i use SVSV naming to insure they run in a particular order. rc.local_shutdown is a script that is hooked into the normal system shutdown. root@unraid:/boot/config/rc.local# ls -l total 200 -rwxr-xr-x 1 root root 148 Feb 20 15:29 S00-syslog-update* -rwxr-xr-x 1 root root 263 Feb 21 14:29 S01-blockdev* -rwxr-xr-x 1 root root 529 Feb 22 03:25 S02-sync-etc* -rwxr-xr-x 1 root root 189 Apr 6 19:34 S03-cpufreq* -rwxr-xr-x 1 root root 1528 May 5 21:36 S10-installpkg* -rwxr-xr-x 1 root root 464 Feb 21 14:05 S20-init.identd* -rwxr-xr-x 1 root root 2224 Mar 8 20:10 S20-init.mt-daapd* -rwxr-xr-x 1 root root 851 Feb 22 04:36 S20-init.proftpd* -rwxr-xr-x 1 root root 229 Feb 21 12:52 S20-init.rsyncd* -rwxr-xr-x 1 root root 584 May 4 15:01 S20-init.spincontrol* -rwxr-xr-x 1 root root 365 Feb 22 03:28 S20-init.sshd* -rwxr-xr-x 1 root root 433 May 5 21:31 S20-install-bind* -rwxr-xr-x 1 root root 608 Mar 8 00:23 S30-inittab-additions* -rwxr-xr-x 1 root root 557 Mar 8 02:25 S30-inittab-powerdown* -rwxr-xr-x 1 root root 554 Feb 22 04:39 S75-useradds* -rwxr-xr-x 1 root root 811 Feb 20 15:49 S80-update_hosts* -rwxr-xr-x 1 root root 129 Apr 13 11:35 S81-rsync_boot_cron* -rwxr-xr-x 1 root root 1784 May 5 22:37 S90-smb-shares* -rwxr-xr-x 1 root root 899 Apr 16 23:45 S90-tmpfs* -rwxr-xr-x 1 root root 79 Apr 13 11:28 S91-init.vfs_cache* -rwxr-xr-x 1 root root 276 Apr 13 11:30 S92-init.slocate* -rwxr-xr-x 1 root root 2050 Apr 7 21:53 S99-syslog-save* -rwxr-xr-x 1 root root 51 May 5 22:42 S99-z-system-up* -rwxr-xr-x 1 root root 2097 Feb 25 06:09 rc.local_shutdown* -rwxr-xr-x 1 root root 917 Feb 21 14:40 rc.local_startup* root@unraid:/boot/config/rc.local# The downside of my set up is, if there is some reason the array startup is delayed, things can get out of whack. We had a discussion a while back about calling special scripts when the array is stopped and started. http://lime-technology.com/forum/index.php?topic=1702.0 There were 6 states of use / run levels that would be helpful pre_array_start_onboot post_array_start_onboot pre_array_start post_array_start pre_array_stop post_array_stop So if emhttp called a special script at specific times with a run level as an argument ( 1 - 6, or text ) Then we can easily create a drop in / plug in directory system of scripts to run at a specific run level . I'm attaching my scripts for examples. There are many ways to do the scripts, but you have to be careful about configurations that already exist or clobbering other files. unzip this in a work area as it has my go script and the plugin direcfory.
May 7, 200818 yr Author fantastic stuff. perhaps we should anoint your system as the unofficial addon framework by documenting it in the wiki and recommending any other hacks become compliant with it? it would be nice to start normalising some of these "hacks" so that more than the hardcore user set can feel confident running it?
May 7, 200818 yr If Tom would allow unRAID to be redistributed (the non-gpl parts), then this would be pretty trivial to do. We could have several different community unRAID versions. This could be a standard feature in any such distro we made, or could be just yet another choice (i.e. we could have versions available that are exactly the same as stock except for the kernel, or could go all out with something like this). In this example, it'd make things easier for the end-user who wants to use any third-party add-ons (assuming those add-ons are compliant with this). Just download the community unRAID version, drop in some add-ons in a specified directory and there you go. I've asked Tom permission to redistribute the files, I haven't heard back one way or another yet (though he would reply to other parts of the same email, so they definitely weren't lost or flagged as spam). My guess is he's trying to decide whether this is a good idea or not, or simply waiting for 4.3 to become final. If/when I get the ok, I can package up some stuff if you want, I already have several goodies just waiting (think of a Slackware install disc that will automatically install unRAID for you).
May 7, 200818 yr Let's choose a directory structure / location we all agree upon for third part addons / plugins. Should the root be as /boot/custom/rc.d/ or init.d (addon scripts) /boot/custom/lib/packages /boot/custom/etc (for custom etc files that will be synced to etc back and forth). I did not put the rc.d under etc as it might normally be because I already have scripts which sync /boot/etc to /etc Should the custom root be called unraid ?? Such as in /boot/unraid/rc.d/(addon scripts) (init.d ) /boot/unraid/lib/packages /boot/unraid/etc (for custom etc files that will be synced to etc back and forth). I'm still a proponent for the 6 levels of unraid control So I would put them in an rc.d structure mimicking sysv startups. /boot/unraid/rc.1/ /boot/unraid/rc.2/ /boot/unraid/rc.3/ /boot/unraid/rc.4/ etc, etc, .... FWIW, this could also be on the cache disk in a hidden directory. I do plan to write some software so we can installpkg to a root on the cache disk. Then have the program handle all the symlinks back to /. This would let you install apache, php, mysql once and be done with it. But upon reboot, have all the linkage recreated on the root ram disk.
May 7, 200818 yr Author Everything you are saying makes alot of sense to me. It also makes alot of sense that if we can flesh this out get this structure and changes into the official unRAID distro. I dont think Tom would object to that since by itself it wont do anything. It also reduces the number of unofficial steps users have to do and simplify it for the less linux savy. I would really like Tom to jump in here incase hes got some plans of his own along these lines.
May 7, 200818 yr If you mean a slackware package that will install the unraid subsystem, then please start a new thread on that. I prefer to keep this thread on topic so we can hammer out a sanctioned structure.
May 7, 200818 yr Author WeeboTech you feel like starting a wiki page or even a seciton on this? Dont worry about formatting and niceness i can follow on your heels with my wiki broom to save you time
May 7, 200818 yr Sure... I'm still thinking about it a lil though.... I may just write a special C daemon that sits in memory and when it either receives commands or notices a change it state automatically runs the unraid run level scripts. (I just have not figured out how to watch for a state change). What I'm looking for is buy in by Tom and the community either for 1 run level on start up or multiple run levels as suggested by Josetnn. Right now we can only do the initial startup run level. Which if agreed upon, will work for now.
May 7, 200818 yr Sure... I'm still thinking about it a lil though.... I may just write a special C daemon that sits in memory and when it either receives commands or notices a change it state automatically runs the unraid run level scripts. (I just have not figured out how to watch for a state change). What I'm looking for is buy in by Tom and the community either for 1 run level on start up or multiple run levels as suggested by Josetnn. Right now we can only do the initial startup run level. Which if agreed upon, will work for now. You can send commands to the underlying MD device, but if you do, the unRAID emhttp process is out of sync and gets very confused very fast. We really need an API to emhttp, so it can be commanded either by the web-front end, or the API, or at a minimum, a signal we could send it ti get it to refresh its view of the system, regardless of what has happened. Perhaps one of the USR signals. (Pretty sure they still exist in Linux... they were there in System5.) I actually did a quick hack/shell script at using localhost to send GET requests to the web front end to control it. That worked until Tom added security and the management web-page needed a password.
May 8, 200818 yr You can send commands to the underlying MD device, but if you do, the unRAID emhttp process is out of sync and gets very confused very fast. I'm aware of this. We really need an API to emhttp, so it can be commanded either by the web-front end, or the API, or at a minimum, a signal we could send it ti get it to refresh its view of the system, regardless of what has happened. Perhaps one of the USR signals. (Pretty sure they still exist in Linux... they were there in System5.) I agree here, another port with a simple tellnet text interface would be helpful. USR signals do exist in linux (have for a long time) I actually did a quick hack/shell script at using localhost to send GET requests to the web front end to control it. That worked until Tom added security and the management web-page needed a password. There's a way to build the username/password pair and send that along with netcat. I forgot the basic mechanism. I actually wrote a network shell which did this, I'll have to locate it. In the meantime, you can use curl or wget which will support the http basic authentication.
May 8, 200818 yr Here's what t I have so far as a recommend setup. I'm sure the formatting can be better.. of all things, I'm not a media wilki expert. I just wanted to get my ideas down and visible. http://lime-technology.com/wiki/index.php?title=Third_Party_Boot_Flash_Plugin_Architecture I'm in the process of re-writing some of my scripts to the new layout. We have not gotten buy in on the additional run levels. My thought was for emhttp to just call one program at the run levels with a parameter. The parameter could be used to determine what directory to scan and what scripts to run. Once there is an agreement, I can write up a packaged install. Comments welcome.
May 8, 200818 yr Author Formatting looks excellent to me. Clear and easy to read. One thought I have is the naming of the root. If this is unofficial we should probably name the path to reflect that. It if is going to be part of future unRAIDs then vice versa. Apart from that everything your documenting still makes alot of sense to me.
May 8, 200818 yr One thought I have is the naming of the root. If this is unofficial we should probably name the path to reflect that. It if is going to be part of future unRAIDs then vice versa. Perhaps something similar to how Unix historically had /usr/bin and /usr/local/bin Joe L.
May 8, 200818 yr Agreed on root name, but it also has to exist on /boot.. So I named the root "unraid" for now, I'm open to a change. Someone else uses "custom" do we want to duplicate a /usr/local tree on the /boot drive? If I had my way that's what I would have done. However I wanted it to be simple and clear these directories reflect to the unraid environment. In effect it would be /boot/local and all needed directories under local would be duplicated. This is cleaner in the sense that it conforms to a normal directory standard and is local per the system. One of my biggest issues with learning unix after coming off the mainframe was.. I did not know or understand the normal directory structure. If we are in agreement, I'll alter it.
May 8, 200818 yr In effect it would be /boot/local and all needed directories under local would be duplicated. This is cleaner in the sense that it conforms to a normal directory standard and is local per the system. I like it... but I'm just one vote. One of my biggest issues with learning unix after coming off the mainframe was.. I did not know or understand the normal directory structure. My first unix version was direct out of Bell Labs... CB Unix version 1.0. I never trusted a computer I couldn't lift, but I won't hold your mainframe experience against you Joe L.
May 8, 200818 yr Just a thought from a Windows user, trying to speak on behalf of the many Windows-based users and new users, I would prefer you use standard Linux structures and naming. Using local or unraid or custom feels like a compromise between Linux and Windows/new-user terminology. A new or simple user won't be using any of this, so won't care. They will see /boot/config and perhaps things like /boot/logs, and that will feel comfortable to their Windows mindset. But any user who is going to use the advanced stuff discussed in this thread will have to adapt to Linux anyway, and it will be less confusing if it is standard Linux usage from the start. I would prefer you guys 'do it the Linux way'. I think that /boot/usr would be easy for all, but only if you guys can confirm that using /usr would correspond well with the Linux use of /usr. 'usr' seems so close to 'user' that it would be easy to understand, and it carries a user-driven connotation that should help make the official/unofficial proponents happy. I'm assuming that in Linux usr does traditionally mean user? I noticed /usr/local, and that seems a bit confusing or redundant, to a Linux neophyte. Is some stuff more local than other local user stuff?
May 8, 200818 yr I noticed /usr/local, and that seems a bit confusing or redundant, to a Linux neophyte. Is some stuff more local than other local user stuff? Generally, things in /usr/local are programs that you compiled from source. Say something wants to be installed in the bin directory. If it's precompiled, it'd go in /usr/bin. If you compile from source, it'll go in /usr/local/bin.
May 8, 200818 yr According to FHS - Filesystem Hierarchy Standard http://en.wikipedia.org/wiki/Filesystem_Hierarchy_Standard#Directory_structure /bin - Essential command binaries that need to be available in single user mode; for all users /usr/bin - Non-essential command binaries (not needed in single user mode); for all users, /etc - Host-specific system-wide configuration files (the name comes from et cetera). /lib - Libraries essential for the binaries in /bin/ and /sbin/. /opt - Optional application software packages. /usr/share - Architecture-independent (shared) data. /usr/local - Tertiary hierarchy for local data, specific to this host. Typically has further subdirectories, eg. bin/, lib/, share/.[2] So if we wanted to conform to FHS in some manner but also keep it specific to unraid I might be inclined to call it as /boot/local (meaning specific to this host/software application's environment). /boot/local/bin /boot/local/etc /boot/local/lib /boot/local/share Where we you propose packages go? /boot/local/share/packages /boot/opt/packages Or are we proposing building a whole tree off boot as in /boot/bin /boot/usr/bin /boot/etc /boot/share/packages /boot/opt /boot/var/log From what I remembered, I started storing logs in /boot/var/log and someone suggested putting them in /boot/log. or build a tree as in /boot/usr/local This is one of the reasons I thought of using. /boot/unraid as the local/root prefix for third party dropins. My thought was to keep it all under one neat structure which mimicked a local FHS.
May 8, 200818 yr Until Tom replies, we have to assume that he has no interest in supporting any of this. So instead of thinking of how he can tune unRAID for us, we need to think how we can modify things ourselves. If he allows his binary files to be redistributed, then you can make any changes to the main filesystem that you want. Then direct users who want to use the third-party add-ons to download a modified version of unRAID. If he doesn't allow this, then we have to assume that things are going to stay the way that they are. We just need to make the work-arounds as easy as possible. I.e. if we really needed different directories in the main filesystem, create a single script that will create them and execute whatever needs to be done. Then that one script is called in the go script. Anything that can stay on the flash drive probably needs to stay there.
May 8, 200818 yr This is one of the reasons I thought of using. /boot/unraid as the local/root prefix for third party dropins. My thought was to keep it all under one neat structure which mimicked a local FHS. Here's a simple idea. Have one directory that stores everything that is going to stay on the flash drive, and another directory that stores everything that must be copied over the current filesystem. For example: /boot/customcopy /boot/customperm Anything that needs to be in the main filesystem (perhaps it'd be too hard to recompile it to run off a flash drive) can be put in /boot/customcopy. Anything that can stay on the flash drive would be put in /boot/customperm. The names I chose may not be too intuitive, but you get the idea. If you do go so far as to have custom startup/shutdown commands for different programs, then that'd be great. For example, say that you have a program that has a config file that you may edit from time to time. It's stored in the main filesystem, so it'd be lost on reboot. Just have the shutdown command copy over any such files back to the flash drive, now everything stays in sync (unless the server crashes or what-not, but hopefully that won't happen to often). BTW, in case anyone didn't already know, you can choose where packages are installed when using installpkg. For the example above, say you want to install a regular program that needs to be in the main filesystem. To install you'd simply do something like this: installpkg -root /boot/customcopy/ perl-whatever Instead of installing it to the root of your filesystem, it'll install it to /boot/customcopy. Install all your packages that way, then you just need to copy over the entire directory over your / directory like so: cp -rfP /boot/customcopy/ / And there you go. The -r says to include directories, -f forces overwrite, and -P creates symlinks for symlinks.
May 8, 200818 yr > Then direct users who want to use the third-party add-ons to download a modified version of unRAID. I'm not too keen on this. Having an unRAID package that gets installed on slackware yes, having a whole third party modifed version of unRAID no. But this is getting off topic. I believe the purpose of this thread was to get a consensus on a mechanism to handle third party plugins and a naming convention. Albeit to make it easy for users to not have to modify the go script. Let us assume that no matter what, we're still going to need a mechanism to all easy drop in installation for the average joe. That being said, we have a number of suggestions, but not a consensus on naming convention. We can handle whatever is decided but we should vote on the convention. FHS directly off /boot /boot/bin /boot/share/packages /boot/etc/rc.d /boot/var/log FHS off boot in named directory local /boot/local/bin /boot/local/share/packages /boot/local/etc/rc.d I like the idea of a static FHS on boot and possibly a sync FHS on boot. Such as /boot/local/ (static) /boot/local.sync (synchronized to / on startup). I'm also considering what a person see's when they navigate to their flash drive. Do we want them to see bin etc usr var or do we want them to see one directory local (or unraid even); then have the appropriate tree under there. I'm a unix guy at heat. I've worked on dos, Windows, dec/vax, mainframes but once I learned unix I found my true love. So I would always consider a FHS that mimics that environment. I already understand it. It's comments from others that lead me into a level down to not overwhelm others.
May 8, 200818 yr If we can have a modified version of unRAID, then it could help out in several ways. I can't remember all the things I thought of before (son keeps waking me up way too early), but there were some good reasons. There'd be less initial hacking to get unRAID ready to accept third-party add-ons. Many essential packages could already be installed (such as httpd, I've been thinking that we could access unRAID via apache and have it proxy to emhttp, giving us some more control and the ability to add extra stuff as needed). Etc. etc. Anyways I'll argue that point later. As far as the directory structure, I think it'd be less confusing to have it named like I did. If you use /boot/local, and assuming you want to install entire packages there, then you could end up with something installed in /boot/local/usr/local/programname. I'm guessing you want to actually store the program directly in /boot/local so it'd be /boot/local/programname, but that won't work. You need a lib directory, var, etc, and who knows what else. Just use a simple directory that stores exactly what would be in /. So a program that would be stored in /usr/local/bin/programname would be put in /boot/customcopy/usr/local/bin/programname. Makes installpkg much easier to use, because of the -root switch. We can then compile packages that can be installed on standard unRAID, or on a regular Slackware install (whether you're using unRAID or not). The only thing that would hurt cross-compatibility in this case would be how to sync. In a regular Slackware install there's no need to sync, so no need to make that a part of the shutdown script. In a standard unRAID install there'd be a need. The only solution my tired brain can think of, is you can touch a specific file in /boot, let's say make a file named /boot/unRAIDflash. I can make the install script for a program look and see if the file exists, if so then do a, if not then do b (i.e., if the file exists then copy over a shutdown script that will do the sync, if the file does not exist then copy over a file that just shuts the program down).
May 8, 200818 yr FHS off boot in named directory local /boot/local/bin /boot/local/share/packages /boot/local/etc/rc.d I like this... I like the idea of a static FHS on boot and possibly a sync FHS on boot. Such as /boot/local/ (static) /boot/local.sync (synchronized to / on startup). Probably Overkill. I'm also considering what a person see's when they navigate to their flash drive. Do we want them to see bin etc usr var or do we want them to see one directory local (or unraid even); then have the appropriate tree under there. They already are guaranteed to see /config, bzroot and bzimage, the few files needed by the boot menu. Other than the additional "/local" folder, to keep confusion to a minimum, it is probably best not to clutter up the top level folder at /boot. Remember, we who use these extensions can add anything we desire as long as it does not break other functionality. It you really need a "specific" folder off of boot, your ad-on script can create it and then use it. One thing to keep in mind... the run-level scripts tied into the "go" script are the main focus here. The additional packages we install and run are as likely to be installed on a hard disk (cache drive, or on an unRAID data disk) as anywhere else. We need to remember that all packages, src, etc not will be on the flash drive... some will be elsewhere, some will install from elsewhere. We need to make it easy for non-programmers to install packages. So /boot/local/packages might be where a package you be dropped and then installed from. I'm a unix guy at heat. I've worked on dos, Windows, dec/vax, mainframes but once I learned unix I found my true love. So I would always consider a FHS that mimics that environment. I already understand it. It's comments from others that lead me into a level down to not overwhelm others. My first UNIX Login ID was in 1980... I've had many since then. The shell pre-dated the Steve Borne shell. It was the Mashey shell, written by John Mashey at Bell Labs, it had labels and goto... this was way prior to anything close to today's shell syntax. Editing was using "ed" the command line editor, "vi" would not exist for several more years on the systems I worked on. Yes, I'm a Unix geek at heart. A not-so-average Joe L.
Archived
This topic is now archived and is closed to further replies.