July 24, 200817 yr Hey guys, just getting back to this subject matter. I've been on vacation and in sort of hermit mode with my meditations. With all of the positive assistance of the community, Tom has given a positive nod to the concept. In addition, Tom has some additional ideas which will make third party additions easier to maintain. I'll leave the details for Tom to post about. For the record, it's the confidence in the community to assist in supporting the environment, that's making this possible. Every PM I have with Tom always has praise for the community. You Guys Rock! Keep up the ideas, and discussions, No matter how we bounce things around, brainstorming makes the environment better. What we need here is patience. It will take some time to review, engineer and implement.
August 18, 200817 yr Author In an attempt to get the ball rolling again here is the custom directory's tarred up. I have also added the hddtemp binary and dbase along with the original wget.tgz which can be installed. There is no go script and no hooks into the go script. In essence this tar wont do al thing unless you call things from it. The idea being that we can somehow all work on the custom addons and release one tarball every now and again. Edit: forum wont let me upload a 600kB file (128KB max), wiki will only allow images, so much for that idea. For this to work we need some sort of development area. Tom can you supply one or does anyone else have any ideas? Can see us needing more than 50MB of webspace and some means to edit... perhaps an ftp would be best.
August 18, 200817 yr Is the .tar file an actual slackware package? I think this would be the best approach. Contact me via email if you want some example slackbuild scripts.
August 18, 200817 yr Author no its a simple tarball of a directory. an installable package sounds better but we really need some development resources to do this right or were going to be all over the place real soon. Im hoping Tom comes back in the not to distant future as a subdomain and a few ftp accounts would sort this all out. Until then im not too excited about spending extra time trying to find fudges to bypass the lack of resources but Id love to see some build scripts. In the interim if we just made one that only had the folder structure and a few sh scripts that is under 128kB that would be a start to prove the concept. If this was a pure GPL project i would have purchased a domain name and had a friend host it long ago but since its not im loathed to spend actual cash/money. I will ponder that some more.
August 18, 200817 yr If this was a pure GPL project i would have purchased a domain name and had a friend host it long ago but since its not im loathed to spend actual cash/money. I will ponder that some more. What are the costs involved. I suppose Tom could give us a subdomain. Even an ip address to ftp to/from would suffice.
August 18, 200817 yr Author costs would be low a few tens of bucks a year. its not the money its the principle at this point.
August 18, 200817 yr Well we've gotten a response from Tom that an initialization directory structure is a possibility. This is purely because he feels that there is enough technical support to handle it via the forums. In addition, Tom has other ideas which add on to this in a significant way allowing us to implement addons easily. I'm going to beta test his idea and see how it works out (tease, tease, tease)... I always say you get more bees with honey then vinegar. Make a sound proposal to Tom directly and see where it goes. There may be a few who could also contribute a few dollars via paypal towards a development repository. Perhaps Tom could make you ftpmaster for a local limetech ftp site.
October 12, 200817 yr Was there ever any consensus on the name of the directory off of "/boot" in which to put local customized additions? I have the package-manager plug-in for unmenu.awk just about ready for publication. It can download packages, allow you to manually install them, and then, once installed it presents to the user a button to allow configuration of an auto-re-install of that package upon reboot. It uses /boot/packages (and creates it if it does not exist) It allows re-definition of the location in its config file for more advanced users. Packages are downloaded to /boot/packages. If you then install using the supplied button, it will do it once. It will create a "packagename.manual_install" shell script to allow you to re-install without the web-interface. Most of the time, there is only one command in the script, the "installpkg ...." command, but it can be multiple lines to locally configure a default slackware package to suit the unRAID server. Once you manually install (or install by using the button on the unmenu.awk plug-in) an "Enable Re-Install on Re-Boot" button is presented. Pressing this button will create a "packagename.auto_install" script in the packages directory. It has the same contents as the ".manual_install" file. A series of shell commands needed for the installation and configuration of a given package. What I need to do now is add a utility "re-install-packages-on-reboot.sh" run by adding a line to the "go" script. That utility will simply do something like: ls /boot/packages/*.auto_install | while read a do sh -c "$a" >>/var/log/syslog 2>&1 done The question is where should I install the shell script that in turn invokes the package auto_install files? I'm thinking /boot/custom/bin and again allow an override in the unmenu.awk config file for some alternate location. I know this is very different that the current /boot/custom/etc/rc.d scheme some of us have in place. I'm thinking keep it simple, but my script can create the /boot/custom/etc/rc.d folder if it does not exist regardless if it is one level below /boot, or several. I just fear it is more complicated than needed for Linux novices. Regardless, I need to append a line to the "go" script... (or, optionally NOT append a line, depending again on the local unmenu.awk config file.) I'll probably put the actual command line I'll append in the unmenu.conf file. that will make it pretty flexible. If it does not exist, I'll not do anything to add a line to the "go" script figuring you are dealing with the auto_install scripts yourself in some way upon reboot. (and once you enable auto-re-install-on-reboot on a package, a button is present to disable auto-re-install-on-reboot ... and if pressed, it deletes the auto_install file.) This is the last of the pieces to put into place the initial version of a unmenu package management plug-in. I need to put "re-install-on-reboot.sh" somewhere... the question is where? Comments? Ideas? Joe L.
October 12, 200817 yr /boot/custom/bin should be the location of major scripts and binaries. /boot/custom/etc/rc.d should be the location of startup scripts which install, initialize and startup programs & daemons. You can install the autoinstall scripts as packagename.sh in /boot/custom/etc/rc.d and use the +x bit to control (install on boot/ignore on boot) Here are some scripts... Food for thought... (Pass the ketchup please) I have a script in /boot/custom/etc/rc.d/init.d as the following for running the init scripts We would just need to add a test for +x , or not, to control execution or ignore. #!/bin/bash logger -trc.local_startup -plocal7.info -is "Initiating Local Custom Startup." for script in /boot/custom/etc/rc.d/* do scriptbase=${script##*/} # Strip pathname if [ $scriptbase = "rc.local_startup" ] ;then continue; fi if [ $scriptbase = "rc.local_shutdown" ] ;then continue; fi if [ -d $script ] ;then continue; fi ( echo "Processing $script" fromdos < $script | sh ) 2>&1 | logger -t$scriptbase -plocal7.info -is # Old way not used. # fromdos < $script > /tmp/$scriptbase # chmod u+x /tmp/$scriptbase # /tmp/$scriptbase # rm /tmp/$scriptbase done # Setup shutdown script RCFILE=rc.local_shutdown if ! grep /boot/custom/rc.d/init.d/$RCFILE /etc/rc.d/$RCFILE > /dev/null 2>&1 then echo -e "\n/boot/custom/etc/rc.d/init.d/$RCFILE\n" >> /etc/rc.d/$RCFILE fi if [ ! -x /etc/rc.d/$RCFILE ] then chmod u+x /etc/rc.d/$RCFILE fi logger -trc.local_startup -plocal7.info -is "Local Custom Startup Complete." Here is my generic package installer it installs any package in the /boot/custom/usr/share/packages folder #!/bin/bash if [ "${DEBUG:=0}" -gt 0 ] then set -x -v fi P=${0##*/} # basename of program R=${0%%$P} # dirname of program P=${P%.*} # strip off after last . character O=${P%_*} # Operand D=${P#${O}_} # Data (last param before _ character) PKGDIR=${PKGDIR:=/boot/custom/usr/share/packages} # Cleans ram drive root directory of unneeded files to save ram clean_ram_root() { [ -d /usr/man ] && find /usr/man -type f -exec rm {} \; [ -d /usr/man ] && find /usr/man -type l -exec rm {} \; [ -d /usr/man ] && find /usr/man -type d -empty -depth -exec rmdir {} \; [ -d /usr/lib ] && find /usr/lib -type f -name '*.a' -exec rm {} \; [ -d /usr/lib ] && find /usr/lib -type d -empty -depth -exec rmdir {} \; [ -d /usr/doc ] && find /usr/doc -type f -exec rm {} \; [ -d /usr/doc ] && find /usr/doc -type d -empty -depth -exec rmdir {} \; [ -d /usr/src ] && find /usr/src -type f -exec rm {} \; [ -d /usr/src ] && find /usr/src -type d -empty -depth -exec rmdir {} \; [ -d /usr/include ] && find /usr/include -type f -exec rm {} \; [ -d /usr/include ] && find /usr/include -type d -empty -depth -exec rmdir {} \; } for package in ${PKGDIR}/*.tgz $* do PKGB=${package##*/} PKGB=${PKGB%.tgz} # echo $PKGB if [ ! -f /var/log/packages/${PKGB} -o ${FORCE:=0} -gt 0 ] then echo "Installing $package" installpkg $package 2>&1 | logger -tinstallpkg -plocal7.info -i clean_ram_root else echo "$package already installed." fi done clean_ram_root I mention this for ideas and also consider the cleaning of root_ramdisk for files not really need. In my /boot/ciustom/etc/rc.d I have specialized scripts for packages requiring special cases bash-3.1# pwd /boot/custom/etc/rc.d bash-3.1# ls -C S00-syslog-update S10-installpkg S20-install-nagios-plugins S30-inittab-powerdown S99-clean-ram-root S01-hdparm S10-installpkg-rtorrent S20-install-nrpe S75-useradds S99-update-profile S02-blockdev S10-link-homes S20-install-nsca S80-update_hosts S99-z-system-up S02-sync-etc S20-init.identd S20-install-ntp S81-cron-rsync_boot bak S02-waitmnt_disk8 S20-init.rsyncd S20-install-proftpd S90-init.nfs init.d S03-mount-ramdisk S20-install-bind S20-install-smartmontools S90-smb-shares monthly_parity_check.sh S04-mount-swap.sh S20-install-exim S20-install-sshd S91-init.vfs_cache more S08-cpufreq S20-install-mt-daapd S30-inittab-additions S92-install-slocate rc.unRAID For example bash-3.1# more S10-installpkg-rtorrent PKGDIR=/boot/custom/usr/share/packages/rtorrent /boot/custom/etc/rc.d/S10-installpkg
October 12, 200817 yr Your realize of course, that if someone saves a file on the flash via a share, that it will set the +x bit? Plus, there is no easy way to toggle the +x bit from windows except telnet, or a back-end management application. I realize that using +x is the conventional way to toggle startup scripts, but you may want to consider something else for unRAID. In discussing this with some others, we have though about using an extension, .autostart to do it.
October 12, 200817 yr /boot/custom/bin should be the location of major scripts and binaries. Sounds good to me. That is where I'll probably put the re-install-packages-on-reboot command. /boot/custom/etc/rc.d should be the location of startup scripts which install, initialize and startup programs & daemons. That is where I have a script somewhat similar to yours. Your script would not work for me... My version is: #!/bin/bash logger -trc.local_startup -plocal7.info -is "Initiating Local Custom Startup." scripts=`ls /boot/custom/etc/rc.d/ | egrep -v "rc.local_startup|rc.local_shutdown"` cd /boot/custom/etc/rc.d for script in $scripts do scriptbase=${script##*/} # Strip pathname ( echo "Processing $script" #fromdos < $script | sh -xv sh -xv $script ) 2>&1 | logger -t$scriptbase -plocal7.info -is done You can install the autoinstall scripts as packagename.sh in /boot/custom/etc/rc.d and use the +x bit to control (install on boot/ignore on boot) I do not like that idea at all. Many of the users will be looking through windows file-explorer, and the whole concept of permission bits on Linux is foreign to them. As I said, experienced users can use that feature, and you can add the logic to your script, but I still think the files with the suffix .auto_install in the /boot/packages is far easier for a newbie to manage. at a glance you can see which packages you have downloaded, and which will be auto-installed just by a file-listing. Here are some scripts... Food for thought... (Pass the ketchup please) # Setup shutdown script The script completely broke when I attempted to start a process in the background. I ended up invoking my one "daemon task as follows to get it to detach from the terminal and allow the main loop to proceed.: cat S20-spin-drives echo "/boot/custom/bin/spin_drives.sh" | at now +1 minute I like your idea to clean up the "ram_root" of man pages. etc a lot..... Perhaps it should be a separate script? (So we can both invoke it) In my /boot/custom/etc/rc.d I have specialized scripts for packages requiring special cases I do all my installation there now, same as you... but will change to the auto-install-on-reboot for some more easily managed tasks. It too can be a script invoked by an S99-auto-re-install-packages script in /boot/custom/etc/rc.d/ Currently, in my "go" script I have just the one line like this: fromdos < /boot/custom/etc/rc.d/rc.local_startup | sh To sum up... I'll put my re-install-on-reboot script in /boot/custom/bin That is the line I'll be appending to the "go" script if I don't find one already there. I'll invoke my re-install-on-reboot script from an S99-auto-re-install-packages script. It will only auto-re-install those packages that have an affiliated .auto_install file. I cannot assume everything downloaded will be .tgz installpackage format and not require local configuration for unRAId use, so cannot just loop through all the files in the packages folder. Some will use an installpkg command followed by other local shell commands to configure the generic package for unRAID use. Others might even use "unzip" followed by other commands, and not be in installpkg format at all. I'm trying to make it flexible enough for those like yourself who are experienced and will configure as needed, and those who are still looking for the "any" key when asked to "press any key to continue" I know we have users with WIDE differences in their level of experience. Joe L.
October 12, 200817 yr Your realize of course, that if someone saves a file on the flash via a share, that it will set the +x bit? Plus, there is no easy way to toggle the +x bit from windows except telnet, or a back-end management application. I realize that using +x is the conventional way to toggle startup scripts, but you may want to consider something else for unRAID. In discussing this with some others, we have though about using an extension, .autostart to do it. I am using suffixes ".auto_install" and ".manual_install" for exactly the reasons you mentioned. Ease of use for the inexperienced and experienced alike. root@Tower:/boot/packages# ls -1 10-unmenu-package.conf* 20-unmenu-package.conf* aaa_elflibs-12.1.1-i486-1.tgz* apcupsd-3.14.3-i486-1kjz.tgz* bubbaraid.zip* cups-1.3.7-i486-2.tgz* cxxlibs-6.0.9-i486-1.tgz* dbus-1.1.20-i486-1.tgz* file-4.21-i486-1.tgz* file-4.21-i486-1.tgz.auto_install* hal-0.5.11-i486-1.tgz* infozip-5.52-i486-2.tgz* infozip-5.52-i486-2.tgz.auto_install* kernel-modules-2.6.24.5-i486-2.tgz* ksh93-2007_03_28-i486-2.tgz* lsof-4.78-i486-1.tgz* lsof-4.78-i486-1.tgz.auto_install* ntfs-3g-1.2216-i486-1McD.tgz* ntfs-3g-1.2216-i486-1McD.tgz.auto_install* pciutils-2.2.10-i486-1.tgz* screen-4.0.3-i486-1.tgz* smartmontools-5.38-i486-1.tgz* sysstat-7.0.0-i486-1.tgz* tcpdump-3.9.8-i486-1.tgz* vlc-0.9.2-i486-1alien.tgz* wget-1.11.4-i486-1.tgz* wget-1.11.4-i486-1.tgz.auto_install* wget-1.11.4-i486-1.tgz.manual_install* Joe L.
October 12, 200817 yr Plus, there is no easy way to toggle the +x bit from windows except telnet, or a back-end management application. I thought this was what was being developed. A backend application to manage download and install. That is why I mentioned it. Your script would not work for me... I'm baffled, I have it on 3 machines and it works fine *shrug* Oh well. The script completely broke when I attempted to start a process in the background. I ended up invoking my one "daemon task as follows to get it to detach from the terminal and allow the main loop to proceed.: Running it via batch or at is another way I run things in the background detached like that. My system isn't perfect. I was presenting what I have working so far, use, adjust or toss as desired.
October 13, 200817 yr Plus, there is no easy way to toggle the +x bit from windows except telnet, or a back-end management application. I thought this was what was being developed. A backend application to manage download and install. That is why I mentioned it. It is an application to do as you describe, but it has to be able to accommodate novice users. I think the ability to override the default behaviour in the unmenu local config file will allow you the flexibility to customize too. Your script would not work for me... I'm baffled, I have it on 3 machines and it works fine *shrug* Oh well. I was trying to start a task using "nohup scriptname >/dev/null 2>&1 &" and by the time it got through a couple of pipes to shell it just did not run some scripts. I just modified slightly the version you wrote to accomodate everything I needed it to do as well. It is basically the same script. The script completely broke when I attempted to start a process in the background. I ended up invoking my one "daemon task as follows to get it to detach from the terminal and allow the main loop to proceed.: Running it via batch or at is another way I run things in the background detached like that. "batch" is just another link to "cron" and "at" on many machines... same scheduler at work. The "at" command was very easy, and easy for beginners to understand too. My system isn't perfect. I was presenting what I have working so far, use, adjust or toss as desired. I plan on using the best of what I encounter... and then make it easy to accommodate sophisticated users like yourself and beginners who are trying to figure out the basics. Every time I write a script that works for me, somebody comes up with something I never considered, or uses it in a configuration I can't duplicate. I'm far from perfect, even if my code could be.
October 13, 200817 yr I was trying to start a task using "nohup scriptname >/dev/null 2>&1 &" and by the time it got through a couple of pipes to shell it just did not run some scripts. This baffles me, I have to play with it s'more to see what's going on. Thanks for the update. There are sometimes quirks with how bash handles things vs ksh. especially in loops and subshells. A point to consider about the current method is the fromdos conversion.
October 13, 200817 yr I was trying to start a task using "nohup scriptname >/dev/null 2>&1 &" and by the time it got through a couple of pipes to shell it just did not run some scripts. This baffles me, I have to play with it s'more to see what's going on. Thanks for the update. There are sometimes quirks with how bash handles things vs ksh. especially in loops and subshells. A point to consider about the current method is the fromdos conversion. I am thinking about the fromdos issue. It is one we constantly fight. It is prudent to leave it in place, but you can see I took it out of my version as I don't edit my files in windows editors... (or rather, my windows editor is "vim") Joe L.
October 13, 200817 yr Try notepad2 http://www.flos-freeware.ch/notepad2.html The syntax highlighting, and auto-detect for line endings is great.
October 13, 200817 yr Author Someone follow my rather obvious but relevant logic path here: unRAIDs' #1 requirement is that its stable As soon as you addon components you need a swapfile to gurantee a stable system on the diverse systems people use A swap file will be on a HDD for virtually all users. If you need a HDD for swap then it makes sense to use the unRAID official cache drive feature. If you are using a cache drive you dont need a second flash since the HDD will outperform it on speed and capacity You still need a flash drive to license unRAID. The unRAID boot flash fails quicker the more you use it so using the cache drive rather than the flash drive wherever possible increases reliabilty. Following this logic a couple of points present themselves: 1. A requirement of all the addons and hacks should be an unRAID cache drive 2. If a cache drive is available unRAID should use it wherever possible aka perhaps a boot from HDD option version should exist. I know this may on the face of it seems trivial but its fundamental to all the hacks going on if we can all assume a a cache drive exists many options become simpler/possible.
October 13, 200817 yr - Swap on cache drive makes ultimate sense.... it is a no-brainer. - Swap file on array is not so bad... you could thrash a bit if you get tight on space however.... very unlikely - Swap on flash should be a last resort. Now: - All apps are not ram hungry and need swap - Some users have more than 512M RAM -- No need to REQUIRE swap in order to use apps
October 13, 200817 yr Author Base requirements ease support and development. If you KNOW a user has a swap then you don't have to worry. Failing that we need to do things like "Dont run this if you dont have XXMB of memory, Dont intsall unless you have XXMB free on your flash and if there is a memory leak expect it to crash quickly". Its a better idea to make one small requirement than risk unRAID crashing on some unwary unskilled user. No one should have to pay for adding an IDE cache drive... a trip to the dump would get you the drive and cables. Less variables = better for all
October 13, 200817 yr - Swap on cache drive makes ultimate sense.... it is a no-brainer. - Swap file on array is not so bad... you could thrash a bit if you get tight on space however.... very unlikely - Swap on flash should be a last resort. Now: - All apps are not ram hungry and need swap - Some users have more than 512M RAM -- No need to REQUIRE swap in order to use apps I agree with all of the above. In fact, most apps are not RAM hungry and do not have memory leaks. I think in the package descriptions, as we define packages that can be automatically downloaded, we will need to warn users of the need for a cache drive if and where appropriate. For example, a swap partition/file is NOT needed to install or use the "lsof" command, the "ntfs-3g" driver, or the "unzip" command, or "wget" Joe L.
October 13, 200817 yr No one should have to pay for adding an IDE cache drive... A cache drive depends on the unRAID license purchased. The PRO version has cache drive capability, the PLUS and BASIC version do not. A user could still add a drive to the system, partition it adding swap and use that without bringing it into the array. All this talk of swap. A point needs to be verified. I'm not sure initramfs blocks will utilize swap. I remember reading somewhere that unused initramfs blocks will not be put on swap yet in contrast, unused (lightly used) tmpfs blocks are subject to being put on swap. This needs to be tested/verified. I know I had enabled a swap partition once and only 256K of it was ever used. On that system I have 8G, so It's not a valid test. Without know the behavior firsthand, I'm not sure of the value of swap. Also, swap on flash is possible at the expense of a flash drive's life period. Years back we used to use a Compact Flash drive for swap with the Roku HD1000. It had only 64MB. The flash drive would last from 6 months to 2 or so years. There were also many reports of it never failing.
October 13, 200817 yr SWAP with initramfs. On my smaller 1GB memory system, I created a 2GB swap partition. I wrote a 1GB file to /tmp (which lives in inittamfs). The system locked up. So, I guess the swap will work for application data, but not for files based on initramfs.
Archived
This topic is now archived and is closed to further replies.