I've been thinking about an idea of creating an unRAID "appliance" sw platform in the sense of providing unRAID with a certain set of addons built in to a stable sw platform. Seeing how people are struggling getting their addons working and solving different kind of conflicts, I think there could be demand for a complete sw platform with fixed content. But all the content would be compatible and fully functional out-of-the-box. The platform would be updated as a whole, updating individual addons would not be supported to ensure 100% compatibility. I fully understand this wouldn't be for everyone but even enthusiasts might consider something like this valuable.

 

In a perfect world where the unRAID plugin system is complete and all the plugins adhere to the architecture this wouldn't be necessary. Eventually we could have multiple appliance sw platforms for different use cases like: basic fileserver and media server.

 

Now for the ideas (and comments) request part:

1. What should be the basic content of an unRAID appliance sw platform? Which addons should be there?

2. How would you implement the update mechanism for the appliance sw platform? It should have the possibility to update unRAID and all the addons with some kind of fail safe (rollback) mechanism.

3. What problems do you foresee in this kind of concept?

 

Thanks in advance for your ideas and comments!

That idea was tried in the past with unRAID-web.  It worked for a time.

However it also means maintaining it going forward which could be allot of work.

 

 

I would think the best investment in time is to work alongside those who make plugins and design a plugin system and repository that makes it easier to add/maintain them.

 

 

Just my 2c.

That idea was tried in the past with unRAID-web.  It worked for a time.

However it also means maintaining it going forward which could be allot of work.

Thank you for the comments.

 

I took a brief look on unRAID-web but to my eyes it seemed like web-server addon for unRAID not a complete platform. unMenu is a bit closer to what I had in mind since it's packages have been tested for compatibility and been updated as time has passed. I've been personally using unMenu from the start and it provides great basics like screen, smtp etc. But even installing those through the package manager can be at least intimidating if not overwhelming for an average user (not average unRAID user). I was going for a totally unified platform without a need to install any additional addons. It should lower the bar for newbies.

 

I would think the best investment in time is to work alongside those who make plugins and design a plugin system and repository that makes it easier to add/maintain them.

Yep this is the way to support the common development. But I would still see benefit in creating one or few default appliance sw platforms utilising a set of compatible plugins. To keep the work somehow minimised I would release new versions of the platform quarterly at most. An average user shouldn't want/need to update his/hers system more frequently.

 

I too have high hopes for the plugin framework but learning from the past and from other similar projects a unified plugin repository in an unlikely outcome.

I think what you are trying to do is make unraid more that what it was initially designed for.

It does what it was ment to do and does it well, once you start messing with it it will become jack of all treids but master of none.  I can understand use of some plug ins/ features that are relates to the unraid itself.

Plugin like snap, which extends the function of unraid to provide acces to more drives that though unprotrcted still usefull for some .

Maybe plugin to provide gui for preclear function that have been discussed in one of the threads here,  plugin for power management ups control

 

Sent from my SGH-T889 using Tapatalk 4

 

 

I think what you are trying to do is make unraid more that what it was initially designed for.

It does what it was ment to do and does it well, once you start messing with it it will become jack of all treids but master of none.  I can understand use of some plug ins/ features that are relates to the unraid itself.

Plugin like snap, which extends the function of unraid to provide acces to more drives that though unprotrcted still usefull for some .

Maybe plugin to provide gui for preclear function that have been discussed in one of the threads here,  plugin for power management ups control

Good points there. But in my mind unRAID does not do well out-of-box even as a very basic file server. I've built several unRAID servers and I've started to see a list of essential/mandatory addons to appear. This has been discussed elsewhere but I personally consider the following addons/packages absolutely mandatory for the very basic unRAID usage with the current unRAID 5.0 version:

- pre_clear

- status alerts

- lm_sensors (you need a way to monitor your temps and fan speeds at least in real time, don't you?)

- apc-ups daemon (no one should run their nas with out on UPS and APC seems to be only widely supported brand)

- cache_dirs

 

And to enable the above functionality you need:

- unMenu (for basic usage just the package manager for easy installation, this could be stripped from the platform)

- screen (for the pre_clear, no other need for basic usage)

- smtp/mail (required for mail notifications)

- gcc (required by smtp/mail)

- perl required by sensors package)

 

One could say that most of the above should be part of unRAID but they aren't and each user has to go through the hassle of installing them. There are quite a few other addons which others might consider mandatory (eg. snap). As long as the addon does not interfere with other addons and it's even close to core functionality, it could be part of the file server platform.

 

Did I miss something in the above list or is there something which shouldn't be there? Pre_clear is perhaps the only one which causes problems for new users since it does not have web ui (yet) and requires cli usage. All the others mostly work under the hood.

 

I also have a proposal for the Media Server but let's tackle the basic platform first.

I doubt your get a pre clear add-on as SF released one then it was withdrawn as author of per clear script had issues with them using his work.

 

I would love to see

 

email plugin that emails you the details you want to see,  if you want to see a smart report of all or a specific drive evey 1hr 20min you can.

 

UPS support - absolute must.

 

Sleep - A low power server without a sleep plug in is nuts, I want schedule the server to sleep when I want. SF sleep plunging was great at this.

 

 

This is a great idea. We have seen numerous requests for missing core functionality to be added and this would be one solution. Your list of basic features is spot on in my opinion (UPS support, notifications, and monitoring) which would make unRAID more like commercial NAS products and competing NAS software (e.g. FreeNAS). I would consider adding the latest SNAP plugin - being able to plug in a USB drive and have it be accessible also seems like core functionality to me.

 

With regard to pre-clear, the SF add-on was designed without the original author's permission or knowledge. If I remember correctly, his concern was that he might need to support an add-on that he did not write. I think that Joe would probably be willing to work with someone on this feature if contacted ahead of time and given the opportunity to have input on it.

Sleep --- not only no but hell no.  Too HW dependent.  Too fraught with peril

UPS support --- less HW dependent, but still HW dependent.  I vote no.

LM Sensors --- less HW dependent, but still HW dependent.  I vote no.

 

With ANYthing that has a HW dependency, you will have is some people it works for, but also a group of whiners that can't get it to work who will drag you into a soul-sucking darkness and your protestations that "it doesn't work and isn't intended to work on your HW" will fall on deaf ears and claims that you HAVE to fix it because there is "nothing wrong" with my hardware.

 

Stick (at least start with) to things that have high usefulness and low compatibility issues.

 

I vote for just one thing --- e-mail.  I don't know anything that has higher usefulness.  While it can be PROVIDER dependent, it is not HW dependent so it is much easier to troubleshoot.  Get it working.  Get a config wizard for common mail systems (gmail, hotmail, etc).  Get a generic POP3/IMAP config wizard.  A blending of iPhone and Thunderbird's set up wizards.  Leave out SSL... just get it working.

 

E-mail gets my vote because so many OTHER useful things will be much MORE useful with e-mail capability, and that will be enabled if a common e-mail API is present.

 

preclear doesn't need screen... it needs to be rewritten with a GUI and use some custom C code rather than dd.  Filling a disk with zeros, while reporting progress via a log file ain't hard to code.  (or use a dd alternative that has better logging capabilities like dc3dd)

 

Another vote... BubbaTools adds a widget to the main unRAID UI that lists the open files that are likely to prevent a shutdown.  I find that particular thing IMMENSELY useful.  I can't tell you how many times I have gone to shutdown the array, and did a "rut-ro.. let me close the other app first or it will hang."  An open-files indicator needs to be right there next to the array stop button.... heck it needs to popup a warning if you try to stop the array with open files.  Here's the php code... it could stand some improvement.

<?php
// Option:  html  - 
//          block - returns 0 or 1
//          array - returns comma-delimited list
//          table - echos html table
//          en    - echos english sentence stating number of files (if any) preventing shutdown.
//

function ofiles($type, $button=0) {
//* cd to /tmp or else lsof itself will shoud up as working dir on websserver home
$res = shell_exec("cd /tmp;lsof -F facn /mnt/disk* /mnt/user* /dev/loop* /dev/md* /mnt/cache 2>/dev/null");
$res1 = split("\n", $res);
$old_c = "~";
$blocked = false;
$bcount = 0;
$process=0;

$return = "";

foreach ($res1 as $stg) {
  $c = substr($stg,0,1);
  $var = substr($stg,1);
  switch ($c) {
    case "c" :
      $name = $var;
      $prog[$process] = $name;
      $count[$process] = 0;
      $blocking[$process] = 0;
      $pnum[$process] = $process;
      break;
    case "n" :
      $count[$process] ++;
      $flist[$process][$i++] = $var;
      if ($cwd) {
        $flist[$process][$i-1] .= "  (working directory)";
      }
      break;
    case "a" :
      if ($var == "u" || $var == "w") {
        $blocking[$process] ++;
        $blocked = true;
        $bcount ++;
      }
      break;
    case "f" :
      if ($var == "cwd" && $prog[$process] !='smbd') {
        $blocking[$process] ++;
        $blocked = true;
        $bcount ++;
        $cwd = true;
      } else {
        $cwd = false;
      }
      break;
    case "p" :
      $process=$var;
      $i=0;
    default :
      break;
    $lastvar = $var;
  }
}

if ($prog) {
foreach ($prog as $pp) {
  //$temp[$pp] = join(",", array($pp, $count[$pp], $blocking[$pp]));
} 
//$return = join("\n",$temp);
}

switch ($type) {
  case 'en' :
    if ($blocked) {
      if ($bcount == 1) {
        $return .= "There is 1 <a href='/BubbaTools/OpenFiles' target='_top'>open file</a> that may block array shutdown.";
      } else {
        $return .= "There are $bcount <a href='/BubbaTools/OpenFiles' target='_top'>open files</a> that may block array shutdown.";
      }
    } else {
      $return .= "No <a href='/BubbaTools/OpenFiles' target='_top'>open files</a> on the array blocking array shutdown.";
    }
    break;
  case 'table' :
    $return .= "<table border=1 cellpadding=3><tr style='background:rgb(200,200,200);' valign=bottom><td>Program*</td><td align=center>Process<br>ID</td><td>Open Files</td><td align=center>Open Files that<br>may prevent<br>unRAID shutdown</td><td>File name and path</td></tr>";
    $bb="";
	if ($pnum) {
		foreach ($pnum as $pp) {
			$ss = $flist[$pnum[$pp]][0];
			if ($button != 0) 		$bb = " <button onclick='killproc($pnum[$pp], \"$prog[$pp]\", \"$ss\");'>Kill</button>";
			$return .= "<tr valign=top><td>$prog[$pp]</td><td align=center>$pnum[$pp]$bb</td><td align=center>$count[$pp]</td><td align=center>$blocking[$pp]</td><td>";
			foreach($flist[$pnum[$pp]] as $pname) {
				$return .= "$pname<br>";
			}
			$return .= "</td></tr>";
		}
    }
    $return .= "</table>";
    $return .= "* An entry for 'bash' means you have a terminal session open with the working directory on the array<br>* If 'smbd' is listed, it means that another computer has a samba share mounted. Files open by Samba usually won't prevent array shutdown.";
    break;
  case 'html' :
    break;
  case 'block' :
    if ($blocked) {
      $return .= "1";
    } else {
      $return .= "0";
    }
    break;
  case 'array' :
    break;
}
return $return;
}

Bubbaq, while I understand your passionate response :-)

I am not entirely agree with some of it.

 

Sleep- not sure, it might be usefull for some but I don't see the point.

 

Ups support-is must. Even a limited one besides unraid already have very nice apc plugin. What it needs is a nice gui config screen so users would not need to drop to cli.

 

Sensors is a nice to have but not a must. The main screen already give you enough info.

 

Preclear is something that a must to have and also some things that needs a gui very much. It would be nice to simply add a drive to the system, go to the main screeb, see the new drive and see options like pre clear, format, moint  etc. In addition to add to array and such.

And you dont really need to change the pre clear script much, just add option to log to file instead of sceen. And just use gui to execute the script using screen with all the param that user select.

 

Sent from my SGH-T889 using Tapatalk 4

 

 

If there is a concise compatibility (and incompatibility) list with anything hardware dependent, that will help clarify who can and cannot use it successfully. UPS support I think is a good example and something that exists with other NASes. If the list is in a wiki, it will make it easier for end users to update the list and help the community. Some of these hardware compatibility lists may exist from the underlying utility.

And you dont really need to change the pre clear script much, just add option to log to file instead of sceen.

 

Not if you want progress from dd you can't.  There are some ways to do it with a cron job to send a sigint but it is a kludge and fraught with peril IMHO.

 

Should be a GUI that says "Hey... I see a new hard drive.  You want to add it to the array?"  And if Yes, just "do it" and if I come back, give me a status report and progress indicator."  And also throw a badge on the main UI that "Disk X is currently being prepared for addition to the array.  Estimated XX hours remaining."

Should be a GUI that says "Hey... I see a new hard drive.  You want to add it to the array?"  And if Yes, just "do it" and if I come back, give me a status report and progress indicator."  And also throw a badge on the main UI that "Disk X is currently being prepared for addition to the array.  Estimated XX hours remaining."

Couldn't agree more, just wanted to see someone else say it too I would have voted integrated preclear way before new web gui if I have had the chance. It will have to be implemented as part of the unRAID core, anything else will be a hack.

 

To clarify my original idea, it was never intended to take anything away from the unRAID or addon development but instead fuel discussion whether they will ever, with the current plan, produce something which will be easily taken into use by an average user. I've been doing computers for over 20 years (though unRAID was the first Linux in 2009). And I've spent hundreds/thousands of hours studying and fine tuning my systems to provide the perfect whole house av system for me and my family. Looking at the unRAID and Plex forums and all the hundreds or thousands threads and all the mis/old information and people spending their precious time one wonders why this has to be so complicated. In my mind it shouldn't and we could start by providing a simple stable NAS platform. It wouldn't have all the bells and whistles but it would have four wheels, a motor and driving wheel... and perhaps a gas and brake pedal + some other critical features. But that's what we are trying to figure out here

 

There are a lot of brilliant people in the unRAID and Plex development teams and community. I wonder what they could achieve if they had a common goal? I just finished adding a Samsung Smart TV and AppleTV3 to the list of unRAID/Plex devices in our house and it just blows my mind how much you can achieve with these inexpensive/free software.

 

There are a lot enthusiasts like myself who want to tinker and discover new things but there is a lot of people who simply miss out from these things just because it's too hard. Someone having recently set up ATV3 with PlexConnect probably would share my feeling.

Sleep --- not only no but hell no.  Too HW dependent.  Too fraught with peril

UPS support --- less HW dependent, but still HW dependent.  I vote no.

LM Sensors --- less HW dependent, but still HW dependent.  I vote no.

[snip]words of wisdom drawn from unRAID history and personal experiece[/snip]

HW dependency is a good point. Sensors can be left out the basic platform but I would reserve the right to strive for something like this.. It's light years away of becoming a solid plugin but I'm running it on all my three servers and it's giving me tons of info especially when Plex gets involved (streaming/transcoding).

 

UPS support is a must and it can be provided with a selected brands/models. I agree that no generic support can be provided. But since APC UPSs are so widely available and cheap (less than one 4TB drive) it cannot be a problem. And they are good too.

 

Sleep seems to divide opinions. Since I'm using unRAID for Plex, the box is always running. I don't think there is any standardised way for consumer electronics to wake each other over IP (like WOL) so at the moment I wouldn't put it the list. The day they implement something like the HDMI-CEC for IP, I would re-consider. For now I would concentrate getting the idle consumption as low as possible.

 

bubbaQ also had a good point about the open file issue. With all the different addons running with different kinds of start/stop mechanisms you quite easily end up hanging on the powerdown. The platform together with a proper plugin architecture should take care of this. To my understanding external connections and open files will not prevent graceful shutdown. So if we make sure (=test) that every addon adheres to whatever is agreed regarding start/stop and since there is no need for shell access for the average user, there should be no problem.

Yes, many open files would be gracefully closed by the shutdown process telling plugins to execute their shutdown script.... but some not.... and some plugin shutdowns may end things uncleanly leaving open files.

 

Once you get hung trying to unmount, you can't get to the GIU to see what files are hanging things up.

 

Optimally there should be a step AFTER all plugins are down, but BEFORE unmounting starts, that checks for open files.

 

For now I would concentrate getting the idle consumption as low as possible.

 

I have done that project too in the past.  But currently, I don't give a rat's pajamas about that... I've got a HW RAID card sucking 80 Watts all by itself.  My point is some things are common to a LOT of users, and some are not.  E-mail IMHO has much wider appeal and utility, and is hardware independent to boot.

For now I would concentrate getting the idle consumption as low as possible.

I have done that project too in the past.  But currently, I don't give a rat's pajamas about that... I've got a HW RAID card sucking 80 Watts all by itself.  My point is some things are common to a LOT of users, and some are not.  E-mail IMHO has much wider appeal and utility, and is hardware independent to boot.

Aah, I meant that if someone would be interested in sleep it had to be related to energy consumption in which case low TDP processor, high effiency psu and minimising spinning disks is the way to go. This would have nothing to do with appliance sw platform since unRAID already provides the means do minimise spinning disks. Well, to be exact you should install all disk related addons like Plex on (ssd) cache disk to allow array disks to spin down.

For now I would concentrate getting the idle consumption as low as possible.

I have done that project too in the past.  But currently, I don't give a rat's pajamas about that... I've got a HW RAID card sucking 80 Watts all by itself.  My point is some things are common to a LOT of users, and some are not.  E-mail IMHO has much wider appeal and utility, and is hardware independent to boot.

Aah, I meant that if someone would be interested in sleep it had to be related to energy consumption in which case low TDP processor, high effiency psu and minimising spinning disks is the way to go. This would have nothing to do with appliance sw platform since unRAID already provides the means do minimise spinning disks. Well, to be exact you should install all disk related addons like Plex on (ssd) cache disk to allow array disks to spin down.

 

Speeding_ant was going to change the sleep and  mail plugins so that they would work with out SF Core but looks like that will never happen, if I had any idea how to do it myself i would as the sleep plugin was great, it would put my server to sleep at 01:00 provided the mover wasn't or any other specified job was working and then I would just wake server up in morning, zero issues.

...and then I would just wake server up in morning, zero issues.

This is the part I'm a bit worried. How do you wake the server up, automatically through something you do on one the clients or something else? In my mind it should not require anything explicit to be done or specific client to be used.

 

But this is good stuff, if people would not tell how they are using unRAID and what works for them and what not, we couldn't come up with the common sw platform.

 

Anyone with an idea on the "firmware" upgrade mechanism?

Regarding the platform update mechanism; I see some problems in the current plugin framework when it comes to independence. As it is designed now, a plugin will download all the packages and other required resources when it is installed. Unfortunately since there is no common repository for such resources, plugins refer to download locations which can easily become invalid. I think this is a design flaw. In my mind the plugin should contain all the necessary resources in itself. Does anyone know why it was designed to be like this?

 

A plugin can of course have dependencies or pre-requisites which should be installed before installing the plugin. For instance if doing a server re-installation from scratch you should be able to trust that the installation media used for the original setup will work exactly the same way. This is not currently the situation, a single missing download link will break the whole thing.

 

You can capture the state of the unRAID after the installation of the plugins complete with all the required packages. And this is basically pretty much what I was planning to do with the SW platform, install unRAID and all the basic plugins and take an "image" of it. This works quite nicely for initial installation but I haven't got my head around on a proper update mechanism. Has any of you thought about such a thing? In general, is there something totally wrong with this approach?

Another problem is packages using libs that break other apps.

 

I think Plex or Newznab or something installs a lib that breaks nzbget and something else breaks lighttpd -- I can't get lighttpd to work any more. It's a mess.

Static linking.    I'm serious.  I have a number of apps that I link statically specifically so they will always work and nothing else can FUBAR them.  It will also provide another layer of protection against other packages or plugins that install different libs.

install unRAID and all the basic plugins and take an "image" of it. This works quite nicely for initial installation but I haven't got my head around on a proper update mechanism.

 

This exactly what I have done when I have worked on/built unraid servers for friends. Since my server has everything I need and its all working together properly I just take a copy of the flash drive. Of course some files need to be removed and replaced with their specific config and some config files need to have log in information changed and what not. This was definitely the easiest way before plugins were around, plugins make add everything slightly less painful. I've always been surprised why people didn't offer up "images" of their flash drive (sensitive info removed of course). Copy, paste, edit appropriate config files and you more then likely will have a working clone of a full built up server that has lots of plugins.

 

I'm probably going to get shot over this but I think it would be nice for there to be a version that comes with all tested and proven working plugins. All packages already in the install/image. (Minimum flash drive sizes are at least 8GB these days anyhow) When the user installs unraid then be asked which plugins they would like to have enabled and then walked through setting up the config file for each plugin. The disabled plugins will still have the packages there but won't be running. When an update comes out it will be for the image as a whole so compatibility between plugins will be tested and different versions of the packages/dependencies will be avoided.

 

Of course there is more to that. Running vanilla unraid first to test your hardware compatibility will of course be advisable. And this definitely wouldn't be for everyone. I have a hard time believing that majority of users here do not have at least a good handful of plugins running on their servers. Look what all the user here use unraid for, serving up media. And a lot of users have come here because of a need to storage their media that is being accessed by clients (XBMX/PLEX, and more)

 

Personally I find unraid has gotten a lot better with plugins, they definitely need some unification, but its a lot easier then installing manually and running from the go script. But I can see how new users are having a hard time getting the functionality they want out of unraid because I think a lot of the new users are coming from media front end forums (XBMC, PLEX, etc)

 

An open-files indicator needs to be right there next to the array stop button.... heck it needs to popup a warning if you try to stop the array with open files.

This is very true! That is the cause of most of my headaches with unraid and most of my unclean shutdowns.

Static linking.    I'm serious.  I have a number of apps that I link statically specifically so they will always work and nothing else can FUBAR them.  It will also provide another layer of protection against other packages or plugins that install different libs.

OK, all package developers should do that then, please. It's not like anyone running add-ons will be short of storage space or RAM.