64 Bit unRAID running natively on Arch Linux with full hypervisor support

[sparklyballs]

Do I get backstage passes if I become a groupie ?

[WeeboTech]

Before this gets too hairy, Let's keep it impersonal and on topic of the item(s) at hand.

FWIW, Personal attacks get bilged rapidly.

 

Since the two people who are key members of this thread brought up the topic of donations for effort, Finka's questions are on topic, albeit not on the pure technical side of this topic.

 

I would ask that we refrain from the topic of donations (and merit) in this particular thread so that the technical merits could be discussed without too many tangents.

I'm not looking to squash it, merely direct it elsewhere.

Start a new thread if y'all want to discuss donations and merits of donations in more depth.

[tr0910]

Before this gets too hairy, Let's keep it impersonal and on topic of the item(s) at hand.

FWIW, Personal attacks get bilged rapidly.

 

Since the two people who are key members of this thread brought up the topic of donations for effort, Finka's questions are on topic, albeit not on the pure technical side of this topic.

 

I would ask that we refrain from the topic of donations (and merit) in this particular thread so that the technical merits could be discussed without too many tangents.

I'm not looking to squash it, merely direct it elsewhere.

Start a new thread if y'all want to discuss donations and merits of donations in more depth.

 

Been watching this with huge interest, but haven't even voted yet.  Brighter minds than me can decide those details.

I'll create a lounge topic where we can discuss Kickstarter, funding, donations, and other off topic stuff. 

 

http://lime-technology.com/forum/index.php?topic=30823.msg277164#msg277164

[nicinabox]

The distro package manager should definitely be leveraged, I'm just not sure that it should be the primary mechanism for plugins. They could easily take advantage of it though. Boiler essentially functions as a url shortener--very simple to implement, but gives tons of convenience. I had to build trolley in order to get basic distro package manger functionality. I'd love to be able to offload that to the distro.

 

I agree on using a more lightweight approach for this...but it also should not open up all gates to let a user start a mess with his/her core system.

've already seen it mentioned here somewhere...what about something like this: https://www.docker.io/ to deliver apps that are not  part of the main distro?

 

I actually considered using docker to distribute boxcar. It's a great piece of software, but it's really a bit heavy handed for plugins and probably not the right tool. Plugins aren't PAAS applications. Users aren't doing continuous integration with unRAID. Users aren't deploying "apps" per se or scaling services. Plugins ARE small, distributable packages that may have slackware package dependencies. They're more akin to ruby gems, node packages, or bower packages.

 

On the topic of opening up all gates: it's something worth thinking about and considering, but it's not a real problem, imo.

 

1. Users can install anything they want right now, with or without a tool. There's nothing really stopping users from buggering their systems.

2. The same "problem" exists in all package systems. What's to stop debian users from breaking their system through apt? Or osx users doing the same through brew?

3. Quality assurance is important to prevent system package compatibility problems. Devs should test their plugins thoroughly.

4. Good tools foster a higher quality end product, help prevent bugs, and give a better experience to the end user.

[ironicbadger]

Hi guys!

 

A lot has been said over the past 24 hours or so since I last posted and given the way that the thread has matured I thought it may be interesting for you know where I stand.

 

I almost don't know where to begin, it's such a hot topic!

 

Looking at the feedback and discussion I think I shall be going with CentOS for the first tentative release sometime in the early New Year. I am going away for 2 weeks to the parents tonight so will be unable to do much in the way of dev work for this period on the project.

 

The aim here, just to make absolutely sure everyone understands, is to make unRAID run bare metal (as it does now) BUT (here's where it starts being different) install it to a local hard drive. It will still require the USB key to function and read config files from (through the usage of symlinks). Your data drives will be directly connected to the host (which is unRAID) and everything else will still function exactly like it does now, at least that's the plan. The ONLY thing that changes is that rather than the OS being in a read only ramfs it is running from a local disk, which means that you can easily install software ONCE and not everytime on boot. This is different from the current model of unRAID but the same as almost every other computing product in existence!!

 

Now, because you are running CentOS (with the unRAID raid module running too and emhttp) you have access to every piece of software that has been put into the package manager (a lot). Think, XBMC running on the SAME OS as unRAID (perfect for people with older non vt-d / iommu capable hardware that can't do passthrough). This is why it's worth doing in and of itself but it doesn't end there.

 

IF a user CHOOSES to then virtualise using KVM or Xen they are able to store all the datastores using unRAID or whatever they CHOOSE. The whole point of this endeavour, certainly from my side of the desk, is to increase the CHOICE that users have.

 

Many valid points have been made and repeated so I will not comment directly on all them, but a few below caught my eye...

 

The poll can be modified to allow users to change their votes. If the OP wants he, or we the moderators, can make an adjustment.

 

Where, I don't see the option? Please go ahead and modify the poll to allow this.

 

It would be a good time to have a hard stop to 32 Bit, those Plugins and Slackware and have unRAID 6.0 be a fresh / new start for 64 Bit and new 64 Bit Plugins.

I'm guessing that this is a direction Tom would not go in as there is still a subset of x86-only users.  Tom would be VERY reluctant to disregard them if they paid for a license.

John

 

As for what we are discussing here, I think it is the very future of unRAID itself. I would honestly like to know just how many people run the latest releases on 32 bit hardware, I'm better it's minority. As previously stated, those users are unlikely to be interested in what's being discussed here anyway! So, why not put a marker in the sand and say quite clearly:

"32 bit users, v5 is your last supported release - you've had version 4 and 5. Whilst there will be no / minimal upgrade fees to v6 this will fairly reflect the work that has gone into making unRAID+ (or whatever you want to call it) a product to last the next 5-10 years."

 

What's clear from this angle is that the current userbase is protected and only left behind if they choose to be. They still have the product they paid for and it's quite reasonable to expect a developer to charge for major new releases of software, even if there's one price for existing customers and another for newbies.

 

I think that is a fair option going forward...

 

I am still  giving this more thoughts...debating about virtualization is nice but is this the right way for making unRAID more poular?

 

I think unRAID needs another deployment/distribution method...more like google or apple does it.

For this the components architecture needs an overhaul.

Yes, I think the existing parts md-module and emhttp need to be re-worked.

 

1. md-kernel module ...this is the real technical CORE, keep it...make it available to everyone..it is open-source already....everyone can incorporate it into a distro or installation of his/her own.

2. integration app/module - this is new...it does everything emhttp does today, but without a GUI....it needs to provide an open API...maybe based on REST...*this* is the real product.

Make it available with the package mechanisms of all major distros for users to purchase

3. UI - this is interacting with the API of no.2 ...can be a Web-UI, a mobile app, a server agent...whatever you wish....this is nothing more or better/other than a plugin...because of the APi, others can roll their own.

Maybe because of no. 2, the first UI will be a plugin for that distro/package where the user purchased unRAID for.

I think the point has already been made that if someone wants to rip off unRAID, with or without tom and profit from it then it isn't a difficult process for some talented people on here, but that isn't what is being discussed. If people are willing to donate long hours to assisting what is a stand alone business or helping a community then there can't be an issue with that

 

Exactly. The md-module is open source, the unRAID 'product' really is the webGUI. I'm not here to steal Tom's work, if I was do you really think I'd be posting about it?!

 

I and others are quite willing to make this a reality but don't want to piss in the wind if it's not a direction Tom wants to go. From his posts in this thread I get the feeling he's just cautious - which is justified I guess - about taking these baby steps. But what about this for an idea, you leave the existing unRAID model, on Slack, as is and he keeps doing what he wants with it as is the status quo. But, allow a certain few of us more developmentally minded members access to how emhttp works and improve it with the end goal of making things better for everyone.

 

I did not see any dancing at all.  grumpy merely said that if some would like to throw some $$$ at IronicBadger it would be appreciated(being a poor college student).    I don't think it can get more direct than that.

 

Now so far as my involvement in asking for donations it has been zero. I am a 'poor college student' that is true, but I havn't asked for any money. If, however, you lovely people so inclined as to donate to me / the project then ofc I would be a silly boy to refuse - wouldn't I? I would like to make sure that users such as Finka don't think I'm here to money grab, I'm not!

I'll have to think a bit more on it as doing this for fun is one thing but the minute you accept other peoples money you have to be sure you can deliver what you promise. A development server would be a 'nice to have' as a result of any donations I suppose. As I say, it would be wonderful if you'd like to donate to me but I'd like a bit more time to be sure about what I'm promising first!

 

 

He will also provide a repo where you can do...

 

slackpkg install mysql couchpotato plexmediasever etc.

 

If you want to install various packages. Since the Slackware repo sucks he can add the popular ones or users requests so none of you have to compile stuff.

 

Would any of you guys be willing to donate money to Ironic if he does that and maintains a repo with software which it updates with new versions as they come out?

 

While this is somewhat in the right direction, this type of gatekeepery is not conducive to distributed growth. It keeps software in silos, unable to be reviewed, changed, or improved by anyone but the creator.

 

What's really needed is a package distribution system that doesn't rely on one person to function. Developers should be able to publish and distribute packages easily, and users consume them in the manner above.

 

I'm totally willing to do this. I do not wish to be 'the gatekeeper' but someone has to have overall responsibility - not to mention bring the thing to life. I'm sure a few users here would be more than capable of policing such a repo between us, but with a repo come things like hosting costs etc. How would these be met?

 

 

 

Phew! Time for a cuppa.

[sparklyballs]

The same benefits of having a full distro in that it allows XBMC to be run on machines that don't support passthrough could equally be provided to run tvheadend or mythtv or any other linux based PVR could it not ?

[prostuff1]

I and others are quite willing to make this a reality but don't want to piss in the wind if it's not a direction Tom wants to go. From his posts in this thread I get the feeling he's just cautious - which is justified I guess - about taking these baby steps. But what about this for an idea, you leave the existing unRAID model, on Slack, as is and he keeps doing what he wants with it as is the status quo. But, allow a certain few of us more developmentally minded members access to how emhttp works and improve it with the end goal of making things better for everyone.

As a dev and business owner myself doing the above is a very, very, very hard choice to make.  Generally speaking a person/company ALWAYS wants to hold onto there own code.

 

If Tom where to "open" emhttp to some of us fellow devs I have no doubt that legally binding contracts and the like would be involved, he would be stupid not to make it a requirement

 

 

I personally would never want to install XBMC on my unRAID machine, I have nettops, fanless pcs, and RPi's for that duty.  Running a mysql DB from the unRAID box, yes but that would be the extent of it.  And I do that now, with the cache drive and plugins.

 

I do agree that package management, dependency fragmentation and the like have gotten out of hand.  Something probably needs to be done to clean that up.

 

 

What originally brought me to unRAID and made me decide to use it was the KISS.  When something went wrong there was a small subset of things that it was likely to be, because the system was a "closed" box.

 

 

Very interesting stuff, looking forward to seeing what comes of it.

[WeeboTech]

The poll can be modified to allow users to change their votes. If the OP wants he, or we the moderators, can make an adjustment.

 

Where, I don't see the option? Please go ahead and modify the poll to allow this.

 

I've modified the poll to allow switching votes.

[jumperalex]

 

The aim here, just to make absolutely sure everyone understands, is to make unRAID run bare metal (as it does now) BUT (here's where it starts being different) install it to a local hard drive. It will still require the USB key to function and read config files from (through the usage of symlinks). Your data drives will be directly connected to the host (which is unRAID) and everything else will still function exactly like it does now, at least that's the plan. The ONLY thing that changes is that rather than the OS being in a read only ramfs it is running from a local disk, which means that you can easily install software ONCE and not everytime on boot. This is different from the current model of unRAID but the same as almost every other computing product in existence!!

 

Based on your description does this mean it would be possible to have this setup such that removing the OS HDD (or a failure of said HDD) would revert the entire setup to a bare-metal flash-loaded ramFS instance of UnRaid?

 

If the system were configured to keep the unraid essentials synched to flash, then if the boot HDD failed the user would config their bios to boot from flash to continue to provide access to data.  I think that would go a loooong way towards the goals of reliability and flexibility for anyone concerned about a perceived move away from the "appliance" model.  And frankly just sounds like a good selling point.

[kegler]

The poll can be modified to allow users to change their votes. If the OP wants he, or we the moderators, can make an adjustment.

 

Where, I don't see the option? Please go ahead and modify the poll to allow this.

 

I've modified the poll to allow switching votes.

 

One more related request - For those of us who are linux novices and have no idea which would be best, add another category such as "I have no idea - I'm clueless"

 

The "Other" category just does not work.  Who is going to go through 14 pages of posts looking for them?

[WeeboTech]

The poll can be modified to allow users to change their votes. If the OP wants he, or we the moderators, can make an adjustment.

 

Where, I don't see the option? Please go ahead and modify the poll to allow this.

 

I've modified the poll to allow switching votes.

 

One more related request - For those of us who are linux novices and have no idea which would be best, add another category such as "I have no idea - I'm clueless"

 

The "Other" category just does not work.  Who is going to go through 14 pages of posts looking for them?

 

OK, I've modified the poll as per your suggestion.

[grumpybutfun]

Speaking of Slackware vs CentOS...

 

I would be more than happy to send Tom a CentOS 6.5 bzimage and bzroot that looks almost exactly like the Slackware one and just as small.

 

Really the only difference Tom would notice is CentOS using systemd and not the god awful Sysvinit.

 

My views on Systemd versus Sysvinit...

 

I hope this all comes out in English so everyone can understand...

 

Hell

 

Until the last year or so, all Linux Distro's used udev and Sysvinit (Ubuntu / Slackware / unRAID still uses this) for System Management (start / stop programs / services / daemons).

 

With udev and Sysvinit, it meant every Linux Distro had their own unique way of installing / starting / stopping programs / services / deamons / etc. and none were the same. Even for experienced Linux Users, that meant you had to "start over" and learn A LOT of things when you switched Linux Distros. It was also a headache for the people who make / create / maintain the various Linux Packages (For example: NFS, Samba, XBMC, etc.) and each Linux Distro Package Manager had to create / write a custom script for the installation and start up of their script. Also, Sysvinit is DUMB. They are basically scripts that fire off and enable, disable, start, stop, etc. It doesn't know / keep track of what is started, stopped, etc. Also, doing this in Ubuntu is quite different than say... Slackware or Debian. Each Linux Distro has their own method and scripts.

 

Example: Autostarting and stopping XBMC in Ubuntu isn't easy and varies from version to version. I centralize my various XBMCs into one database instead of maintaining 4 separate machines. Because my PC boots up so fast, its pure hell to find the correct start up method to make XBMC start after mysql has. I would have to google it and come up with 50+ ways of doing it and half of them do not work.

 

Heaven

 

Say hello systemd!

 

A system management daemon designed exclusively for the Linux kernel API. Like sysvinit, systemd is a daemon that manages other daemons. However, It's the first daemon to start and the last one to terminate (during shutdown) and it's SMART. It knows what is started, stopped, running, etc. and it's a STANDARD.

 

What this means...

 

Linux Distros that have switched over to Systemd all look / feel / work the same. A systemd daemon is agnostic to which Linux Distro it is running on. Really the only difference in between Linux Distros running systemd is the package manager. The same systemd daemon that works in CentOS also works in Arch Linux or openSUSE.

 

A real life example of systemd in action on various systemd Linux Distros:

 

To install XBMC:

 

yum install xbmc <---- For CentOS, Fedora, Red Hat
pacman -S xbmc <---- For Arch
yast --install xbmc <---- For openSUSE

 

To enable XBMC to start on boot:

 

systemctl enable xbmc.service <--- This works on CentOS, Arch, openSUSE, Red Hat, Fedora, etc.

 

If you want to start, stop, restart XBMC on any of the systemd Linux Distros:

 

systemctl start xbmc.service
systemctl stop xbmc.service <--- All of these commands work the EXACT same on CentOS, Arch, openSUSE, Red Hat, Fedora, etc.
systemctl restart xbmc.service

 

Again I wanted to start xbmc.service after mysql. Lets see how easy that is using systemd:

 

The XBMC systemd (looks the same on all systemd Linux Distros):

 

[unit]

Description = Starts instance of XBMC using xinit

After = mysqld.service <--- That is the only change I needed to make.

 

[service]

User = xbmc <---- If I wanted to change the user that starts it, look how easy that is.

Group = xbmc

Type = simple

ExecStart = /usr/bin/xinit /usr/bin/xbmc-standalone -- :0 -nolisten tcp <--- If I wanted to change how / where XBMC is located or how it starts... look how easy it is to tell Linux what to do.

Restart = on-abort <-- If XBMC crashes (which happens). Just restart it so I don't have to get off the sofa or do anything.

 

[install]

WantedBy = multi-user.target

 

 

What this all means:

 

If you pick a Linux Distro that has Systemd and learn it. Switching from CentOS, Fedora, Red Hat, Arch, openSUSE, etc. is easy. The only thing you have to learn is the Package Manager and how to add "repos" to them. Everything else after that is the same.

 

If you haven't noticed, the reason I can't stand Ubuntu, Mint, Slackware, Debian, etc. is due to the fact they haven't switched over to Systemd yet.

 

Trying to write out a guide for doing Xen or KVM in any of those is a MFer too and it's way to hard to explain to Novices how to do configure it without it being 10 pages long. Trust me, I tried and failed.

 

Look how much more information you would get during boot. On whether or not a service / process started, stopped, failed and has warnings:

 

 

Again, this gives Tom A LOT more control over things and instead of unRAID being stupid. Tom and the plugin guys would know if Samba Started, NFS or any other service needed by Tom or the Plugin guys. You wouldn't have to go check on just PID files (which it's 100% accurate). If you are using a plugin that has two or more services needed. It would know to stop / start / reset them both or if one wasn't running. A lot of the hoops (and lots of code) the tvheadend plugin has to jump through to work correctly would be addressed just by switching to systemd.

[WeeboTech]

I looked at this the other day.

I was intrigued.

I suppose we wouldn't need go script modifications anymore.

Perhaps the plugin system could be revamped or managed differently also.

emhttp could call out to systemctl to start/stop services.

 

[ironicbadger]

I looked at this the other day.

I was intrigued.

I suppose we wouldn't need go script modifications anymore.

Perhaps the plugin system could be revamped or managed differently also.

emhttp could call out to systemctl to start/stop services.

 

Emhttp itself would also be a systemd service!

 

Sent from my Nexus 5 using Tapatalk

 

 

[Ice_Black]

How about this:

 

- unRAID running natively on CentOS Linux via USB

- XEN Support and ability to create VM's from web GUI

- Change Reiser4 to modern Linux file system.. ZFS?

 

- It would be better to install XBMC or any apps on a VM.  To keep it separate from unRAID.

 

on ESXi 5.5, I have a unRAID VM and other Linux VM for sabnzbd, etc.

[WeeboTech]

I looked at this the other day.

I was intrigued.

I suppose we wouldn't need go script modifications anymore.

Perhaps the plugin system could be revamped or managed differently also.

emhttp could call out to systemctl to start/stop services.

 

Emhttp itself would also be a systemd service!

 

Agreed, but there may be dependancy issues.

emhttp currently creates the samba and nfs export files.

Restarts the daemons in question.

 

In it's current methodology, emhttp is almost it's own system service manager.

time, samba, nfs, afp, usershares.  So there's allot to consider there.

[ironicbadger]

How about this:

 

- unRAID running natively on CentOS Linux via USB

- XEN Support and ability to create VM's from web GUI

- Change Reiser4 to modern Linux file system.. ZFS?

 

- It would be better to install XBMC or any apps on a VM.  To keep it separate from unRAID.

 

on ESXi 5.5, I have a unRAID VM and other Linux VM for sabnzbd, etc.

 

Sure but what about people who can't passthrough due to hardware limitations?

 

My overarching point is that people would have the choice to do either approach.

 

Sent from my Nexus 5 using Tapatalk

 

 

[grumpybutfun]

I looked at this the other day.

I was intrigued.

I suppose we wouldn't need go script modifications anymore.

Perhaps the plugin system could be revamped or managed differently also.

emhttp could call out to systemctl to start/stop services.

 

Yes.

 

Also think how much easier it would be to support / help users.

 

How many "I can't access the unRAID WebGUI" posts do we get?

 

We have no idea why and have to go through a bunch a crap to determine if the network works, hostname was set, unRAID flash drive was mounted, tmpfs for logs (and other stuff) was created, NFS, Samba, etc. was started, etc.

 

With systemd the users would know clear as day (with an error message and blurb about what is wrong) when they boot up.

 

Example:

 

 

Instead of "I can't access the unRAID WebGUI" posts we would get "My Network didn't start" or "unRAID can't mount the unRAID USB Flash Drive" or "My hostname is not being read", etc.

 

Support and assisting users with their issues is much easier for Tom and everyone else here.

 

 

[Finka]

 

My views on Systemd versus Sysvinit...

 

Hell

...

Heaven

...

this gives Tom A LOT more control over things and instead of unRAID being stupid.

I don't think unRAID is stupid.  Nor do I think it's Hell vs Heaven.  This is the third time I've seen this exact same HUGE post copied/pasted in this forum.  Posting it three times doesn't make it more convincing.  That's what I meant when I compared this thread to a shouting match.

 

 

[kizer]

I choose I'm clueless

 

Even thou I have years of Redhat, FreeBSD and Slackware experience.

 

I will admit Slackware was/is one of the oddest ones in the bunch and a different level of community support.  RTFM is what I normally got when I asked for help.

 

Redhat and FreeBsd was/had the easiest Package Managers ever to use. Type a few commands and wham'o something was installed or updated and took hardly any effort what-so-ever.

 

However Tom/unRAID has made my job so much easier with just slapping in a USB stick and forgetting about everything else. Keep it simple and keep it useful. I'm not tempted to run 1234534's of apps on a Server that was built just to house my data. I can't be the only one that has ran a Linux server and next thing you know its doing tons of stuff and then you step back and think. "Uh what did I do to this thing?"

[grumpybutfun]

Agreed, but there may be dependancy issues.

emhttp currently creates the samba and nfs export files.

Restarts the daemons in question.

 

In it's current methodology, emhttp is almost it's own system service manager.

time, samba, nfs, afp, usershares.  So there's allot to consider there.

 

It creates the shares using the Samba shares with config files that are saved unRAID USB Flash Drive.

 

Tom would only need to change the start / stop / restart command.

 

Instead of...

 

/etc/rc.d/rc.rc.samba restart

 

It would be...

 

systemctl restart samba.service

[WeeboTech]

 

My views on Systemd versus Sysvinit...

 

Hell

...

Heaven

...

this gives Tom A LOT more control over things and instead of unRAID being stupid.

I don't think unRAID is stupid.  Nor do I think it's Hell vs Heaven.  This is the third time I've seen this exact same HUGE post copied/pasted in this forum.  Posting it three times doesn't make it more convincing.  That's what I meant when I compared this thread to a shouting match.

 

I thought his opinion was informative, well thought out and supported with examples.

It was an opinion with constructive criticisms.

 

While I agree, unRAID is not stupid, it's discussion, debate and supported examples that bring about new understanding.

[sparklyballs]

I don't think he was trying to insult UNRAID....

 

 

[grumpybutfun]

I don't think unRAID is stupid.

 

Regardless of what you say / think....

 

It's a PROVEN fact that Sysvinit is indeed STUPID and therefore Linux (and all the other services / daemons) do not know if Samba, NFS, etc. is enable / disabled / working.

 

If you go look at the Sysvinit manual, you will CLEARLY see that it is just a Linux script. Where as Systemd is an ACTUAL system management daemon designed exclusively for the Linux kernel API and a industry standard.

 

Not only that a MAJORITY of Linux Distros have already switched to systemd and Debian / Ubuntu / Etc. are going to in the next release.

 

Now I ask you, do you think Red Hat, CentOS, Arch, openSUSE, Debian (which is switching soon) is doing that because it makes things much harder for the users / developers / Linux Distros / packages or because there is a unified system wide industry standard to make like easier for everyone?

 

As much as you seem to love Sysinitv... Slackware is ALSO switching to it. Unless you keep your unRAID at the current version it is now... Regrettably, you are going to be using Systemd whether you like it or not.

[ironicbadger]

 

My views on Systemd versus Sysvinit...

 

Hell

...

Heaven

...

this gives Tom A LOT more control over things and instead of unRAID being stupid.

I don't think unRAID is stupid.  Nor do I think it's Hell vs Heaven.  This is the third time I've seen this exact same HUGE post copied/pasted in this forum.  Posting it three times doesn't make it more convincing.  That's what I meant when I compared this thread to a shouting match.

 

I thought his opinion was informative, well thought out and supported with examples.

It was an opinion with constructive criticisms.

 

While I agree, unRAID is not stupid, it's discussion, debate and supported examples that bring about new understanding.

 

I think unraid being stupid in this context meant its current usage of rc.d system isn't as clever as the system implementation for all the reasons listed.

 

I very much doubt grumpy meant "unraid itself is stupid".

 

Come on man, positive intent please.

 

Sent from my Nexus 5 using Tapatalk