64 Bit unRAID running natively on Arch Linux with full hypervisor support



Recommended Posts

we need to keep pressing him for is a 64 bit emhttp would speed things up EVEN MORE.

Speed what exactly, and how?  Normally emhttp just sits there doing nothing, until you click something in your web browser, then it serves you a web page, and goes back to doing nothing.  How is that speed-critical for anything? 

 

 

Link to comment
  • Replies 451
  • Created
  • Last Reply

Top Posters In This Topic

we need to keep pressing him for is a 64 bit emhttp would speed things up EVEN MORE.

Speed what exactly, and how?  Normally emhttp just sits there doing nothing, until you click something in your web browser, then it serves you a web page, and goes back to doing nothing.  How is that speed-critical for anything?

 

unRAID in a 64 Bit Linux Kernel is a no brainer. However, emhttp is a 32 bit program so it require you running multilib (64Bit and 32Bit). Linux can do that no problems but its not ideal. Keeping 64 and 32 Bit libraries is unneeded. Even the Linux Kernel dropped i386 support from the Kernel. Anyone bought a 32Bit processor in the last 3 or 4 years?

Link to comment

we need to keep pressing him for is a 64 bit emhttp would speed things up EVEN MORE.

Speed what exactly, and how?  Normally emhttp just sits there doing nothing, until you click something in your web browser, then it serves you a web page, and goes back to doing nothing.  How is that speed-critical for anything?

 

unRAID in a 64 Bit Linux Kernel is a no brainer. However, emhttp is a 32 bit program so it require you running multilib (64Bit and 32Bit). Linux can do that no problems but its not ideal. Keeping 64 and 32 Bit libraries is unneeded. Even the Linux Kernel dropped i386 support from the Kernel. Anyone bought a 32Bit processor in the last 3 or 4 years?

I get that.  I've been runnung the last few months my own unRAID build with 64-bit kernel without the need of any multilib (64-bit kernel with the stock 32-bit userspace), and it's been rock-solid.  My comment was only to ironicbadger that he can not possibly expect any speed improvements to come from emhttp, as emhttp is not affecting the speed of anything.

 

 

Link to comment

I get that.  I've been runnung the last few months my own unRAID build with 64-bit kernel without the need of any multilib (64-bit kernel with the stock 32-bit userspace), and it's been rock-solid.  My comment was only to ironicbadger that he can not possibly expect any speed improvements to come from emhttp, as emhttp is not affecting the speed of anything.

 

Agreed but symbolic linking all the 64 libraries to what emhttp is looking for isn't what I would consider a "good time". Makes managing updates a nightmare to support.

 

Symbolic Link libcrytpo.so.0.12.0 to libcrypto.so.0 and so and so on. There is an update to libcrypto so now you need to remove the old Symbolic Link to libcrytpo and create a new one. Imagine writing the documentation for that. LOL! Easy for you and I and other Linux Veterans but I wouldn't dare ask a Linux Novice who want / need access to TBs of important data to manage that process.

 

Hopefully it's not to long before Tom releases his 64 bit version of unRAID (and emhttp). That way, we don't even need to worry about 32Bit support for emhttp.

Link to comment

we need to keep pressing him for is a 64 bit emhttp would speed things up EVEN MORE.

Speed what exactly, and how?  Normally emhttp just sits there doing nothing, until you click something in your web browser, then it serves you a web page, and goes back to doing nothing.  How is that speed-critical for anything?

 

unRAID in a 64 Bit Linux Kernel is a no brainer. However, emhttp is a 32 bit program so it require you running multilib (64Bit and 32Bit). Linux can do that no problems but its not ideal. Keeping 64 and 32 Bit libraries is unneeded. Even the Linux Kernel dropped i386 support from the Kernel. Anyone bought a 32Bit processor in the last 3 or 4 years?

I get that.  I've been runnung the last few months my own unRAID build with 64-bit kernel without the need of any multilib (64-bit kernel with the stock 32-bit userspace), and it's been rock-solid.  My comment was only to ironicbadger that he can not possibly expect any speed improvements to come from emhttp, as emhttp is not affecting the speed of anything.

 

Do any of us know the inner workings of emhttp? Nope. Therefore we cannot categorically draw any conclusions yet...  8)

Link to comment

- Proxmox-VE

I know it is Debian based but as a hypervior host it comes with the best/easiest way of managing VMs since it is truly Web-based.

IMHO this is what we should keep with unRAID, because that's what makes it a server OS and what makes it so simple.

Edit: I know that PVE is using 2.6.x Redhat kernel but the 3.10 kernel has just hit the test-repos @ PVE.

 

- Fedora

AFAIU CentOS is following RedHat lifecycle, whilst Fedora is always upstream.

The advantage of enabling the 64bit nRAID host as a hypervisor is better supported by Fedora as we would get upstream versions of kvm/qemu.

 

...this concludes my 2 cents.

Keep up the good work!

 

You know your stuff.

 

I am with you on CentOS.

 

Arch Linux while great and my Linux Distro of choice...

 

It's not for Linux Novices and it's way to cutting edge for a NAS / KVM / Xen Server Solution if you do not know your way around Linux. Their very up to date packages they release do not go through the same rigorous testing that CentOS (a 100% Red Hat Clone), Debian, etc. does. Also, since Arch is a rolling release... It would be hard to support updates / upgrades. Since there are no release versions, if one goes a long time between updates...  There is no telling what major changes have taken place and the amount of time you will have to spend to get everything working correctly again. When Arch switched from Sysvinit to Systemd you had to make sure all your Sysvinit scripts were enabled and working like you wanted in Systemd. The same experience was had when Arch switched over from netcfg (Network Manager) to netctl.

 

I suggest CentOS with an updated Linux Kernel

 

1. There were some speed enhancements for many of the File Systems in 3.12.

 

2. You get the needed Reiser FS updates / enhancements / bug fixes.

 

3. VFIO (Needed for nVidia Video Card Passthrough) and many other improvements for PCI Passthrough.

 

4. HD Audio and VDPAU Video Hardware Acceleration for for AMD Proprietary Drivers and Raedon (AMD Open Source) drivers.

 

5. Plus numerous updates / enhancements / improvements to KVM and Xen.

 

Along with a newer Linux Kernel, I would also use a more upstream QEMU, KVM and Xen packages to support many of the things I describe above for PCI Passthrough.

 

Proxmox is something to consider because the main issue I see is for people who want to run a Headless System. I haven't found a WebGUI for Xen / KVM that I would recommend for Linux Novices (Something like oVirt is way to complex). Proxmox-VE has developed that. Since it can run on Debian, like CentOS (Red Hat Essentially)... It's without question considered a Stable / Reliable / Enterprise worthy Linux Server Distro.

 

I would need to look at how they do their licensing for their proprietary software / WebGUIs. Since you are more familiar with it, is it under the GNU license agreement or is it like emhttp (a custom compiled binary program with no source code under GNU Licensing)?

 

Anything you could offer / educate us on would be helpful / appreciated.

 

It's great that Ironic, has offered to put in the time and effort to put together a Distro for us all. I'm sure if we continue to provide feedback that will help narrow this selection process to a couple of choices. Maybe he can do one and one of you Linux Pros (Perfect Ford perhaps?!?!) will do / support another one. I'd do it but I don't have the time / bandwidth once the new year rolls around.

Link to comment

I get that.  I've been runnung the last few months my own unRAID build with 64-bit kernel without the need of any multilib (64-bit kernel with the stock 32-bit userspace), and it's been rock-solid.  My comment was only to ironicbadger that he can not possibly expect any speed improvements to come from emhttp, as emhttp is not affecting the speed of anything.

 

Agreed but symbolic linking all the 64 libraries to what emhttp is looking for isn't what I would consider a "good time". Makes managing updates a nightmare to support.

You misunderstood.  I booted my freshly compiled 64-bit kernel with the stock 32-bit unRAID userspace -- no modifications to unRAID's bzroot whatsoever. (apart from inserting the newly compiled kernel modules in it of course.)  This way I can use more than 4GB RAM without a crappy PAE kernel, and at the same time not worry about ugly multilib packages.  It's still the same unRAID, booting from the same USB stick, but only with the additional choice of using the 64-bit kernel in the boot menu.

 

 

Link to comment

I will be following this and it may be something that I install on my test machine.  I will likely not be moving my ESXi box over to this though... if it aint broke don't fix it.

 

Good luck.

 

From my point of view, this would give us ESXi users an option from VMware.  While my system is very stable and does exactly what I expect of it now, there is no guarantee that it will remain that way.  When Win8 came out, ESXi required a patch release to support it.  I want the option to go in another direction, should VMware decide us free users are just not worth the trouble (if they haven't already done so with 5.5).

 

Will I be moving my current setup to this in the foreseeable future?  Probably not.

 

Will I be trying it out on a testbed?  Absolutely.

While I agree, I am still running ESXi 4.1 purely for the fact that if it aint broke I aint fixin it.

 

When the time comes to build a new server for myself then I will look at my options for Virtualization.

Link to comment

You misunderstood.  I booted my freshly compiled 64-bit kernel with the stock 32-bit unRAID userspace -- no modifications to unRAID's bzroot whatsoever. (apart from inserting the newly compiled kernel modules in it of course.)  This way I can use more than 4GB RAM without a crappy PAE kernel, and at the same time not worry about ugly multilib packages.  It's still the same unRAID, booting from the same USB stick, but only with the additional choice of using the 64-bit kernel in the boot menu.

 

I agree 100% if if all you are doing is unRAID with plugins as we have always done.

 

However, what we are discussing is unRAID running on a full Linux Distro along with being a Type 1 Hypervisor that doesn't have the entire root FS (bzroot) loaded into ram (read only).

 

That way users can customize, keep their system up to date and install all the various Linux Distro packages they want once and not have to rely one person (Tom) or Plugin writers (one person) to maintain / provide support / add additional functionality they want. For many of us, it seems like a waste to spend all that money on a motherboards, CPUs, memory, RAID Controllers, Network Cards, etc. and use 5% or less of the "horsepower" we have at our disposal. Sure, for many this isn't an issue and exactly what their goal was / is with unRAID and with their hardware.

 

Many of us do want to take advantage of the equipment we have and not be beholden to what one person (Tom) or what company (ESXi or XenServer) does / does not do. Many of us do not want to sit around and wait / hope someone writes a plugin for a feature / software package we want. Or have to wait for ESXi, XenServer, etc. to catch up with KVM or Xen (If you wanted to passthrough our nVidia 4xx, 5xx, 6xx Video Card to XBMC, Windows OSX, Linux VM). Not to mention, you do not have to jump through a bunch of Hardware "hoops", buy special CPUs / Motherboards, RAID Controllers to run a Type 1 Hypervisor so you can passthrough hard drives to an unRAID VM so you can run other VMs on your hardware. Only if you wanted to passthrough PCI Devices like Video Cards to a Windows / XBMC VM throughout your house would you need to concern yourself with VT-D / AMD-V CPUs and Motherboards. Otherwise, run your VMs on your Intel K Processor and save yourself from having to purchase and install a new Intel CPU and possibly a motherboard. How long have people waited for ESXi to fix PCI Passthrough? From 5.0 to 5.1 it broke it for some, fixed it for others. Same with 5.1 to "crippled" 5.5. It fixed PCI passthrough for some but plenty are still unable to upgrade because "crippled" ESXi 5.5 broke what worked in 5.1.

 

As I stated earlier, you will have over 2,000+ hardware drivers. To put that in comparison unRAID only has 170 or so.

 

All your Sata, SCSI, Network, Video, Sound, IR, TV Tuners, Fiber, Block Devices, Input, Hardware Monitoring, USB, etc. will work out of the box. If it works in Linux (Ubuntu, Fedora, Arch, CentOS, etc.) it will work on what Ironic is putting together for us all.

 

No longer will you have custom compile an unRAID kernel to add support for your TV Tuner Card, Raid / SAS / Fiber Controller when a new version of unRAID comes out.

 

Do you know how many users who I have helped with running unRAID on XenServer were running their drives IDE and not AHCI? A LOT! They were completely unaware they were doing this even though they selected AHCI in their bios. Again, those users with those motherboard / RAID controllers wouldn't need to run a script in unRAID to take advantage of AHCI support (and the added benefits / speed that AHCI offers).

 

How many of you have TBs of important data and do not have the hardware monitoring support enable in the unRAID Kernel for your motherboard, CPU, Video Card, etc.? Out of the 130+ hardware monitoring and voltage regulator drivers only 8 are enabled in unRAID.

 

Wouldn't you like to get lm_senors to work on your hardware, take advantage of mcelog and various other monitoring tools built into Linux, use Linux or other software packages to get notification via email / text when something is wrong? How many of you are running ESXi and can't get it to do fan control and or get hard drive temps? I know of one user who lost a lot of data because his fan died and his hard drives overheated. All of that could have easily been avoided.

 

Do you want encryption now or do you want to wait till unRAID adds it via the kernel and updates Slackware?

 

Bottom Line...

 

This allows unRAID users a CHOICE / FREEDOM and 100% total control how / what they want to do with the easy to use / managed storage solution that unRAID offers. Again, it's not for everyone but Hardware / Software / OSes has evolved to were running a dedicated machine simply for a NAS device has long since past. Even Fortune 500 companies, many of them do not even do this anymore.

Link to comment

I will be following this and it may be something that I install on my test machine.  I will likely not be moving my ESXi box over to this though... if it aint broke don't fix it.

 

Good luck.

 

From my point of view, this would give us ESXi users an option from VMware.  While my system is very stable and does exactly what I expect of it now, there is no guarantee that it will remain that way.  When Win8 came out, ESXi required a patch release to support it.  I want the option to go in another direction, should VMware decide us free users are just not worth the trouble (if they haven't already done so with 5.5).

 

Will I be moving my current setup to this in the foreseeable future?  Probably not.

 

Will I be trying it out on a testbed?  Absolutely.

While I agree, I am still running ESXi 4.1 purely for the fact that if it aint broke I aint fixin it.

 

When the time comes to build a new server for myself then I will look at my options for Virtualization.

 

By that logic you would still run XP, or internet explorer 6.

 

This thread is about trying to make progress, if that's not your game then that's cool.

 

Sent from my Nexus 5 using Tapatalk

 

Link to comment

Is there a reason that Slackware 64 wasn't used or suggested for a new 64 bit full distro unRAID ?

 

1. It's a pain in the ass.

 

2. It's not user friendly.

 

3. Documentation doesn't exist.

 

4. No Slackware Forum that is worth a crap where users go get help / assistance / guidance.

 

5. When you google Slackware, it's rare if you can find answers to questions or see examples of people doing X, Y or Z on it.

 

6. If you install a package that isn't in their Package Manager,  you have to compile everything unless you want to download a random version of say... XBMC, mariadb, phpmyadmin, etc. from somewhere on the internet.

 

7. Using XBMC as an example, you would have to go download the source code for the following: libcec, pulseaudio, libbluray, libnfs, libdvdcss, lirc, cwiid, pybluez, id3lib, libvdpau, rtmpdump, afpfs-ng, libmicrohttpd, libva, avahi, lame, ffmpeg, libshairport, etc and compile each of them (and each of the follow package dependencies) before you even get to the point of compiling XBMC.

 

8. The software packages in their package manager is in the hundreds. Otherwise, you will spend hours trying to determine what dependencies are needed and compiling those (not to mention the dependencies of those packages) before you get to compiling the software you want.

 

9. "apt-get install xbmc owncloud phpmyadmin" (Debian / Ubuntu) or "yast --install xbmc owncloud phpmyadmin" (openSUSE) or "yum install xbmc owncloud phpmyadmin" (Red Hat, Fedora, CentOS, etc.) or "pacman -S xbmc owncloud phpmyadmin" (Arch Linux) is 1,000,000 times easier.

Link to comment

Is there a reason that Slackware 64 wasn't used or suggested for a new 64 bit full distro unRAID ?

 

1. It's a pain in the ass.

 

2. It's not user friendly.

 

3. Documentation doesn't exist.

 

4. No Slackware Forum that is worth a crap where users go get help / assistance / guidance on issues.

 

5. When you google Slackware, it's rare if you can find answers to questions or see examples of people doing X, Y or Z on it.

 

6. If you install a package that isn't in their Package Manager,  you have to compile everything unless you want to download a random version of say... XBMC, mariadb, phpmyadmin, etc. from somewhere on the internet.

 

7. Using XBMC as an example, you would have to go download the Slackbuild (or source code) for the following: libcec, pulseaudio, libbluray, libnfs, libdvdcss, lirc, cwiid, pybluez, id3lib, libvdpau, rtmpdump, afpfs-ng, libmicrohttpd, libva, avahi, lame, ffmpeg, libshairport and compile each of them, install them before you even get to the point of compiling XBMC.

 

8. The software packages in their package manager is in the hundreds. Otherwise, you will be spending hours trying to determine what dependencies are nedded and compiling those (not to mention the dependencies of the dependencies) before you get to compiling the software you want.

 

9. "apt-get install xbmc" (Debian / Ubuntu) or "yast --install xbmc" (openSUSE) or "yum install xbmc" (Red Hat, Fedora, CentOS, etc.) or "pacman -S xbmc" (Arch Linux) is 1,000,000 times easier.

 

and that. granted unRAID itself would be easier on slackware, but EVERYTHING else would be more bothersome.

Link to comment

Proxmox is something to consider because the main issue I see is for people who want to run a Headless System. I haven't found a WebGUI for Xen / KVM that I would recommend for Linux Novices (Something like oVirt is way to complex). Proxmox-VE has developed that. Since it can run on Debian, like CentOS (Red Hat Essentially)... It's without question considered a Stable / Reliable / Enterprise worthy Linux Server Distro.

 

I would need to look at how they do their licensing for their proprietary software / WebGUIs. Since you are more familiar with it, is it under the GNU license agreement or is it like emhttp (a custom compiled binary program with no source code under GNU Licensing)?

 

Anything you could offer / educate us on would be helpful / appreciated.

 

...PVE is completely open source...everything they do is released under the GNU Affero General Public License, version 3 (GNU AGPL, v3) -> http://www.proxmox.com/proxmox-ve/features

I am not sure what this means and there are things to consider though.

 

- For sure one can contribute to PVE (which would mean to submit the unRAID md-module under the same license to the project (and with no commitment that it will be released)

 

- not sure if running "our own" release, based on Debian, by pulling the PVE source-code repos and then include unRAID md-module (best with DKMS to automate builds when the PVE repo changes ?)

  will be according to the license. I also don't think there is an option to migrate the UI to another distro, like CentOS.

 

..I am not a lawyer  ::)

 

It's great that Ironic, has offered to put in the time and effort to put together a Distro for us all. I'm sure if we continue to provide feedback that will help narrow this selection process to a couple of choices. Maybe he can do one and one of you Linux Pros (Perfect Ford perhaps?!?!) will do / support another one. I'd do it but I don't have the time / bandwidth once the new year rolls around.

I am not Perfect...barely managing to keep track of my towel  ;D

 

Is there really the need for a choice of the distro?

Maybe there is a more generic way to think of.

Integrating the md-module in an automated fashion would be best but also the first challenge...I'd opt for using DKMS.

Once this is done, it should be able to be integrated into any distro, once you have a package/repo for each tool.

Shipping emhttp is again, just a matter of packaging with the correct tool.

There could be a repo for unRAID and the several distros own tools (yum, apt).

If you *want*, integrating it into the procedure to build a custom install, like with an ISO can be done for a distro later.

 

With PVE, I think running the unRAID packages and repo alone would be best and sufficient....no need for a complete distro as the instructions for install would be dead simple.

 

As for the time I can spend during the holidays, I am not sure...lots of family business  ;D

I would like to help though...what do you want me to do?

Link to comment

..I am not a lawyer

 

Neither am I. It would be their WebGUI for managing KVM or adding unRAID capability into Proxmox itself.

 

If I was Tom, I would seriously look into that.

 

Is there really the need for a choice of the distro?

 

Maybe there is a more generic way to think of.

 

Integrating the md-module in an automated fashion would be best but also the first challenge...I'd opt for using DKMS.

Once this is done, it should be able to be integrated into any distro, once you have a package/repo for each tool.

Shipping emhttp is again, just a matter of packaging with the correct tool.

There could be a repo for unRAID and the several distros own tools (yum, apt).

If you *want*, integrating it into the procedure to build a custom install, like with an ISO can be done for a distro later.

 

Once I took a look under the hood of unRAID. That was the first thought I had too. If I was Tom, I would have done this years ago.

 

Have a repo for CentOS, Fedora, openSUSE, Debian, Arch, etc. that installs unRAID and emhttp onto any number of the Linux Distros and it makes updates a lot easier too. There is a reason why FreeNAS has over 5.5 Million downloads. I suspect it has to do with innovation, support, updates, upgrades, beta program, etc. and thousands of "open source" developers who contribute to it because FreeNAS makes it easy for them to do so.

 

Perhaps Ironic will do exactly that. If it were me, I wouldn't invest all the time / effort / money to house / support that unless Tom compensated me for it.

 

As for the time I can spend during the holidays, I am not sure...lots of family business  ;D

I would like to help though...what do you want me to do?

 

With your knowledge, experience, ideas and suggestions... Between you and Ironic, I'm sure you two would come up with the best way / method for moving unRAID forward.

Link to comment

By that logic you would still run XP, or internet explorer 6.

 

This thread is about trying to make progress, if that's not your game then that's cool.

 

Sent from my Nexus 5 using Tapatalk

hehe, I do still run an XP VM on the ESXi box, though it is not running IE6.  The XP VM is not used for a whole lot since I set up the Windows 7 VM but it is still there and gets used to test things once a week or so for my job.

 

 

There is nothing wrong with progress and I applaud you for your efforts in this thread.  I will be following it with interest and will be awaiting what comes.  I think for my test bed system this could work... but we shall see.

Link to comment

Once I took a look under the hood of unRAID. That was the first thought I had too. If I was Tom, I would have done this years ago.

When I looked into that some years back in order to get it to work with PVE or XEN (these days the disk nodes naming conventions were unusual, like /dev/vda or /dev/xda and I had to patch emhttp)

I didn't look further as I found the integration with emhttp the biggest hurdle and shortly after that my requirements for full-disk-encryption arose and I droped unRAID completely.

What I found missing was a documented API on how emhttp and md-module would interact....I had - and still have - no urge to reverse engineer that interface.

 

Also this is Tom's product and there is a difference (I am *not* saying a strategic decision) between being compliant with open source or running an open source project as a business.

 

Perhaps Ironic will do exactly that. If it were me, I wouldn't invest all the time / effort / money to house / support that unless Tom compensated me for it.

+1

It is fine to get to a certain stage because you feel that urge to proof that it can be done....maybe just call it an exercise.  :)

But offering it on a regular basis, I would not do until Tom announced this as a community project supported by the company not only with loyalty/blessing but with commitments.

I think unRAID with its md-module is a unique product filling a specific gap in the market.

Maybe a change for the "Nas4Free or PVE Model" is an option worth considering today....times have changed with nthe introduction of the paypal button, haven't they?

 

With your knowledge, experience, ideas and suggestions... Between you and Ironic, I'm sure you two would come up with the best way / method for moving unRAID forward.

Thank you for the kind words, sir...really appreciated.

However, thinking for moving unRAID forward is one thing, making decisions for the future of unRAID is not ours, I am afraid.

Link to comment

Thank you for the kind words, sir...really appreciated.

However, thinking for moving unRAID forward is one thing, making decisions for the future of unRAID is not ours, I am afraid.

 

I'm liking you more and more.

 

Will someone get Toms attention please and get him in on this? I've sent him a PM, what else next...

 

Your comments about donations etc are true, they would be welcomed of course (being as I'm a poor student!) But before much further progress is made I reckon it'd be sailing a bit close to wind without Tom on board.

 

 

 

Sent from my Nexus 5 using Tapatalk

 

Link to comment

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.

Guest
Reply to this topic...

×   Pasted as rich text.   Restore formatting

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.