64 Bit unRAID running natively on Arch Linux with full hypervisor support



Recommended Posts

Ladies and Gentlemen on the unRAID forums I have some rather exciting news to announce...

 

unRAID has been converted to run in 64 bit, natively on Arch Linux, with full type 1 hypervisor support for KVM and Xen.

 

archey.png

 

Right then, OK. So this was not an easy task and for that reason I will not be releasing a howto guide, instead the aim is make this a really user friendly project by releasing an entire 'customised' distribution with all the hard work already done. This will likely take me a few weeks, maybe longer, to accomplish but I'll making test releases available as soon as is sensible to do so. Huge, huge thanks to my grumpy friend for helping make this a reality. There are many, many reasons why this is a huge step forward for the unRAID product so let me list a few for you now...

 

  • Speed - I am achieving roughly a 30% increase on the same hardware over stock unRAID speeds. With these 2 old drives I conducted my tests with I acheived roughly 85Mb/s parity speeds, but as you can see below that has increased to almost 110Mb/s! Not bad, but shows how limiting the 32 bit version of unRAID and it's old kernel really is.
    parity_check.png
     
     
     
  • Latest Kernels and Patches - unRAID is quite a long way behind kernel wise, and you're missing out on security patches, speed increases and bug fixes as a result. "ArchRAID" or whatever you want to call is running 3.12-1 a bang up to date version. You're not reliant on one developer deciding whether or not to include a certain patch, kernel flag or software. As it's Linux (any old distro will do) you're free to install any software you want, free'd from the shackles of a USB thumb (if you want to be). It's worth noting my test system is entirely running off two flash drives. I'm not trying to bash on Tom here, but there's only so far a one man development model can take you, and with the lack of progress on the 64 bit version I've seen publicly to date I wanted to see just how hard it was. Turns out, not very. emhttp is still 32-bit granted but that's not open sourced so I can't modify it (besides that's what you pay for with your license).
     
  • Flash Drive - Yes, you still need to pay for unRAID. I'm not out here to let people steal Tom's work I'm merely repackaging his open sourced code and emhttp on top of a 64bit OS, something he's not done despite user requests for many, many years. I do not condone piracy of this software, nor should any requests be made to me to do so. The flash GUID is passed through automatically by the OS naturally, as there's no emulation layer in the way like with a VM.
    flashGUID.png
     
     
  • Native Hypervisor support - This is where my release will blow any other NAS product out of the water. Type 1 hypervisors are natively support which means KVM and Xen. unRAID itself becomes the host OS so everything else runs natively on top of that, if you understand the power of this single fact you'll understand why I'm so excited to share it with you all. On top of that, unRAID storage is immediately available to the OS (no need to wait for a VM to load up etc) and thus can be used to run your VMs etc (although performance may not be fantastic that way). Samba, NFS etc all work the way you'd expect.
     
  • Software / Plugins - It's Linux, full Linux. You can install whatever the hell you want. If you want a desktop, install it. If you want XBMC fullscreen with TBs of storage on the SAME OS, install it. If you want to remain headless, IT'S YOUR CHOICE. On top of all this my distro package will allow users to customise their installs at download time using something similar to OpenSUSE studio so you can choose at download time what you want - that said, it's Linux so you can change whatever you want after the fact. It's all manageable via bit of software such as webmin, pictured below.
    webmin.png

 

 

Future Plans / Roadmap

 

As my Uni work winds down over the Xmas period you can expect a flurry of updates, except for the weeks I'm at my parents house! The current state of the project is not ready for a public release, many paths are broken and some things don't quite work in the most user friendly of fashions. My aim is to make a project which is suitable for Linux newbies, I'll document the whole install process and make available for download a custom .ISO file via .torrent. I aim to have something ready by the very early parts of the New Year. Don't hold me to that other priorities come first and this isn't going to help me pass my exams!!

 

I cannot wait to see what you guys make of this and hope to generate a massive amount of interest and hopefully involve Tom in this. It's aimed at making his product better anyway!

Link to comment
  • Replies 451
  • Created
  • Last Reply

Top Posters In This Topic

I am not very liunx savy as I am strictly a windows user.  The most I have really done with linux is install kubuntu and played around with it a few days.  I like the idea of being able to run software on a full disro so I can run full software  and not use a stripped down/altered plugin that will work with unraid.

 

How easy will this be able to install an use for the linux illiterate?  Will there be other distros avalible?  Is unraid still managed using the web interface and if so could I still use the simple features plugin?

 

Like mentioned above I would run xbmc with library sql server, with a tv tuner program that way I could get my tuner cards out of my htpc and into my server.  Right now I have to keep my htpc server on for tv recording this way I could turn it off.

 

Sent from my SCH-I545 using Tapatalk

 

Link to comment

ironicbadger, first of all, i think this is awesome work !!

 

now, playing the part of the devi's advocate:

 

my current use case is to run two virtualized unraids.

 

i don't think there should be a problem with having a host unraid and then a guest unraid ... but then this begs the question ... why do i need unraid as a host ?

 

for single unraid installs this could probably become the way to go, but i'd like to think of unraid in terms of an appliance

 

in my currenty esxi setup, i have

- a virtualized freebsd appliance on usenet downloading duty

- a virtualized ubuntu appliance on postgresql, etc duty

- a virtualizes solaris appliance on zfs encrypted store duty

 

and then i have two virtualized unraids on data vault duty

 

i really like the idea of having a 64bit archx as the host, and then adding other appliances as needed.

 

i might be missing some benefit with unraid as host (some passthrough goodness perhaps?).

Link to comment

I am not very liunx savy as I am strictly a windows user.  The most I have really done with linux is install kubuntu and played around with it a few days.  I like the idea of being able to run software on a full disro so I can run full software  and not use a stripped down/altered plugin that will work with unraid.

 

How easy will this be able to install an use for the linux illiterate?  Will there be other distros avalible?  Is unraid still managed using the web interface and if so could I still use the simple features plugin?

 

Sent from my SCH-I545 using Tapatalk

 

Yep. The first post has the traditional webGUI screenshot in there for a reason! It all behaves the same so far as unraid goes.

 

It becomes as difficult or complex as you make it after that!!

 

Sent from my Nexus 5 using Tapatalk

 

 

Link to comment

I am not very liunx savy as I am strictly a windows user.  The most I have really done with linux is install kubuntu and played around with it a few days.  I like the idea of being able to run software on a full disro so I can run full software  and not use a stripped down/altered plugin that will work with unraid.

 

How easy will this be able to install an use for the linux illiterate?  Will there be other distros avalible?  Is unraid still managed using the web interface and if so could I still use the simple features plugin?

 

Sent from my SCH-I545 using Tapatalk

 

Yep. The first post has the traditional webGUI screenshot in there for a reason! It all behaves the same so far as unraid goes.

 

It becomes as difficult or complex as you make it after that!!

 

Sent from my Nexus 5 using Tapatalk

 

I must have missed that screen shot. I'm on my phone.

 

Sent from my SCH-I545 using Tapatalk

 

 

Link to comment

ironicbadger, first of all, i think this is awesome work !!

 

now, playing the part of the devi's advocate:

 

my current use case is to run two virtualized unraids.

 

i don't think there should be a problem with having a host unraid and then a guest unraid ... but then this begs the question ... why do i need unraid as a host ?

 

for single unraid installs this could probably become the way to go, but i'd like to think of unraid in terms of an appliance

 

in my currenty esxi setup, i have

- a virtualized freebsd appliance on usenet downloading duty

- a virtualized ubuntu appliance on postgresql, etc duty

- a virtualizes solaris appliance on zfs encrypted store duty

 

and then i have two virtualized unraids on data vault duty

 

i really like the idea of having a 64bit archx as the host, and then adding other appliances as needed.

 

i might be missing some benefit with unraid as host (some passthrough goodness perhaps?).

 

You're probably not my target user, but then again maybe you are exactly my target user!! I'm aiming to bake in the unraid functionality directly into the OS to prevent some users even needing to go down the VM route.

 

Sure arch as the host is more flexible than say, xenserver or esxi because you're not wasting the video out of the CPU / mobo as you could easily use it for XBMC duties (as I do) or install gnome / KDE or whatever and use it as your workstation desktop.

 

I think that unraid in its current form does a very good job as just a storage appliance type product. With the power available in new PCs it seems daft to waste all those spare clock cycles and increase complexity with specialised plugins when Linux distros already do that work for us.

 

Sent from my Nexus 5 using Tapatalk

 

Link to comment

I have been running unRAID 64bit on various Distros for over a month. I have thrown everything I can at it.

 

Countless parity checks

Drive Failures (unplugging the drive)

Installed some Drives that are faulty

Rebuilding the raid numerous times after drive replacement

Adding / Removing Drives

Using the mover function to move TBs of data (which would not be needed since you aren't doing plugins)

Running 50 or so of the most popular Linux Apps

I have loaded 10+ VMs

Crashed VMs

Running it on a HDD and Flash Drive

Having several HDs that are LVMs for the VMs outside of unRAID.

Etc.

 

Everything worked as it should. When I crashed a VMs it didn't crash the Linux Distro (unRAID)... Just the VM.

 

Hardware Support

 

I added over 2,000+ kernel modules. To put that in comparison unRAID only has 170 or so.

 

All your Sata, SCSI, Network, Video, Sound, IR, TV Tuners, Fiber, Block Devices, Input, Hardware Monitoring, USB, etc. will work out of the box. If it works in Linux (Ubuntu, Fedora, Arch, CentOS, etc.) it will work here too.

 

Additional Network Transports

 

NFS4

Samba4

iSCSI

AoE

VirtFS

 

Security

 

If your unRAID is exposed to the internet, most of the packages installed on it are 3+ years old with TONS of security flaws. PHP, Samba, NFS, SSH, etc. The Plugin guys do a good job of keeping the packages they need (like python, python-cheetah, mysql, etc.) up to date but those still run on a very unRAID which hasn't had many updates in 3+ years.

 

For those of who you have nVidia Cards that want to pass them through to XBMCs throughout your house

 

You can passthrough NVIDIA GEFORCE 7, 8, 4xx, 5xx, 6xx. On ESXi or Xen you cannot do this unless you have an Nvidia Quattro, Tesla or GRID series or upload firmware and make your 5xx or 6xx thinks it one. ESxi is probably a year or two away. Xen cannot do this at the moment unless you patch patch it but with the next release 4.4 (which starts RC anyday) you will be able to.

 

Significantly increase the speeds of your transfers rates / file transfers between Hosts and VMs?

 

Virtfs is a virtualization aware filesystem passthrough.. Sharing host files on the guest through generic network file systems like NFS and CIFS suffer from major performance and feature deficiencies as these protocols are not designed or optimized for virtualization. To address the needs of the virtualization paradigm... Virtfs, a new paravirtualized file system with uses the 9p2000.L protocol.

 

Your host (unRAID) will share it's files to your VMs using VirtFS (A Paravirtualized File Transport).

 

VirtFS vs NFS and Samba

 

1m1bYJ0.jpg

Link to comment

How is the configuration stored, is it stored on the unraid flash drive still?

 

Yes

 

If I really goon up the Linux install can I just unplug the Linux OS drive and boot from the unraid flash and continue on like nothing happened?

 

Yes

 

How are unraid updates handled?

 

Through the Package Manager. There would be unRAID "repo" which would handle upgrades.

 

For example, if the Distro is CentOS:

 

yum update

 

The Linux Kernel rarely changes, it's when Tom makes updates or changes to Slackware (what unRAID runs on). Depending on what updates Tom makes, an update may not be required. For example, the 5.0.4 didn't do anything that isn't already done with the latest packages. For example, Tom is running Samba 3.6.21 where as most Distros are well into version 4.0.0. Red Hat even runs a later version of Samba than unRAID (due to security holes and patches needed for it to be safe / stable).

Link to comment

I will be following this and it may be something that I install on my test machine.  I will likely not be moving my ESXi box over to this though... if it aint broke don't fix it.

 

Good luck.

 

From my point of view, this would give us ESXi users an option from VMware.  While my system is very stable and does exactly what I expect of it now, there is no guarantee that it will remain that way.  When Win8 came out, ESXi required a patch release to support it.  I want the option to go in another direction, should VMware decide us free users are just not worth the trouble (if they haven't already done so with 5.5).

 

Will I be moving my current setup to this in the foreseeable future?  Probably not.

 

Will I be trying it out on a testbed?  Absolutely.

Link to comment

This is all so very exciting!

 

I take it you dont feel like writing a how-to guide on this.  But how about at least writing up the changes in the kernel config?  That would be immensely useful for future reference.  Also, maybe attach the actual .config file from that compile?

 

I wonder, did you have to make any changes in the unraid driver sources so they could compile in the 3.12.x kernel?  Or did they compile just the way they are?  (I presume you used the sources that are in 5.0.4, right?)

 

I can't wait to test what you have done there!  Please let me know if I can help.

 

 

Link to comment

From my point of view, this would give us ESXi users an option from VMware.  While my system is very stable and does exactly what I expect of it now, there is no guarantee that it will remain that way.  When Win8 came out, ESXi required a patch release to support it.  I want the option to go in another direction, should VMware decide us free users are just not worth the trouble (if they haven't already done so with 5.5).

I am a IT Consultant and my clients are Fortune 500 companies and one of my verticals is ESXi (Hyper-V and XenServer and some KVM) in very large multinational deployments. I'm what you would consider a "fanboy" of ESXi not to mention it's how I also earn a living.

 

The VPs and the various Client VMware Sales Reps I have spoken too (plus the ones at the VMWorld Convention)... think us home users are a joke. From now on ESXi will be crippleware and it will only get worse. Don't be surprised in the next version if it isn't a hardcore 60 license that will not function in anyway without a license. Yes, they are aware that XenServer and Hyper-V are free and not "crippled" but they don't care. They hardly care about the small business market much less home users / test labs. ESXi 5.5 "crippled" without a license after 60 Days was a "shot across the bow". It's all downhill from here if you don't purchase a realitiy inexpensive license.

 

Having said all of that...

 

ESXi 5.0 / 5.1 or "crippled" 5.5 works like a champ as long as they do updates for things like they did Windows 8.2 (when it comes out). Most users do not need the new functions of 5.5. ESXi 5.0, 5.1, 5.5 fixed passthrough for some and broke passthrough for others. If needed, upgrade / downgrade and have a good / solid system.

 

Will I be moving my current setup to this in the foreseeable future?  Probably not.

 

Will I be trying it out on a testbed?  Absolutely.

 

You know it's 100% Linux Kernel, 100%unRAID kernel module and 100% unRAID emhttp, right?

 

Tom's unRAID code hasn't changed in forever. It's only the versions of Slackware Linux Packages that he has changed or made changes to emhttp. Like I said earlier, his version of Samba or PHP or NFS wouldn't be allowed in any Enterprise Environment I work in. Red Hat who has 1,000+ developers and millions of customers runs a later version of Samba. Do you think they are on top of managing Linux, RAID, where they have clients running Samba on thousands of different hardware platforms, with exabytes+ of data and using it in thousand different ways is more knowledgeable of which version of Samba is best or 1 man developer?

 

His unRAID code has NOTHING to do with NFS, Samba, SSH, Mysql, Python, PHP, telnet, the Linux Kernel, etc. and all the other Linux Packages Slackware 13.1 released in 2010.

 

In summary...

 

100% Linux Kernel, 100% unRAID Kernel Module, 100% unRAID emhttp with updated / stable Linux packages underneath that isn't in a read only compressed ramfs (even though Ironics takes up less than 200mb of memory).

 

You can install say... MySQL ONCE. Instead of using a plugin to download MySQL from the internet, unzip it, install it (all using your ram), read a config file from the flash drive, etc. (and hope like hell it doesn't break other plugins when one plugin guy uses one version of a package and a different plugin guy uses a different one). You can install owncloud instead of waiting YEARS for someone to write a plugin and maintain it (still doesn't exist). I dunno... To me it's easier to type in "yum -install owncloud" once and not have to worry about it again instead of waiting on someone to write a plugin.

 

Online Back Up is important to me, Crashplan doesn't require all the BS I have to go through to make it work on unRAID via plugin. Plus, I am not stuck with Dropbox or Crashplan either. I have plenty of choices besides those two plugins due to the fact I am running unRAID on a non-stripped down version of Slackware.

 

Enterprises who spend millions of dollars on IT to run their 10s of millions of dollars of ERP software so they can continue to makes billions of dollars wouldn't dream of running their Storage in a VM (much less on an outdated / unpatched  Linux Kernel and Linux Packages.

 

Why is running unRAID using passthrough via a VM considered stable / safe but it's not when it's the Host? Whoever made that rule doesn't know what unRAID (RAID) is, how it works, clueless about Linux and Software / Hardware Lifecycle Management.

 

People have been asking for a 64Bit version of unRAID for 3+ years and still we aren't there yet. Sure, Tom is a one man show and due to life / life circumstances, he goes through "spurts" and long "droughts" between development and innovation. However, I got tired of waiting and did it in 10 minutes (THREE Linux Kernel settings and updated a couple of packages). Imagine what other innovations other users will bring to the table. Tom will find himself with many "open source" developers like the plugin writers, Ironic, Simplyfeatures guy, etc. who freely volunteer their time, energy and effort to make unRAID an even better product than it already is. Not to mention, Tom is will continue to reap the financial rewards. A win / win for Tom.

 

Wait till some of the web development guys write a module for Webmin:

Jqomr2L.jpg

or WebYast:

 

AO5M32a.png

 

Link to comment

Looks great and can't wait to see where this goes, what advantage would this have over the xenserver route I was going to take in the new year.

 

Bear in mind my needs are very simple I was planning to Install xenserver and have 1x unRAID VM and then have 1 VM's to handle the plugins, ie serviio, torrents, owncloud and maybe another to run SQL and any other stuff I screwing around with / learning.

 

I think I would prefer the xenserver route as it keeps everything separate, don't like the idea of installing the above plugins/software on the unraid/arch host just in case I screw something up and can't access my data. At least if its in a VM its only a matter of delete VM and start again, knowing all the time that my unRAID VM is safe and running as Its never going to really get touched once its up and running.

 

Will you guys still be helping us guys that want to go the xenserver route or will this be your primary focus now? Will you still complie and provide the files needed to get unRAID running on xenserver?

 

Maybe I don't understand this enough or maybe I'm not your target audience, either way looking forward to reply and will follow  with interest.

Link to comment

My vote(s) for other OS

 

- Proxmox-VE

I know it is Debian based but as a hypervior host it comes with the best/easiest way of managing VMs since it is truly Web-based.

IMHO this is what we should keep with unRAID, because that's what makes it a server OS and what makes it so simple.

Edit: I know that PVE is using 2.6.x Redhat kernel but the 3.10 kernel has just hit the test-repos @ PVE.

 

- Fedora

AFAIU CentOS is following RedHat lifecycle, whilst Fedora is always upstream.

The advantage of enabling the 64bit nRAID host as a hypervisor is better supported by Fedora as we would get upstream versions of kvm/qemu.

 

...this concludes my 2 cents.

Keep up the good work!

Link to comment

Looks great and can't wait to see where this goes, what advantage would this have over the xenserver route I was going to take in the new year.

 

Bear in mind my needs are very simple I was planning to Install xenserver and have 1x unRAID VM and then have 1 VM's to handle the plugins, ie serviio, torrents, owncloud and maybe another to run SQL and any other stuff I screwing around with / learning.

 

I think I would prefer the xenserver route as it keeps everything separate, don't like the idea of installing the above plugins/software on the unraid/arch host just in case I screw something up and can't access my data. At least if its in a VM its only a matter of delete VM and start again, knowing all the time that my unRAID VM is safe and running as Its never going to really get touched once its up and running.

 

Will you guys still be helping us guys that want to go the xenserver route or will this be your primary focus now? Will you still complie and provide the files needed to get unRAID running on xenserver?

 

Maybe I don't understand this enough or maybe I'm not your target audience, either way looking forward to reply and will follow  with interest.

 

Your data is just as safe with unraid as the host, if not safer, than with it in a VM. Its a sea change in attitude on these forums I suspect that will be required to achieve mainstream adoption. Once you see it working, you'll get it.

 

As I'll include kernel support for hypervisors you can still run all your "rogue" software safely there with absolutely no impact to your unraid stuff. If you didn't want to, don't install anything on the host and just use a VM purely for "plugins".

 

To all asking the main reason for this, in my eyes there are several (listed in the OP) but a 30% speed increase, for free, has got to be a great one. Add that to the thoughts of grumpy regarding esxi longterm support and the progress of unraids traditional package upgrade path and its a no brainer really.

 

Sent from my Nexus 5 using Tapatalk

 

 

Link to comment

Excellent work, I cannot wait to convert to have full control of plugins and not rely on the good people on here to rewrite every time there is an update. This really expands the possibilities of unRAID and in the end increase user base and ultimately its sales.

 

IMO, you are 100% right on wanting to take the basic functionality of unRAID and give people the option who have better hardware to expand the functionality to a home server. This is exactly the reason why I went with unRAID in the first place. I wanted a server that had the redundancy to protect my data plus the ability to install plex server. I almost bought a synology/readynas, but instead bought two pro licenses and made my own. Then I managed to install transmission/CP/SB/Headphones/Dropbox/APC and couldn't be happier with my purchase.

 

I do hope this remains amicable with Tom though and that you can actually work together to get the best out of each other.

 

Good luck and look forward to a brighter outlook for unRAID (and moving with the times)

 

 

Link to comment

I do hope this remains amicable with Tom though and that you can actually work together to get the best out of each other.

 

This would indeed the best for the community!

 

That's the plan guys. I hope to work with him directly now he can see my envisaged future more clearly. I've sent Tom a PM about this thread, I'm sure he'll read it and get in touch...

 

I'm not against Tom, sure I do point out a few of his flaws now and again! but the security issues touched on by grumpy and other stuff can't be denied. The aim here is to use the collective developers of Linux rather than just rely on Tom - it makes it easier for him in the long run if you ask me. All we need to keep pressing him for is a 64 bit emhttp as that would speed things up EVEN MORE.

 

Link to comment

Your data is just as safe with unraid as the host, if not safer, than with it in a VM. Its a sea change in attitude on these forums I suspect that will be required to achieve mainstream adoption. Once you see it working, you'll get it.

 

As I'll include kernel support for hypervisors you can still run all your "rogue" software safely there with absolutely no impact to your unraid stuff. If you didn't want to, don't install anything on the host and just use a VM purely for "plugins".

 

To all asking the main reason for this, in my eyes there are several (listed in the OP) but a 30% speed increase, for free, has got to be a great one. Add that to the thoughts of grumpy regarding esxi longterm support and the progress of unraids traditional package upgrade path and its a no brainer really.

 

Sent from my Nexus 5 using Tapatalk

 

Thanks ironicbadger for the explanation,

 

I spouse this is comparable to how you use flexraid, have a base system and then install flexraid, but this would have a base and all the unRaid goodness already built into it.

 

Guess I'm just a little perplexed as to why you would need a desktop on a server, my server sits in a corner and is forgotten about, no monitor or keyboard attached, I like the fact I just open the browser and point it to the server and your in, If its got a full blown desktop wouldn't i need to use vnc or something, seems a waste of resources for something i not going to look at that often.

 

That was one of the things that got me excited about xenserver and VM's, but your right in the fact that it requires a different thought process, I'm sure ill get it once its released, I don't know lol.

 

Only linux distro's I've played with are ubuntu/xubuntu and a little with ubuntu server when i was playing with flaxraid many moons ago, maybe its time to have a look at arch linux and learn a little, time permitting.

 

I do like the idea of keeping everything upto date and the speed increase would be welcomed also.

Link to comment

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.

Guest
Reply to this topic...

×   Pasted as rich text.   Restore formatting

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.