64 Bit unRAID running natively on Arch Linux with full hypervisor support

[Ford Prefect]

I'm back from my Xmas travels and have an exam on 6th but the after that its full steam ahead once I've begged and borrowed the components for a dev machine.

 

Waiting to hear back from Tom on things, loosing a bit of hope as its been nearly 2 weeks since he last made contact. I'm all for collaboration but it takes two to tango.

 

So, in light of this I will be pushing ahead with a fork of unraid myself much in the same way as other Linux distros have forked. Mine will be a two pronged approach...

 

First, get unraid-fork packaged for an alpha release. I approximate by mid to late Jan on this.

 

Second, tidy up any loose ends and reach a state where this can be 'given back' to the community entirely open sourced with some documentation by the end of Feb.

 

Third, take what I've learned and push forward with a replacement modernised version of emhttp that I will write. It is still very early days on point 3 mind you...

 

So with this all in mind, and leveraging this awesome community and what was discussed earlier in this thread, I feel comfortable accepting donations. I realise this model isn't for everyone but I would really appreciate any contributions towards the development costs as I'm a full time student right now without a job.

 

If you'd like to donate please do so via PayPal. My address is

 

[email protected]

 

Thanks for reading and for donating from the bottom of my heart if you do.

 

Sent from my Nexus 5 using Tapatalk

 

Yes, IMHO this is the right move.

 

Based on a quote from TOM here

Right, I really apologize for this guys.  Here's what I'm aiming for:

 

1/ Publish 5.0.5 - this has a couple minor fixes

2a/ Publish 5.1-beta1 - this is 5.0.5 with latest 'webGui' integrated.

2b/ Push this 'webGui' onto github as 'webGui-master'.

3/ Publish 6.1-beta1 - this is 64-bit unRaid with numerous updates including 'webGui-master' and virtualization support.

 

Idea is webGui will run on both platforms.

 

...what TOM is aiming for is not unRAID-NG as laid-out/conceived in this thread.

 

Going to donate now...

[grumpybutfun]

As Fx stated before, unRAID is something more than raid 5. Their jbod application is pretty useful when adding new drives. HOWEVER, if I could get this with other distro/solution, I'd have a go at it. The thing is that I haven't found it and so far unRAID is the NAS solution that better fits my needs.

 

Have you seen / managed Raid / LVM / [insert File System of choice] on Linux with GUIs / WebGUIs?

 

I am going to go out on a strong limb and bet that if you loaded Fedora, openSUSE, CentOS, etc. that you will be blown away if you are comparing it to emhttp.

 

1. Hot Spares (or cold) that automatically are added when a drive has errors.

 

2. More than Single Parity.

 

3. Notifications (text, email, on your server screen) - Again TONS of choices and Developers dedicated to this specific feature / ability.

 

4. Reporting / Monitoring / States -  Again TONS of choices and Developers dedicated to this specific feature / ability.  See Monitorix on Arch as one example but I show you many more.

 

5. Can easily add / remove drives of any shape or size.

 

6. Shrink / expand the file system / lvm.

 

7. Snapshots

 

8. CoW

 

9. Compression

 

10. Encryption.

 

11. If you tell RAID you want to remove a drive, it will automatically move all the data from it to the other drives for you (takes time of course and you space or might have to add another drive, etc.)

 

Etc... All done through GUIs on a Desktop or you could use one WebGUI like webmin, C-Panel, Plesk, Ajenti, Kloxo, ZPanel, etc.

 

How do you think all the VPS, Dedicated Servers that sit in data centers that are 1,000+ times more complex and have 1,000+ times the storage we do are managed all remotely?

 

Bottom line, I'm not trying to accommodate my set-up to unraid, but rather unRAID to my set-up, because it's the best media storage solution I could find.

 

Makes Sense.

 

However, I think there is a misconception on here that Linux can't do RAID 5/6 (It can and does it very well) and that one can't add drives or remove them (It's actually easier than doing it in unRAID).

[dheg]

However, I think there is a misconception on here that Linux can't do RAID 5/6 (It can and does it very well) and that one can't add drives or remove them (It's actually easier than doing it in unRAID).

I'm certainly nowhere close to be a linux guru  , but, can you set up a not-stripped raid?

 

IMO, this has 2 advantages:

[*]I do not have to spin up the whole array to open a small word file (for example)

[*] if parity+1 drives fail, I only lose the information on those drives, and not the whole array. This might seem unlikely, but it happened to me one month into unraid, with brand new drives, that have been precleared. What are the odds!

[grumpybutfun]

I'm certainly nowhere close to be a linux guru  , but, can you set up a not-stripped raid?

 
You can do whatever you want.
 
Set up a mirror for OS partitions / drives / both.
 
Have a "cache" drive where you movies / tv shows download to and copy to the your movies / tv show array at X time (what unRAID does).
 
Set up a RAID 5/6 for your data files.
 
Set up another RAID 5/6 for your Movies / TV Shows
 
Set up another RAID 0/1/5/6/10 for X, Y and Z.
 
It's all point and click. I want these drives in this array. These Drives in that array. I want this Hot Spare for that RAID and this one for the other one. Etc.
 

IMO, this has 2 advantages:
I do not have to spin up the whole array to open a small word file (for example)

 
1. Hard Drives consume about 5 - 10 watts
 
2. Spinning Down the drive does not extend the drives lifespan. There have been countless studies / research on large scales to prove otherwise.
 
3. You can make your array drives all sleep. It's when you access the array that they spin up. I separate mine RAID into smallers ones so I don't have all my drives spinning up at once.
 

if parity+1 drives fail, I only lose the information on those drives, and not the whole array. This might seem unlikely, but it happened to me one month into unraid, with brand new drives, that have been precleared. What are the odds!

 

1. You would have to lose 3 drives at once to loss data if running RAID 6.

 

2. It would have been nice to have a Hot Spare automatically remove the bad drive, add itself, notify you via text / email and "rebalance" the RAID wouldn't it?

 

3. Single Parity has major downsides as you now know.

[mattkhan]

How do you think all the VPS, Dedicated Servers that sit in data centers that are 1,000+ times more complex and have 1,000+ times the storage we do are managed all remotely?

By teams of people whose job it is to administer and operate them.

 

The fact that there are enterprise class tools available to the home user does not necessarily make that solution suitable to that home user.

 

The point of unraid is surely to offer a turn key solution. I don't see that in the Linux offerings (I don't really see it in unraid either btw but still).

[grumpybutfun]

By teams of people whose job it is to administer and operate them.

 

That simply isn't true.

 

I have several clients that have close to a petabyte of data and 1,000s of drives that are co-located in Data Centers all of which THEY manage remotely and the Data Center doesn't even have Access to their server racks.

 

If you can't use the built-in GUIs (or install the WebGUIs) in CentOS, Fedora, Red Hat, Ubuntu, etc... to manage RAID / LVM / etc. then I would argue that you have never even clicked on the various Linux Apps or used something like Webmin or ZPanel.

 

 

 

 

 

 

 

The fact that there are enterprise class tools available to the home user does not necessarily make that solution suitable to that home user.

 

I am going to bet there are many users who will look at the screenshots above and not find them at all complicated or confusing at all. In fact... I suspect many will find them better and more user friendly than emhttp.

 

The point of unraid is surely to offer a turn key solution. I don't see that in the Linux offerings (I don't really see it in unraid either btw but still).

 

I don't think you have actually loaded openSUSE, Fedora, CentOS, Ubuntu, etc. and tried installing / using RAID or LVMs. Otherwise, you would find that you can do everything I said throughout this entire thread already build in / installed by default. The things that are missing like the WebGUIs and Sickbeard, Owncloud, etc. are easy to install via the package manager.

 

Also, you could setup all my RAID Arrays, LVMs, LVs and run in it CentOS (or any Modern Linux Distro using a GUI / WebGUI)... Delete it and load a totally different Linux Distro. When I boot into the new Linux Distro, it will see all my RAID Arrays, LVMs, LVs, etc. without me doing a single thing. All I have to do is tell them to start on boot, mount them if I need too and set up the shares again using an easy GUI / WebGUI.

 

 

[ironicbadger]

Well I think they're all far too complicated grumpy. How the heck do you expect me to cope with a graph? 

[mattkhan]

By teams of people whose job it is to administer and operate them.

 

That simply isn't true.

 

I have several clients that have close to a petabyte of data and 1,000s of drives that are co-located in Data Centers all of which THEY manage remotely and the Data Center doesn't even have Access to their server racks.

perhaps we're talking at cross purposes but I find it hard to believe that any serious server infrastructure doesn't have a sufficient supply of SAs to manage it. This is beside the point though really which is that having the tools and knowing what to do with them are 2 different things. The point of something like unraid should be to provide a solution that just works with as close to zero user config as possible. A well constructed/architected solution would be composed of standard libraries under the covers, present a simple appliance like experience to the user while giving power users the ability to tweak/tune to their hearts content. A poorly constructed one will use a variety of non standard or legacy tools and/or be difficult to extend/maintain.

 

Also, you could setup all my RAID Arrays, LVMs, LVs and run in it CentOS (or any Modern Linux Distro using a GUI / WebGUI)... Delete it and load a totally different Linux Distro. When I boot into the new Linux Distro, it will see all my RAID Arrays, LVMs, LVs, etc. without me doing a single thing. All I have to do is tell them to start on boot, mount them if I need too and set up the shares again using an easy GUI / WebGUI.

another tangent but I'll ask anyway, if you have this then why do you use unraid?

[peter_sm]

Hi grumpy!

 

I have made more progress to update current 5.0.4 with support of KVM/QEMU

 

It was a lot of packages (40) to find to get this up and running 

I have som VM running, unraid and some linux distro.

 

 

https://dl.dropboxusercontent.com/u/46562247/KVM_2.png

 

But this is another ball game than you are doing, you do a amazing  stuff  but this is maybe something that we can look forward to when 64bit is released!

 

Keep up the good work you are doing,  it's take unraid to a higher level, hope Tom can look into this.

 

//Peter

[ironicbadger]

Hi grumpy!

 

I have made more progress to update current 5.0.4 with support of KVM/QEMU

 

It was a lot of packages (40) to find to get this up and running 

I have som VM running, unraid and some linux distro.

 

 

https://dl.dropboxusercontent.com/u/46562247/KVM_2.png

 

But this is another ball game than you are doing, you do a amazing  stuff  but this is maybe something that we can look forward to when 64bit is released!

 

Keep up the good work you are doing,  it's take unraid to a higher level, hope Tom can look into this.

 

//Peter

 

Awesome stuff. The possibilities here are absolutely endless... LOVE IT! 

[grumpybutfun]

Hi grumpy!

 

I have made more progress to update current 5.0.4 with support of KVM/QEMU.

 

You sure have.

 

Since you like running the QEMU commands (which I also do in KVM) you should take a look at the following:

 

 

 

[peter_sm]

Hi grumpy!

 

I have made more progress to update current 5.0.4 with support of KVM/QEMU.

 

You sure have.

 

Since you like running the QEMU commands (which I also do in KVM) you should take a look at the following:

 

 

That is something I like to have and try! but that's not possible on the system I running, looking forward to see what you come up with 

 

//Peter

[garycase]

Hi grumpy!

 

I have made more progress to update current 5.0.4 with support of KVM/QEMU.

 

You sure have.

 

Since you like running the QEMU commands (which I also do in KVM) you should take a look at the following:

 

 

Just as you should attribute quotes of other's comments, you should provide proper attribution to graphics when they're from another source.    For example, you should have included the link to sourceforce and noted that the graphic you provided was from the screenshots there.

http://sourceforge.net/projects/aqemu/

 

[grumpybutfun]

Just as you should attribute quotes of other's comments, you should provide proper attribution to graphics when they're from another source.    For example, you should have included the link to sourceforce and noted that the graphic you provided was from the screenshots there.

http://sourceforge.net/projects/aqemu/

 

Apparently you missed the 4 or 5 times I have shown screenshots of aqemu (in the previous thread that is now deleted and others on here)...

 

I have also done the same for qemu-launcher, webvirt-manager, virt-manager, qtemu and all of these programs too: WebGUI for Virtualization Platforms (Xen, KVM, etc)

 

In my Arch thread, I list out 4 or 5 WebGUI Server Managers with links to their site.

 

In my Arch thread I list out 4 or 5 Server Monitoring Tools with links to their site.

 

The name of the software package is on the picture itself and instructed Peter / everyone else to go check it out. If people can't look at a picture, use their deductive reasoning skills and google it for themselves.... They shouldn't consider installing it.

[tallnerd1985]

Hi grumpy!

 

I have made more progress to update current 5.0.4 with support of KVM/QEMU.

 

You sure have.

 

Since you like running the QEMU commands (which I also do in KVM) you should take a look at the following:

 

 

That is something I like to have and try! but that's not possible on the system I running, looking forward to see what you come up with 

 

//Peter

 

Now does KVM/QEMU require a 64-bit kernel or can this be completely baked into the current UNraid 32-bit builds without issues?

[grumpybutfun]

Now does KVM/QEMU require a 64-bit kernel or can this be completely baked into the current UNraid 32-bit builds without issues?

 

You can run 64 Bit VMs on KVM / Xen even if the Host OS is 32 Bit. In fact, up till recently... That was the recommended way of doing it.

[Fx.]

"loading initial ramdisk"

 

i had some problems in here before, options were as below to fix: (if using traditional kernel build in arch)

 

(in-case someone else is searching)

https://www.archlinux.org/news/users-of-unofficial-kernels-must-enable-devtmpfs-support/

 

  - These options are required:

      CONFIG_DEVTMPFS=y

      CONFIG_HOTPLUG=y

      CONFIG_INOTIFY_USER=y

      CONFIG_NET=y

      CONFIG_PROC_FS=y

      CONFIG_SIGNALFD=y

      CONFIG_SYSFS=y

      CONFIG_SYSFS_DEPRECATED*=n

      CONFIG_UEVENT_HELPER_PATH=""

 

  - These options might be needed:

      CONFIG_BLK_DEV_BSG=y (SCSI devices)

      CONFIG_TMPFS_POSIX_ACL=y (user ACLs for device nodes)

 

 

(rebuilding now....)

 

*crosses fingers*

 

 

edit: nope not that

 

;(

 

editedit: i need to learn kernel patches and do it properly ;(

 

editeditedit: its not a 64/32bit thing...

[hernandito]

Hey Ironic,

 

I just sent you a little donation....  my heads is spinning reading all these posts... can you please reply here as to which topic thread to follow this?

 

Many thanks and Godspeed...

 

H.

[ironicbadger]

Hey Ironic,

 

I just sent you a little donation....  my heads is spinning reading all these posts... can you please reply here as to which topic thread to follow this?

 

Many thanks and Godspeed...

 

H.

 

Wow, thanks! That's so kind and generous of you.

 

This thread is for running stock unraid on arch, which I will update soon and the other thread is for

Discussing the merits of the fork project.

 

Clear as mud?

 

Sent from my Nexus 5 using Tapatalk

 

 

[finish06]

Can you please give us instructions on how to do this?  I see that you have done it, but I don't are any instructions on how to start, pursue this myself.

[Ice_Black]

centOS joins forces with RedHat!

 

http://www.centos.org/

[JustinChase]

I've done a quick check, and my processor is not able to pass-thru the hardware to the hypervisor (forgot the acronym).

 

So, what would be a good motherboard/processor to purchase to allow this to happen, and be as "future-proof" as reasonably possible?

 

Thanks!

[meep]

I've done a quick check, and my processor is not able to pass-thru the hardware to the hypervisor (forgot the acronym).

 

So, what would be a good motherboard/processor to purchase to allow this to happen, and be as "future-proof" as reasonably possible?

 

Thanks!

 

Hi Justin

 

I've just build a new system for this purpose and while I haven't gotten passthrough etc. up and running yet, all the checks indicate that the system is capable.

 

It's centered on a Gigabyte SKT-AM3+ 990FXA-UD5 Motherboard and AMD FX8320 Black Edition 8 Core CPU. Full details at my blog, link in sig.

 

Peter

[JustinChase]

...It's centered on a Gigabyte SKT-AM3+ 990FXA-UD5 Motherboard and AMD FX8320 Black Edition 8 Core CPU. Full details at my blog, link in sig.

 

Peter

 

Thanks for the link.  I enjoyed the read.  I posted a couple of comments on your blog, which would take this thread off topic, if you care to reply there or PM me.

[LinuxGuyGary]

Hi Badger,

 

 

Any new on this project ? I've been anxiously watching this because I have a server that can't do pass-through, but has lots of drive bays, cores and memory.  This looks like just the ticket.

 

 

Regards

LGG