64 Bit unRAID running natively on Arch Linux with full hypervisor support

[grumpybutfun]

If you ask me, the "secret sauce" is...

 

1. Running your VMs with Paravirtualized Drivers (Disk, Network, CPU, Filesystems, etc.) in Linux, Windows and OSX. That means your Host doesn't have to "emulate" an entire PC that your VMs run in. The more the Host has to "emulate" a PC the more work it has to do and your VMs will run slower.

 

2. LVM's - For your Host (even if you install it on a Flash Drive) and your VMs. This allows a lot of freedom with expanding, shrinking, moving things around, adding drives, removing them, mirroring, etc. and snapshots (Snapshots are similar to what Apple Time Machine does).

 

3. If you are running XBMC on your host or in your VMs... Use Video Hardware Acceleration.

 

If you do all those above, your CPU Utilization will hover around 10% or lower 95% of time unless you are transcoding Blurays, DVDs or things in Plex. Another example would be playing Crysis 3 in a Windows VM.

 

I did brief overview / write up explaining why you would want to use LVMs for your Host and VMs and why Snapshots are so great... HERE. Ironic also did a very nice write up too and you can read that HERE.

[dirtysanchez]

This is a PROVEN and RELIABLE technology and like I said earlier... If Fortune 500 companies are spending / investing billions into virtualization millions of mission critical servers into ESXi, XenServer, Hyper-V and KVM, you are going to be okay with your stuff too.

 

As far as this, I couldn't agree more.  I'm an IT guy myself, albeit mainly a networking guy and not a server guy.  My workplace has hundreds of virtualized servers and they are just as reliable as our bare metal servers.

[dheg]

I have a couple of points in which I'd really appreciate your input guys. Just bear with me for a sec.

 

I'm running an ESXi hypervisor and one of the VMs is a NAS4Free zfs server. I keep there the VMs and the cache drive. I'm happy with this setup because:

a. My cache drive has some level of protection

b. My other VMs not only are also protected but also have daily snapshots (which have come handy in a  couple of occasions)

c. I'm getting very good speeds with spinners

 

When moving to unRAID-extreme (man i hope this does actually happen!) my plan is to set up some kind of raid through the motherboard, which will keep the hosts and datastore 'protected' and daily snapshots with LVM saved in unraid shares. This will keep the host/VMs side covered, but what about the cache drive? I don't really want to lose the advantages (speed / redundancy) of my current setup.

 

So my questions are:

1. Overall, what do you think? Does it make sense?

2. Would it be better software raid, in case the mb dies?

3. What do you recommend for the cache drive?

 

Sent from my GT-I9305 using Tapatalk

 

 

[grumpybutfun]

When moving to unRAID-extreme (man i hope this does actually happen!) my plan is to set up some kind of raid through the motherboard, which will keep the hosts and datastore 'protected' and daily snapshots with LVM saved in unraid shares. This will keep the host/VMs side covered, but what about the cache drive? I don't really want to lose the advantages (speed / redundancy) of my current setup.

 

So my questions are:

1. Overall, what do you think? Does it make sense?

2. Would it be better software raid, in case the mb dies?

3. What do you recommend for the cache drive?

 

1. What little speed you might gain using Hardware Raid for what you are doing... Isn't worth the hassle.

 

2. You can use Software Raid and use a LVM on top of it. You could do RAID 0,1,5,6,10. Then you basically have an unRAID. You can add (doesn't matter what size, shape or speed) and remove it and expand the RAID / LVM very easily. You could mirror or have single or dual parity if you want. Then you have X amount of more space (or less if you removed a drive) for your Host, VMs, Data, ISOs, etc.

 

3. Why do you need a cache drive? Your VMs will download your TV Shows, Movies, etc. and copy them over to unRAID for you.

[WeeboTech]

2. You can use Software Raid and use a LVM on top of it. You could do RAID 0,1,5,6,10. Then you basically have an unRAID. You can add (doesn't matter what size, shape or speed) and remove it and expand the RAID / LVM very easily. You could mirror or have single or dual parity if you want. Then you have X amount of more space (or less if you removed a drive) for your Host, VMs, Data, ISOs, etc.

 

unRAID replaces the md layer.

When Software raid is suggested, is this via the motherboard fake raid drivers?

Does that avoid use of the md layer?

[jumperalex]

Indeed, this is my intention. I'm thinking the only plugin ill keep is apc and perhaps plex as its pre compiled. BUT and would be where the questions come from someone like grumpy who is clearly more knowledgable....if it was their server how would they do it.

 

I have a Xeon CPU

 

PS, its refreshing to see this forum with so much activity.

 

Well, I would still run Plex Media Server in a VM on CentOS.  While Plex is "precompiled" for UnRaid it is still what I would call sub-optimal.  Arguably your BEST functionality would come from PMS running in a Win7 VM because it would also give you access to Flash and Silverlight channels.  But ignoring that, I think the best option would be to use one of the other Linux packages they offer for CentOS, Fedora, Ubuntu, or FreeBSD.

 

In fact because we seem to be settling on CentOS, if you really wanted to run PMS on a bare-metal host you could just install it as a CentOS package vice an UnRaid Plug in. 

 

Come to think of it, I'm not sure the UnRaid plugin would work on Ironic's proposed distro because the PMS plugin is based on Slackware.  Anyone know enough to answer that bit?  Assuming I'm right I would be very surprised if the Plex Devs would see any benefit to creating an UnRaid-CentOS based plug-in vice just telling us to install the CentOS package [was that confusing enough?]

[dheg]

3. Why do you need a cache drive? Your VMs will download your TV Shows, Movies, etc. and copy them over to unRAID for you.

I do a bit of lightroom and photoshop, plus my Dropbox share.  These are continuously writing files,  without a cache my array will be up and running recalculating parity way too often.

With cache is just once per day.

 

Sent from my GT-I9305 using Tapatalk

 

 

[Ford Prefect]

2. You can use Software Raid and use a LVM on top of it. You could do RAID 0,1,5,6,10. Then you basically have an unRAID. You can add (doesn't matter what size, shape or speed) and remove it and expand the RAID / LVM very easily. You could mirror or have single or dual parity if you want. Then you have X amount of more space (or less if you removed a drive) for your Host, VMs, Data, ISOs, etc.

 

unRAID replaces the md layer.

When Software raid is suggested, is this via the motherboard fake raid drivers?

Does that avoid use of the md layer?

 

No, there is another layer based on dm/LVM...unRAID will use its md-module and you'll use LVM outside of the array.

[dheg]

1. What little speed you might gain using Hardware Raid for what you are doing... Isn't worth the hassle.

 

2. You can use Software Raid and use a LVM on top of it. You could do RAID 0,1,5,6,10. Then you basically have an unRAID. You can add (doesn't matter what size, shape or speed) and remove it and expand the RAID / LVM very easily. You could mirror or have single or dual parity if you want. Then you have X amount of more space (or less if you removed a drive) for your Host, VMs, Data, ISOs, etc.

Makes a lot of sense

[grumpybutfun]

No, there is another layer based on dm/LVM...unRAID will use its md-module and you'll use LVM outside of the array.

 

Thanks for cleaning this up for me.

 

I assume people have my knowledge on some things and need to provide more detail / explain things better.

[jumperalex]

3. Why do you need a cache drive? Your VMs will download your TV Shows, Movies, etc. and copy them over to unRAID for you.

I do a bit of lightroom and photoshop, plus my Dropbox share.  These are continuously writing files,  without a cache my array will be up and running recalculating parity way too often.

With cache is just once per day.

 

Sent from my GT-I9305 using Tapatalk

 

Ditto.  I still find a cache drive useful for keeping write speeds up (in the future I would call them non-local-VM writes?), my array spun down, and because of this http://lime-technology.com/wiki/index.php?title=Improving_unRAID_Performance#Use_Cache_Drive

A side effect of this is that file fragmentation is also reduced, if there is simultaneous streaming from the User Shares. Although the simultaneous writes produce fragmented files, the transfers later to their permanent locations on the parity protected drives are done one file at a time, which should normally create files without fragmentation.

[grumpybutfun]

Ditto.  I still find a cache drive useful for keeping write speeds up (in the future I would call them non-local-VM writes?), my array spun down, and because of this http://lime-technology.com/wiki/index.php?title=Improving_unRAID_Performance#Use_Cache_Drive

A side effect of this is that file fragmentation is also reduced, if there is simultaneous streaming from the User Shares. Although the simultaneous writes produce fragmented files, the transfers later to their permanent locations on the parity protected drives are done one file at a time, which should normally create files without fragmentation.

 

Mover is just a script. I'm sure the resident programing experts on here will have a script that you would customize to your system.

 

You could set up a cron job on your Host that would move files from X, Y and Z to you unRAID at whatever time and as often as you would like.

 

Also, your apps will have access to unRAID through NFS, Samba, HTML, sql, etc. and if you want things to write directly to unRAID or read from unRAID, etc. you could do that also.

[peter_sm]

So in this case, I do not need to move files between my drives from windows using samba ( thats slow ;-) )? so i can mount each drive direct and moves files direct on the server ?

 

Setup a VM to DL nzb's, store all DL on my cache on unraid .....

 

I can't wait to test this, and it looks very cool, so easily set up VM for difference need and testing

 

I'm reading up all I can regarding KVM LVM ..... 

 

 

So would I be able to connect my server from Windows RD, and control it this way? my server are hidden i my basement

 

And all plugin can easily be replaced by install software with yum instead ?

//Peter

 

 

[jumperalex]

?? Sorry if I'm being obtuse, but how am I to do that when I want for example, to run a backup of my PC and save it to the array?  Or copy a batch of photos from my PC onto my array?  Or my GF does the same from her laptop?

 

They have to land somewhere right?  Either directly on the array, or some "holding area".  No?  I would prefer they land on a disk as fast as possible, and then sometime at night the script runs to move them all into the parity protected array.

 

Or are you talking about using the physical host boot drive as that holding area?  Functionally still acting as a "cache" drive for the array even if it isn't an UnRaid-Cache-Drive as we define them now in the gui?

[grumpybutfun]

So in this case, I do not need to move files between my drives from windows using samba ( thats slow ;-) )? so i can mount each drive direct and moves files direct on the server ?

 

You need to ask this question better. There are several ways I could take this and do not want to give you wrong info.

 

So would I be able to connect my server from Windows RD, and control it this way? my server are hidden i my basement

 

You have always been able to do this using SSH or Telnet.

 

And all plugin can easily be replaced by install software with yum instead ?

 

That is correct.

[grumpybutfun]

?? Sorry if I'm being obtuse, but how am I to do that when I want for example, to run a backup of my PC and save it to the array?  Or copy a batch of photos from my PC onto my array?  Or my GF does the same from her laptop?

 

unRAID works the same as it did before. If you have Samba / NFS shares... They show up on the network.

 

Going into the POWER USER Zone...

 

You can also set up additional shares to various folders, drives, etc. on your Host Server. You aren't stuck with only what unRAID shares. If you have a SSD drive on your Host outside of unRAID with a folder called Stuff. You would share Stuff on the Host (not in unRAID because unRAID can't see it) and you could copy things there if you so desired.

 

Or are you talking about using the physical host boot drive as that holding area?  Functionally still acting as a "cache" drive for the array even if it isn't an UnRaid-Cache-Drive as we define them now in the gui?

 

Basically.

 

For most of you, I would just use unRAID and the cache drive as you always have. Doing the things above is getting down into the weeds and doesn't really get you anything. Eventually some of you will see posts / learn things and might want to do more. The good thing is you can do this in steps and learn / expand your knowledge and apply what you learn when ready.

[peter_sm]

 

 

You need to ask this question better. There are several ways I could take this and do not want to give you wrong info.

I map all my disk as disk shares in windows, them moving files between disk share (over samba), I need/like to have direct access instead ( one way i have try is using putty and MC)

 

//Peter

[Harpz]

Keep coming back to this thread after a days work and getting excited to see all the new ideas being batted about and what could be achieved with an unRAID based distro, but please don't say its going to be called "unRAID Extreme edition"  I'm sorry to say but what a shit name lol.

 

Playing devils advocate I have a question though, who would be ultimately responsible for the maintaining the distro, Tom or is it a community effort?

 

I ask because we've all seen how easy it can be for people / maintainers / helpers past and present to take offence and bugger off with the arse.

 

The way I see it at the moment is that it would only take a few very intelligent individuals to get disheartened, the strop, annoyed, etc, etc, with the way things are going and leave and we could be in allot of trouble with no support, thus leaving people up shit creek with out a paddle.

 

What guarantees do we have this isn't going to happen as I'm sure like me allot of the people here value there data as much as I do?

[peter_sm]

 

So would I be able to connect my server from Windows RD, and control it this way? my server are hidden i my basement

 

You have always been able to do this using SSH or Telnet.

 

 

 

 

Do I get the GUI / desktop of CentOS using telnet or SSH  connecting my server;) or is it only the command prompt ?

 

//Peter

 

 

EDIT

 

it will probably be much easier to try things out for us when we can help with beta testing, instead if asking stupid questions   

[jumperalex]

?? Sorry if I'm being obtuse, but how am I to do that when I want for example, to run a backup of my PC and save it to the array?  Or copy a batch of photos from my PC onto my array?  Or my GF does the same from her laptop?

 

unRAID works the same as it did before. If you have Samba / NFS shares... They show up on the network.

 

Going into the POWER USER Zone...

 

You can also set up additional shares to various folders, drives, etc. on your Host Server. You aren't stuck with only what unRAID shares. If you have a SSD drive on your Host outside of unRAID with a folder called Stuff. You would share Stuff on the Host (not in unRAID because unRAID can't see it) and you could copy things there if you so desired.

 

Or are you talking about using the physical host boot drive as that holding area?  Functionally still acting as a "cache" drive for the array even if it isn't an UnRaid-Cache-Drive as we define them now in the gui?

 

Basically.

 

For most of you, I would just use unRAID and the cache drive as you always have. Doing the things above is getting down into the weeds and doesn't really get you anything. Eventually some of you will see posts / learn things and might want to do more. The good thing is you can do this in steps and learn / expand your knowledge and apply what you learn when ready.

 

Yup, OK I gotcha.  Makes sense and I can see when and where I might find myself doing it that way if for no other reason than to avoid wasting a SATA-port and having a drive sit empty.  And you're right that to start I'll probably use unRaid's cache and mover functionality while learning my way around things. 

[grumpybutfun]

Playing devils advocate I have a question though, who would be ultimately responsible for the maintaining the distro, Tom or is it a community effort?

 

From what little I heard and rumors... I believe this is going to be a Limetech product which they create / own.

 

Haven't heard anything official from Tom but it sounds like he is going to leverage some talent to assist him in this endeavor. That is not to say that Tom doesn't posses the skills, knowledge, experience to do it... I think it's more of a time / bandwidth deal.

 

What guarantees do we have this isn't going to happen as I'm sure like me allot of the people here value there data as much as I do?

 

I don't see Tom releasing a Limetech product that he doesn't believe in / support / maintain.

[WeeboTech]

Question.  Re: Passwords & /etc/shadow

 

There will now be two interfaces to add users.

Will there be issues with the passwords?

 

I remember there being issues years back between systems the old crypt style password and the md5 style password.

 

Then we have uid/gid issues.

I actually wish unRAID handled uid/gid/groups better.  In any case, have user accounts/passwords been changed and tested.  I know there are bigger fish to fry, however this could be a point of contention.  I'm not sure if anyone has thought of it yet.

[grumpybutfun]

Do I get the GUI / desktop of CentOS using telnet or SSH  connecting my server;) or is it only the command prompt ?

 

Worst case scenario

 

"unRAID Nerds Edition" installs with only a command prompt.

 

Solution

 

yum groupinstall “KDE desktop"

 

[dirtysanchez]

"unRAID Nerds Edition"

 

;D

[peter_sm]

OK I thought it should be a GUI, so you can maintain all VM remotely (RD) using virt-manager