[GUIDE] Virtualizing unRAID in KVM on openSUSE 13.1 (64Bit) <--- Completed 12/19


Recommended Posts

  • Replies 159
  • Created
  • Last Reply

Top Posters In This Topic

I must say that this is an exciting project. I applaud grumpybutfun and ironicbadger for what they did for the community.

 

I have silently read all of the pages here, plus the 25 pages of this thread: 64 Bit unRAID running natively on Arch Linux with full hypervisor support. I was silent because I do believe that this was a great discussion between Linux "adults" where I do not belong to. Anyhow, I'm now feeling ready to move on and trying to reach out the next level of virtualization, using my actual HW environment (that should be fine, only caveat is that the MB doesn't support S3)

 

ESXi is running stable but I feel "locked in" - at least with the free version of ESXi. Also the maintenance via a Windows app is not ideal for a Mac user.

 

Currently I'm having AirVideo, mySQL (for XBMC), openSSH, Cache Directories and Open VM Tools running as plugins. Additionally I have two VMs:

  • Windows 7 for anything that needs Windows
  • Ubuntu 13.10 for openHAB (my Home Automation) and ownCloud

 

Before I begin I do have some questions:

  • Is there any way to migrate the two VMs or is it true, that KVM can deal with VMDKs? Is there any downside? This would assure a smooth migration before I move openHAB and ownCloud to the openSuse VM
  • Is the MV8 hack that was needed for ESXi an issue?
  • Should I use one SSD of my HW Raid to install openSuse according to grumpys guide, leaving the second SSD untouched?? In that case I would be able to switch between ESXi and the openSuse 64bit KVN setup by using one or the other SSD.

Link to comment

You are right, switching back and forth between KVM and ESXi is not the objective, it's just about keeping the SSD with the ESXi install as life line as long as the openSuse/KVM setup is not finalized.

 

After migrating all apps I will of course put both SSD back into the Raid enclosure (this 3-5" enclosure contains two 2.5" SSDs and is running them in Raid 1)

Link to comment

Before I begin I do have some questions:

  • Is there any way to migrate the two VMs or is it true, that KVM can deal with VMDKs? Is there any downside? This would assure a smooth migration before I move openHAB and ownCloud to the openSuse VM

 

There are plenty of free migration tools to convert from ESXi to Xen to KVM and back and forth between them.

 

  • Is the MV8 hack that was needed for ESXi an issue?

 

I believe there are several people who have used an MV8 card in Xen without a hack. I do not have one so I can't speak to that. However, if you do not have to hack it in Xen... you wont in KVM. Xen and KVM both use QEMU (the machine emulator and where the "magic" happens).

 

  • Should I use one SSD of my HW Raid to install openSuse according to grumpys guide, leaving the second SSD untouched?? In that case I would be able to switch between ESXi and the openSuse 64bit KVN setup by using one or the other SSD.

 

If I were in your shoes, did not posses a lot of Linux Experience / Knowledge... I would think of this a project (a learning one at that) / Hobby that you pick up and put down at your own pace and time.

 

There are some benefits to switching to Linux but let's be honest... This would be a hobby / toy / learning experience.

 

XenServer / ESXi does everything you want. You can run unRAID in a VM and use VMs for all the various Linux / Windows apps you want and never need to install openSUSE and use KVM.

 

The good news is, if you want to "play" / try things out / get your feet wet with unRAID in a VM on Xen or KVM... You can do this all in stages without affecting unRAID or XenServer and ESXi.

 

My advice would be to install openSUSE to a USB Flash Drive (a project), test passthrough (a project), put unRAID in a VM (a project), load XBMC if you want (a project), Install the WebGUIs for server management and learn them (a project), etc.

 

You could stop / start / pick up any of those projects at any point if you install openSUSE to flash drive. When done "playing"... Simply unplug your openSUSE flash drive, plug in your unRAID drives again (if you haven't gotten to the point of unRAID in a VM) and boot right back into ESXi. As far as unRAID and the Data is concerned... nothing has changed.

 

Once you get comfortable with openSUSE and Linux and have success with unRAID in a VM, passthrough working, creating VMs... THEN I would consider / make plans for installing it onto a local disk and converting ESXi VMs to KVM or Xen.

 

Bottom Line...

 

If you do not want to learn, if you do not have time, if you do not want another project (same type one when you installed / learned ESXi / XenServer), if you do not want to use your Machine a dual purpose (Mine for example, I boot straight into XBMC so I have one less XBMC PC to deal with in my house)... Stick with XenServer / ESXi. You really do not gain anything.

One last thing to consider...

 

If what we think might happen.... Tom possibly has a separate version of unRAID where it runs KVM as a Host.... There are TONS and TONS of advantages to doing that over XenServer / ESXi. If that interests you and Tom delivers on that... This is a good way of dipping your toe in the pool and learning everything you need before taking the next leap should that ever become a reality.

Link to comment

I have a couple of blanks posts on page one for a reason.

 

Once more people are successful in doing this on openSUSE they are going to want to you use LVMs, run it headless, manage it remotely with the WebGUIs and possible a KVM too. You can remote control the Server via VNC, add a PXE Server, add shares on the host via VirtFS (A LOT faster than NFS and Samba) that your VMs use, provide a newer Linux Kernel you easily install so you can passthrough nVidia Cards to your VMs, etc.

 

I will continue to add guides (to this guide) once more people are running this and have their feet underneath them.

 

I picked openSUSE for a reason. It's EASY, it's a good Linux Distro to get your feet wet / easy to learn, their documentation / forum / wikis are REALLY good, it's popular, their webgui(s) to manage the server is top notch, its easy to upgrade / bolt on new things, it uses systemd, the versions of KVM and QEMU that it runs is more upstream than other Linux Distros, it was designed to make installing / enabling KVM / Xen simple compared to other Linux Distros, etc.

Link to comment

Hello All,

First, a big thank you to grumpybutfun for producing this installation guide! These recent threads on KVM virtualization have finally made me want to get my feet wet. I cobbled together some older hardware which should be VT-d compliant (Intel DQ45CB mobo + Core2 Quad Q9300 processor + 6gB DDR2) and have been stepping through grumpy's guide.

 

OK, I'm at Step 9 of "Installing unRAID in a VM," and I have grumpy's "starter" unRAID.vhd running. There are a couple of hiccups with the vhd that I encountered and I understand it could be something I missed in the guide. When the vhd is booting the message, "waiting for /dev/disk/by-label/UNRAID" echoes for 30 seconds or so but finally goes away and we get a logon prompt. Also, when this vhd running I can't connect to unRAID's webGUI and the vhd will not gracefully power down. Hopefully this is not a big deal, maybe it's by design. However, I can telnet into the virtual unRAID server.

 

I'm stuck for the moment since I don't have a separate SATA controller card yet. But at this point, I don't want to move my existing licensed unRAID server and drives to it; I'd just like to use the free/demo version of unRAID and perhaps build a test array.

 

So, while I'm waiting for the additional controller to show up, what do I need to do to get a working virtual demo-unRAID system. Would it be as easy as plugging in a demo USB key?

 

Thank again for the detailed guide!!

Link to comment

OK, I'm at Step 9 of "Installing unRAID in a VM," and I have grumpy's "starter" unRAID.vhd running. There are a couple of hiccups with the vhd that I encountered and I understand it could be something I missed in the guide. When the vhd is booting the message, "waiting for /dev/disk/by-label/UNRAID" echoes for 30 seconds or so but finally goes away and we get a logon prompt. Also, when this vhd running I can't connect to unRAID's webGUI and the vhd will not gracefully power down. Hopefully this is not a big deal, maybe it's by design. However, I can telnet into the virtual unRAID server.

 

It is looking for an unRAID USB Flash Drive. If it doesn't find it, the unRAID WebGUI is not going to work, telnet, etc.

 

Format and make a new unRAID USB Flash Drive per the instructions on this site and try starting the VM again. If successful, you shouldn't see the waiting for /dev/disk/by-lable/UNRAID and the webGUI should work.

 

IMPORTANT NOTE:  You are simply testing if your CPU, Motherboard, USB Controller, sata controller, openSUSE, your unRAID VM, etc. is set up correctly.

 

Do not have your unRAID drives hooked up or powered on or use your Original unRAID USB Flash Drive!

 

 

 

Link to comment

It is looking for an unRAID USB Flash Drive. If it doesn't find it, the unRAID WebGUI is not going to work.

 

Format and make a new unRAID USB Flash Drive per the instructions on this site and try starting the VM again. If successful, you shouldn't see the waiting for /dev/disk/by-lable/UNRAID and the webGUI should work.

 

IMPORTANT NOTE:  You are simply testing if your CPU, Motherboard, USB Controller, sata controller, openSUSE, your unRAID VM, etc. is set up correctly.

 

Do not have your unRAID drives hooked up or powered on or use your Original unRAID USB Flash Drive!

 

Thanks!! I will plug in that new unRAID USB Flash Drive. Thanks also for the quick reply!!!

Link to comment

I've got the USB passed through and it was recognized.  Got into unRaid OK, and was able to connect via separate pc to the unRaid webgui, so we are making progress.  Still no joy on passing through the pci sata ports.  Either a separate 2 port controller or a Supermicro PCI-X SAT2-MV8.  As soon as I try to pass it through I get errors like:

 

7nJDHaf.png

 

However in mucking around pci passthrough I seem to have killed my network connection.  Likely caused by having to powerdown while the system was unresponsive.  Now bitrfs,  is unable to recover a previous snapshot for me and take my system back to a prior good state.

 

What is the best way to do a btrfs file system check?  I've tried the following without joy....

 

# snapper undochange 54..57

mkdir failed path errno 2 (no such file or directory)
failed to modify /.readahead

# btrfs check /dev/sda2

Check tree block failed, want=20975616, have=2334390125295050853
... and a bunch more of these errors
read block filed chec_tree_block
Couldn't read tree root
Critical roots corrupted, unable to fsck the FS

Link to comment

A buyers question regarding choice of video card for XEN/KVM and media/game use.

 

I want to buy a new video card, but I was wondering if someone can recommend 3 videocards (price category) that currently is nice for the following:

 

- Good xen/kvm support (mostly ati?)

- Good madvr support (mostly nvidia?)

 

-cheap -middle -$$

 

Also, I will end up most likely using 2 SATA cards in both my PCIe 3.0 x16 slots, Is it possible to use a PCIe 2.0 x1 slot for a video card? with an adapter or crazy mod? Is there a better option? Supermicro SATA cards are x4 and x8 slots mostly. What is a good solution when having in mind 2x SATA card and a GPU? I use the Asrock z77 extreme4 board. thanks

Link to comment

Looks like you are hosed. Even if you weren't, you would probably spend more time trying to fix it than reinstalling.

Did your Motherboard, CPU, Controller work in XenServer or ESXi?

 

This system was never used with Xen or Esxi.  The fact that the usb was so easy to pass through suggests that the sata ports should work too, shouldn't they??  Can I run unRaid without sata hardware passthrough?

 

Reinstalling suse now. 

 

Once I get it to a stable state I need to clone the boot drive so its easier to recover from.  Will clonezilla live cd do a good job on a btrfs boot disk -> USB drive, or network?

Link to comment

This system was never used with Xen or Esxi.  The fact that the usb was so easy to pass through suggests that the sata ports should work too, shouldn't they??

 

I have no idea. I'd say google is your friend.

 

Can I run unRaid without sata hardware passthrough?

 

You could boot into and unRAID and it will work.

 

You could do a Full Slackware install using the wiki (Good Luck, you will need it!)

 

Once I get it to a stable state I need to clone the boot drive so its easier to recover from.  Will clonezilla live cd do a good job on a btrfs boot disk -> USB drive, or network?

 

That is exactly what I do. Fresh Install the use Clonezilla to make an image.

 

I PXE Boot in Clonezilla and back it up over NFS to an Backup Image Folder. It's very fast unless you have a lot of NTFS partitions.

Link to comment

I want to buy a new video card, but I was wondering if someone can recommend 3 videocards (price category) that currently is nice for the following:

 

- Good xen/kvm support (mostly ati?)

- Good madvr support (mostly nvidia?)

 

-cheap -middle -$$

 

Host

 

Either nVidia or AMD is fine.

 

VM

 

If you get an nVidia card, better make sure it is Quattro Series or one where you can flash the ROM to think it is. Otherwise you will not be able to passthrough the card. At least not for a while anyway.

 

AMD, I haven't run across a card that wouldn't. 5XXX, 6XXX, 7XXX. If you look at the first post in this thread, I have links to places to tell you what is known to work.

 

I'm not a gamer so I can't suggest a card. I just got a couple of AMD 6540 for XBMCs ($15 or less) and they work well for me.

 

Also, I will end up most likely using 2 SATA cards in both my PCIe 3.0 x16 slots, Is it possible to use a PCIe 2.0 x1 slot for a video card? with an adapter or crazy mod? Is there a better option? Supermicro SATA cards are x4 and x8 slots mostly. What is a good solution when having in mind 2x SATA card and a GPU? I use the Asrock z77 extreme4 board. thanks

 

How many drives are you installing?

 

I have that same motherboard.  You can use 4 of the SATA ports on the motherboard for passthrough and leave the 5th one for the OS and VMs.

 

I haven't been in the market for hardware for a while so I am not up to date like many of the other people here will be.

Link to comment

I've got the USB passed through and it was recognized.  Got into unRaid OK, and was able to connect via separate pc to the unRaid webgui, so we are making progress.  Still no joy on passing through the pci sata ports.  Either a separate 2 port controller or a Supermicro PCI-X SAT2-MV8.  As soon as I try to pass it through I get errors like:

 

I am getting just about the same error message.  Tried both an M1015 and a two port sil3132 board with same results.  Changing PCIe slots has no effect either. But Unraid  would start up without trying to pass through a sata board.  I am using the same MB and CPU listed in my sig below.  Its my backup hardware so I can play around with this at no risk.

 

Are you using an Intel processor by chance?

 

The error message is confusing to me but I think the key may be in the "No IOMMU found" part.

 

Take a look at this at paragraph 12.3.4:

 

http://activedoc.opensuse.org/book/opensuse-virtualization-with-kvm/chapter-12-running-virtual-machines-with-qemu-kvm 

 

Make sure that CONFIG_DMAR_DEFAULT_ON is set in the host's running Kernel:

 

grep CONFIG_DMAR_DEFAULT_ON /boot/config-`uname -r`

 

If this option is not set, edit your boot loader configuration and add intel_iommu=on (Intel machines) or iommu=pt iommu=1 (AMD machines). Then reboot the host machine.

 

My system did not have "CONFIG_DMAR_DEFAULT_ON" but I don't know how to go about properly putting "intel_iommu=on" in my boot loader.  And I checked the bios thoroughly just to make sure everything related to virtualization was turned on.  I'm just a linux novice at this point.  Give me time and I will start coming up to speed.  Just a lot of new stuff for me to get my head around right now.

 

However in mucking around pci passthrough I seem to have killed my network connection.  Likely caused by having to powerdown while the system was unresponsive.  Now bitrfs,  is unable to recover a previous snapshot for me and take my system back to a prior good state.

 

I had that happen to me too while I was trying to my networking configured.  For whatever reason my gateway/dns settings would just disappear.  They seem to be stable now though.  I didn't use btrfs but I did use LVM when setting up my system.

Link to comment

The error message is confusing to me but I think the key may be in the "No IOMMU found" part.

 

You have to have a Motherboard that supports IOMMU and I don't know if yours does.

 

My system did not have "CONFIG_DMAR_DEFAULT_ON" but I don't know how to go about properly putting intel_iommu=on in my boot loader.  I'm just a linux novice at this point.  Give me time and I will start coming up to speed.  Just a lot of new stuff for me to get my head around right now.

 

Edit the Bootloader in Yast. I don't have it up in front of me but you should be able to edit the settings.

 

You will see something similar to this...

 

GRUB_CMDLINE_LINUX_DEFAULT="quiet" <--- You might have more stuff there.

 

Make yours look like this...

 

GRUB_CMDLINE_LINUX_DEFAULT="quiet intel_iommu=on"

 

When you Apply and go back to Yast it will update your bootloader for you.

Link to comment

I have the same errors as tr0910 and kegler and was trying to edit the bootloader. These are the optional parameter:

resume=/dev/system/swap splash=silent quiet showopts

 

...and here the Failsafe-Parameter:

showopts apm=off noresume edd=off powersaved=off nohz=off highres=off processor.max_cstate=1 nomodeset x11failsafe

 

The system was running with ESXi btw. so should be fine from a HW perspective.

 

Another issue that I observed: my CPU is not listed. You can choose SandyBridge and Haswell but not my XEON E3-1240V2 which is an IvyBridge.

Link to comment

I have the same errors as tr0910 and kegler and was trying to edit the bootloader. These are the optional parameter:

resume=/dev/system/swap splash=silent quiet showopts

 

...and here the Failsafe-Parameter:

showopts apm=off noresume edd=off powersaved=off nohz=off highres=off processor.max_cstate=1 nomodeset x11failsafe

 

Somewhere in Yast under bootloader there is an EASY way to add kernel options to grub without actually editing the grub.cfg itself.

 

GRUB_CMDLINE_LINUX_DEFAULT=

 

Might be a few days before I have time to look. I know it is in there but I don't have the exact menu option or screenshots.

 

Another option is for you to go ask the guys in the openSUSE forum what steps to take in Yast / Bootloader to open that setting.

 

Another issue that I observed: my CPU is not listed. You can choose SandyBridge and Haswell but not my XEON E3-1240V2 which is an IvyBridge.

 

Your CPU is not listed where?

Link to comment

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.

Guest
Reply to this topic...

×   Pasted as rich text.   Restore formatting

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.