Virtualizing unRAID with Xen on Arch Linux, XBMC & Windows with VGA Passthrough

[ironicbadger]

If you want to see all the instructions the best place is my blog, but this post has a nice overview of what you can achieve.

http://blog.ktz.me

 

This is the setup as it currently stands, should give you an idea of what's possible. Bear in mind this is all running on one box!

 

Dom0 (Host) – Arch Linux

 

Arch Linux with Xen Hypervisor

XBMC via the motherboard HDMI out powered by the i5 3470?s HD4000 internal GPU.

XBMC MySQL database host.

 

DomU1 – unRAID 5.0

 

18TB of protected storage (JBOD + parity drive) which only spins the drive it’s currently reading from – this is very important for me when I have 12 drives!

PCI Passthrough of 2x SATA controllers – 1x Supermicro SAS-AOC-SASLP-MV8 (8 ports), 1x motherboard SATA controller (6 ports)

unRAID Server is a Network Attached Storage server operating system that boots from a USB Flash device and specifically designed for digital media storage.

 

DomU2 – ArchU

 

SABnzbd, Sickbeard, Couchpotato, Subsonic, OwnCloud, Dropbox, Google Drive, Headphones, Plex Media Server, BTSync and Air Video Server.

This VM automates all the media collection from the Internet and makes sure, via BTSync, that my photo collection is exchanged with a friend for backup purposes.

 

DomU3 – Windows 8.1 with VGA passthrough

 

My desktop computer.

My gaming rig.

VGA passthrough of a Sapphire HD7870 Ghz edition.

HDMI output from GPU split 2 ways – 1 to Lounge TV, 1 to desktop monitor upstairs via 30m Cat6 HDMI extenders.

USB passthrough connected to wireless Xbox 360 controller for Lounge gaming.

30m USB over Cat5e extenders for keyboard and mouse to a powered USB hub upstairs on my desk.

 

 

Introduction

 

In the last week I've spent a great deal of time learning about virtualisation and hypervisors. Having messed about with ESXi previously but finding various issues with it on my hardware, I decided to give Xen a go after reading through 'ddeeds' thread http://lime-technology.com/forum/index.php?topic=28665.0. He's been a bit of an inspiration and a great help getting this setup working. Thanks mate! There is a ton of great info here and I urge you to read it as it will answer a great many questions you didn't know you had!

 

Now, I wanted to make my unRAID box do a bit more, running one PC per task is so 1999. My vision was simple, run XBMC and unRAID on the same box as my server sits within HDMI cable range of my main TV near the lounge. What I've ended up with is Arch Linux as my main OS, which runs XBMC via the Intel HD2500 graphics on my CPU (which comfortably handles bluray quality MKV playback with video at 35MB/s) and a mysql database for the library, unRAID as a HVM guest to serve the house with media, another HVM linux guest to take care of usenet / plex server duties and a Windows 8 HVM guest (VGA passthrough coming soon) to replace my desktop PC completely.

 

To cap this all off, I get better parity check speeds using Xen than I did on ESXi and, unbelievably, better than I got "bare metal"! Well worth doing this then...

 

 

So, to summarise one box runs unRAID, XBMC to my TV, Windows to desktop monitor (upstairs), pfsense and also all my usenet / plex / tvheadend stuff. One box. Very, very cool and something I've been trying to achieve for many years and I'm pleased to share with you all the info required to make this happen. Here we go!

 

Hardware

 

My most important requirement was to use my existing hardware as I can't afford to replacement motherboard or CPU often and wanted to get the most from the gear I'd already got.

 

• Case Antec 902 - chosen for it's 9x 5.25" optical bays. Ultimately, I aim to have 15 drives up from the current 9 in the system by using 5x3 drive caddies.
  
• CPU Intel i5 3470 - very important to make sure that the CPU supports vt-d, check on the Intel website for support in this regard. vt-d is required to make the good stuff like PCI passthrough work, more on this later.
  
• Motherboard Asus P8H77-V LE - chose this board due to the high number of expansion slots as there's plenty of room for further SAS cards, TV cards and multiple VGA cards. There seems to be some confusion online as to whether or not this motherboard supports vt-d virtualisation technology, I can confirm 100% that it does.
  
• RAM 24GB DDR3
  
• PSU Corsair TX750M - partly modular power supply, nice and beefy to power all the gear in this box and thanks to the modular nature it will easily supply power to the 2x video cards and 1x pci-e dvb-s2 card in the box as well as all the hard drives
  
• HBA / SAS Card Supermicro SAS-AOC-SASLP-MV8
  
• NICs
  • Realtek 8169 based on-board NIC (bridge for Arch and all VMs)
  • Intel Gigabit PRO/1000 GT Desktop Adapter (used only for pfsense)

  [*]Hard drives (ArchX)

  • 60GB OCZ SSD boot drive
  • 500GB Seagate HDD data store drive

  [*]Hard drives (unRAID)

  • Parity: 3TB WD Red
  • Data: 3x 3TB Hitachi
  • 1x 3TB WD Red
  • 1x 1.5TB Seagate
  • 1x 1TB Seagate
  • Total stats 14.5TB of protected data

 

Installation of Arch Linux

 

See my blog here for simple installation instructions. http://blog.ktz.me/?p=131

 

Installation and setup of Xen

 

Once you've installed Arch, it's time to install Xen - which is essentially just another application like any other. Currently, the version of Xen-git available from yaourt is broken but ddeeds kindly compiled a version that I will share with you all at this link.

 

https://dl.dropboxusercontent.com/u/6775695/xen-4.3.0-4-x86_64.pkg.tar.xz

 

Once Xen is installed, run 'sudo xl info' and you should get something similar to this. If you do, you're all set to start virtualising unRAID or any other VM for that matter!!

 

host                   : ArchX
release                : 3.10.6-2-ARCH
version                : #1 SMP PREEMPT Tue Aug 13 09:47:01 CEST 2013
machine                : x86_64
nr_cpus                : 4
max_cpu_id             : 3
nr_nodes               : 1
cores_per_socket       : 4
threads_per_core       : 1
cpu_mhz                : 3200
hw_caps                : bfebfbff:28100800:00000000:00007f00:77bae3ff:00000000:00000001:00000281
virt_caps              : hvm hvm_directio
total_memory           : 7877
free_memory            : 248
sharing_freed_memory   : 0
sharing_used_memory    : 0
outstanding_claims     : 0
free_cpus              : 0
xen_major              : 4
xen_minor              : 3
xen_extra              : .0
xen_caps               : xen-3.0-x86_64 xen-3.0-x86_32p hvm-3.0-x86_32 hvm-3.0-x86_32p hvm-3.0-x86_64 
xen_scheduler          : credit
xen_pagesize           : 4096
platform_params        : virt_start=0xffff800000000000
xen_changeset          : 
xen_commandline        : /xen-4.3.0.gz xsave=1
cc_compiler            : gcc (GCC) 4.8.1 20130725 (prerelease)
cc_compile_by          : sbraden
cc_compile_domain      : (none)
cc_compile_date        : Wed Aug 14 03:42:55 EDT 2013
xend_config_format     : 4

 

Virtualising unRAID

 

Step 1 is setting up PCI passthrough so unRAID can see your drives. I've written an extensive blog post about it here

http://blog.ktz.me/?p=166

 

Step 2 is using ddeeds unRAID files to get your actual VM going using info from this thread (near the bottom of post #1 is want you actually want if you're already at this stage, but there's a ton of great info so don't miss out).

http://lime-technology.com/forum/index.php?topic=28665.0

[vl1969]

Nice work Ironic,

 

I have used your writeup for Ubuntu and Xen

and as of yesterday have my box running Ubuntu 13.10.5 + Xen 4.3

there is a small issue with your Xen writeup

in the clean up section , it need to be a little clearer for a  layman like me :-) not to make mistake. as I was deleting the xen-4.3.0.gz file along with others and could not for the love of god figure out why my setup was going "kaput" every time I install Xen

 

*--*

e.g. - sudo rm xen-syms-4.3.0

xen.gz xen-4.gz xen-4.3.gz xen-syms-4.3.0

*--*

 

So I will continue on my endeavor to VM unraid on my setup (for an uninitiated I am trying to build same setup as ironic here, on AMD hardware from TAMs. even though most believe it is not possible. I am set to prove them wrong  ) 

 

PS>> will you help ddeeds with writeup for GIU Management ?

or he is all alone on that quest :-P

[Dephcon]

How do you find unraid compares between ESXi 5.x and Xen 4.x?

[ironicbadger]

Xen performance is roughly 10-15% better. About an hour quicker and when I pass thru my satellite card to tvheadend it doesn't pixelate like it did on esxi

 

Sent from my GT-I9505 using Tapatalk 2

 

 

[Dephcon]

That would really help on AMD hardware, which is balls slow to begin with.  Argh I hate going from what I know (esx) to something new (linux anything).

[vl1969]

That would really help on AMD hardware, which is balls slow to begin with.  Argh I hate going from what I know (esx) to something new (linux anything).

 

I don't understand why so many people dislike AMD.

all I hear it's slow, it's slow...

 

how is it slow?

 

I have been using AMD PCs for the last 20 years.

never noticed that it's slow. 

all of my PCs are AMD based.

 

I have a laptop with AMD  and 4GB ram

 

that migrate over last 5 years from Vista to XP to Win7

and it works great.

I have even transcode movies on it, created DVDs

done photoshop stuff.

 

I have a friend with similar Intel based machine same age and very close parameters, and he always complains on how slow it is compared to mine.

 

what are all of you guys doing?

 

[Dephcon]

Single thread performance is brutal, that's just how it is.  I'm piecing together a FX-8320 system for virtualizing unraid so I don't hate AMD, but they're WAY behind Intel.

[BRiT]

That would really help on AMD hardware, which is balls slow to begin with.  Argh I hate going from what I know (esx) to something new (linux anything).

 

I don't understand why so many people dislike AMD.

all I hear it's slow, it's slow...

 

how is it slow?

 

Simply put another way, Intel is significantly faster.

[ironicbadger]

Updated my blog with a nice simple guide on how to install Arch Linux.

 

http://blog.ktz.me/?p=131

[jbrodriguez]

how do you get to the text install menu ? there's an automated_script.sh in the root home dir, but it does nothing when invoked, seems it needs an argument.

[ironicbadger]

how do you get to the text install menu ? there's an automated_script.sh in the root home dir, but it does nothing when invoked, seems it needs an argument.

 

how have you booted the ISO? via USB or DVD or....

 

You did download the .iso and not any other installer right? This link should help if you didn't.

 

http://mirror.bytemark.co.uk/archlinux/iso/2013.08.01/

[tr0910]

The FX-8350 is $100 cheaper than the i7-3770... I would buy an AMD every day of the week and twice on Sunday if it's purpose is to power an ESXi / Xen Server.

 

But the Intel has a TDP of 77w, and the AMD is 125w.  I wonder what the bottom line idle power usage will be for both system??  Isn't the Intel likely to use less power just idling?

[tr0910]

However, I do understand that to some people this is very important... They will look at the Intel vs AMD and make the best choice based on their wants / needs and factor in energy use / consumption.

 

Presently my one current technology server is Xeon 1220 / SUPERMICRO MBD-X9SCM-F-O / 2-m1015 and 1 - MV8.

 

What motherboard would I use for AMD that gives me the same IPMI manageability as the Supermicro above??  I don't see any server boards for the AMD FX 8350.

 

I have 3 of these Tam Solutions Xeon servers, along with 2 more AMD servers that are the same as vl1969 got working with Xen (have been following your discussion with interest)  The Intel ones have X7DBE-X, Xeon 5130, 3 SAT2-MV8.  Wonder if these Intel Xeon's will work with Xen too??

 

[ironicbadger]

Little update to my server this week, I purchased a pair on 5in3 drive cages. I'm extremely pleased with the results, apart from having the bend those tabs things on my Antec 902 to get it into the 5.25" space it was simple to install.

 

If you'd like some more pictures and my initial thoughts on the units then I've done a small write up on my blog.

http://blog.ktz.me/?p=145

 

 

 

 

 

[ironicbadger]

I've been threatening this post for a while. It details how to complete PCI passthrough on an Arch Linux, Xen 4.3 host from start to finish.

 

Good luck!

 

http://blog.ktz.me/?p=166

[ironicbadger]

Big update today.

 

Windows 8 VM with working VGA passthrough on Arch Linux with Xen 4.3. It's absolutely rock solid and just plain awesome. For anyone wondering why to pick Xen over anything else - here is your answer. Near bare metal performance for all VMs and gaming works flawlessly whilst watching a video from unRAID. All on the same box.

 

Here's the post. Let me know what you think below!

 

http://blog.ktz.me/?p=219

[Ford Prefect]

Big update today.

 

Windows 8 VM with working VGA passthrough on Arch Linux with Xen 4.3. It's absolutely rock solid and just plain awesome. For anyone wondering why to pick Xen over anything else - here is your answer. Near bare metal performance for all VMs and gaming works flawlessly whilst watching a video from unRAID. All on the same box.

 

Here's the post. Let me know what you think below!

 

http://blog.ktz.me/?p=219

 

..most interesting. I'll give it a deep look as I just started to prepare a similar config from my "spare" parts.

My old SM X8SIL-F is going to be repurposed to form my backup-array and a new Windoze Workstation.

The additional challenge is to keep the internal GPU active for use with IPMI.

I already allocated a PCIe-x1 ATI HD 6540 (the mobo only has x8/x4 slots, closed at their ends).

With the help of a XEN based setup, I was hoping to allocate the ATI (as secondary VGA on host) to the Win7 VM.

[ironicbadger]

Win 7 was a non starter for me. 8 was much easier.

 

Posted with my S4 on Tapatalk.

 

 

[Ford Prefect]

Archlinux and XEN4.3 was a no-go...finally couldn't get XEN to boot on the host.

 

Got it working based on this: http://forums.linuxmint.com/viewtopic.php?f=42&t=112013&sid=a6d42a17a76f04851a83b99b1bc12be4 ...on Linux Mint-13 and forcing XEN 4.1 not to upgrade (otherwise only one PCIe slot can be assigned to the VM (which means no sound, no mouse/keyboard).

 

Primary VGA of my X8SIL-F is still operational, running Linux Mint Dom0...nice.

 

[ironicbadger]

Archlinux and XEN4.3 was a no-go...finally couldn't get XEN to boot on the host.

 

Got it working based on this: http://forums.linuxmint.com/viewtopic.php?f=42&t=112013&sid=a6d42a17a76f04851a83b99b1bc12be4 ...on Linux Mint-13 and forcing XEN 4.1 not to upgrade (otherwise only one PCIe slot can be assigned to the VM (which means no sound, no mouse/keyboard).

 

Primary VGA of my X8SIL-F is still operational, running Linux Mint Dom0...nice.

 

How is this going?

 

I am finding my solution to be incredibly stable and reliable. I write this reply to you on my Win8 VM, which has morphed into my main workstation! I've broken down and sold my dedicated gaming PC and bought a laptop instead, which I've secretly been craving to do for sometime!

[Ford Prefect]

 

How is this going?

 

 

...quite simple...I did not get as far as you did. 

As I said, I couldn't get XEN to boot on that install.

Without that, no functional XEN+Dom0...no DomX for my Desktop-VM.

 

Either grub did not pick up the XEN kernel or XEN did not pick up the root disk of the dom0 to boot into this kernel.

I am not into ARCH and I tried to follow your blog-post as closely as possible.

As I wanted to test my physical setup I tried with another Howto and succeeded.

...and this with a Win7-VM  ...at least I know now that my combo is physically able to do what I need.

[smakovits]

Can you elaborate a bit on how you are running windows on the computer upstairs?  Are you simply using a console view of the VM through the server in the basement?  Essentially extending mouse keyboard and video?  Or something completely different like a wise terminal?

[ironicbadger]

Can you elaborate a bit on how you are running windows on the computer upstairs?  Are you simply using a console view of the VM through the server in the basement?  Essentially extending mouse keyboard and video?  Or something completely different like a wise terminal?

 

The Windows 8.1 (installed the update no problem) runs via a 30ft pair of cat6 to hdmi converter boxes / cables to my monitor upstairs without a hitch. The HDMI feed goes into a splitter in the cupboard under the stairs (the server cupboard) where one output goes to my AVR in the lounge for gaming, and the other goes into the cat6-hdmi extender. This is HDMI out from the ATI PCIe GPU. The HDMI out from my motherboard goes directly to my AVR as that runs XBMC natively under Arch.

 

I also use a cat5 - usb extender to a USB hub upstairs in order to connect my keyboard and mouse.

 

This setup is working flawlessly in conjunction with the reboot scripts I posted before. It's been well over a month now and I couldn't be happier with the performance of the whole setup. It is a pain when you need to reboot the host I guess, but I just reach for my phone and load up my SSH client and do it all from there.

[ironicbadger]

Can a moderator move this thread to the right place please.

 

Sent from my Nexus 5 using Tapatalk

 

 

[ScienTiFIC]

Can a moderator move this thread to the right place please.

 

Nice Writeup.. I was looking at ESXi But now you got me looking into Xen. So would Xen be a better solution for my AMD system? I'm Looking to get into virtualization but have a old system and wana take it to its grave. what would be the lowest resource hog for my system (Asus A8N-E Deluxe W/Athalon X2 3800+) ESXi or Xen? All I want is VM for Unraid and a VM for OS W/Plex,subsonic,etc.