New virtualization-capable unraid server

coppit

By coppit
in Unraid Compulsive Design

Recommended Posts

coppit Enthusiast

coppit

Posted
Posted

With the upcoming support for virtualization, I thought it would be a good time to upgrade my system. I'll do a couple of posts -- one covering the hardware and one covering the software. My goals were to (1) improve performance, (2) move services out of the unraid host, and (3) move some server-like services from my Windows machine to the unraid server.

 

Parts list (about $1100)

 

 

I'm super-happy with the case. The quality and features are top-notch. My only complaint is that the back-side SSD locations require that you remove the motherboard to attach the SSDs using the bottom-mount screw locations. They should have made some bent tabs for screwing in the SSDs from the side. See pic below. Well, okay, I have one more complaint... It's too wide to fit in my computer table. :)

 

The motherboard has temperature-based fan speed control for the CPU and one case fan. (The other two fan headers are fixed speed ones.) So I skipped using the case's manual fan speed buttons, and also skipped using the water cooler's software-based control. Speaking of water cooling, I don't know why I didn't do this sooner. It relocates the small, loud fan in the middle of your computer to a large, quiet one on the case. No more worrying about cables getting caught in your cpu fan.

 

The SAS cables work fine but are pretty cheaply made. The biggest requirement is that they lock -- too many times I've had a loose connection cause a drive to drop off the bus.

 

I bought 2 SSDs, with the intent to use one for a cache drive and the other for hosting VMs.

 

I reused a 750 watt power supply, 3 or so 4-pin molex to SATA power cable adapters, a computer cable lock, a DVD drive, a 3.5-to-5.25 hard drive adapter, and all my hard disks.

 

Click images for full-size.

 

Front view

 

width=200http://coppit.org/temp/unraid 1.jpg[/img]

 

Back view

 

width=200http://coppit.org/temp/unraid 2.jpg[/img]

 

Here you can see my unraid USB drive, and in the upper right my drilling to fit a computer lock. I figure if the weight doesn't prevent a thief from stealing the computer, maybe the flimsy lock might. :)

 

Open case view

 

width=200http://coppit.org/temp/unraid 3.jpg[/img]

 

The case comes with the 140mm fan you see, plus another at the front bottom. The front also has a spot for another fan, which I installed. Actually, I swapped the water cooler fan with that one because the power cord was longer. In hindsight I really don't think the 4th fan is really necessary -- temps are pretty low.

 

The case has a bottom-side filtered air intake for the power supply, but I went ahead and mounted the power supply upside down. I was more concerned about setting the computer on carpet and having the power supply overheat. The case temp isn't a problem with 4 case fans.

 

You can't really see it, but the water cooler just barely clears the memory card levers. There's no way to mount this water cooler on the other top vent because it hits the motherboard header. Like the case fans the water cooler fan is 140mm -- super-quiet.

 

This picture shows the DVD drive in the top bay being connected to one of the gray motherboard SATA headers. This killed my parity check performance, so I disconnected it for now. I plan on plugging it into one of the high-speed slots below, as the sticker suggests. (I didn't think it would matter for a DVD drive.)

 

Back of motherboard view

 

width=200http://coppit.org/temp/unraid 4.jpg[/img]

 

Here you can see where I cinched up the unused case fan speed button cables. My solution for the SSD mount problem was simply double-sided mounting tape -- you can see the tabs sticking out. If the water cooler mount plate looks crooked, it is. I took me a while to figure out that's the way it's supposed to be. :)

 

VM and Software Setup

 

In the Unraid dom0, I don't have any plugins and very minimal stuff in my /boot/config/go. I'll attach it. It installs the trolley package manager and vim. I have a disk dedicated to a Time Machine share for backing up my Mac, and another share for backing up Windows using simple destination folders. One trick is to wait for unraid to mount the shares before starting the VMs, to avoid any problems with them accessing the shares.

 

I followed Tom's notes on the 6.0 beta3 thread to create an Ubuntu VM. Inside I'm running transmission and flexget. Since I don't need much for this VM, it only gets one CPU core and 512MB of RAM. Transmission is storing files on the cache drive, mounted via fstab. I also set up an rsync daemon so that I could export music and photos from my Mac's iPhoto and iTunes to the array, in a standard format, and a cron job to delete old Windows backups. (Apparently the Windows backup service just fills up the disk and then stops doing backups... Sigh.)

 

I also set up a Windows VM. In this VM I'm running SageTV PVR software. Two MPEG2 streams are coming in over the NIC from my HDHomeRun tuner, and being stored on a dedicated disk that is outside the Unraid array. Originally it was a share from inside the array, but I noticed that file delete operations were pretty slow. Plus I wanted to give the VM a 3.5 GB disk, and my Unraid array is based off of a 2GB parity drive. I allocated 4096MB of RAM, although it seems like the VM could get away with just 2-3GB. I also allocated 8 virtual CPUs. The reason is that I have a scheduled task running HandBrake to compress the mpeg2 streams to h264.

 

I also have a couple of Logitech security cameras at the front and back doors, sending the streams to the server software running in this VM. Finally, I'm running Air Video server so that my media library and TV recordings are available on i-devices inside the home as well as outside.

 

I went ahead and installed the paravirtualized drivers for Windows. Besides better performance, they have a hook that allows you to run "xl shutdown windows7" from dom0 and have the VM shut itself down gracefully.

 

I'll attach my xen config files. I didn't specify the MAC address and let Xen generate one. Once the VM was up, I got on it and ran ifconfig/ipconfig to get the MAC address. Then I updated the config file to set it permanently, and updated my DHCP config to reserve IPs for these VMs. You'll also see the assigned hard drive in the Windows config. Add "localtime=1" to sync the time zones up.

 

Performance

 

With this setup and all the disks, I get about 55 MB/s parity checks. I have one 5400 RPM drive that's probably slowing me down. I don't notice any speed drop when the CPU cores are busy doing encoding.

 

The SSD cache disk made a huge difference. Copying a single large file into the array tops out at 90 MB/s, up from 14 MB/s in my old system.

 

With all the CPU cores burning through a HandBrake encode, and all the disks churning on a parity check, hard drive temps are a cool 33C. The fans don't speed up.

 

TODO

 

Figure out how to hide the VM SSD and Windows dedicated HDD from Unraid. I worry that I'm going to accidentally assign one of them during some disk swap operation...

 

Figure out how to get the unraid UI to shut down VMs. This might require help from Tom... I don't know if there's a way to hook into that button. (Seems like the process is backwards... The UI should be calling a powerdown script that we can modify, not the other way around.)

 

Transmission's RPC interface isn't always accessible after the VM boots. Restarting it fixes the problem -- not sure what's up with that. Edit: It turns out that the daemon was starting before the share was mounted. This page gave a hint: modify the upstart config to wait for eth0 and the mount point. It's also possible that the real problem was that unraid hadn't started the share yet -- see my comments on the "go" script above.

 

I also bought this dual-ethernet card with the intent of dedicating one NIC to a linux VM and the other to a Windows VM. Unraid wouldn't boot up reliably with the onboard NIC, so I pulled it out for now. I can probably use PCI passthrough as described in this post, although I haven't tried it.

 

I also have a gaming PC that I could convert into a VM, using PCI passthrough for the graphics card. Maybe I'll do that later.

 

After 6.0 beta4 adds CD/DVD support pass the DVD drive through to the Windows VM so that SageTV can play DVDs.

windows7.cfg

ubuntu.cfg

go.txt

Link to comment
  • 2 years later...
coppit Enthusiast

coppit

Posted
  • Author
Posted

I thought I would update this thread since I've extended this build a bit.

 

For a while I was running a Xen VM as my PVR. But once KVM became preferred, I switched over. Then I doubled down on virtualization, adding two GeForce GTX 960 so that I could set up two gaming VMs.

 

Here's the new hardware:

 

  • Two EVGA GeForce GTX 960 2GB GAMING ACX 2.0+
  • AOC 2460G5. This monitor is 3D Vision capable, or g-sync can be turned on
  • 2nd monitor for gaming
  • Cheapo D-Sub monitor for the onboard graphics to manage unraid as needed
  • Two IO Crest 4 Port SATA III PCI-e 2.0 x1 Controller Card Marvell Non-Raid with Low Profile Bracket SI-PEX40064. Sadly, I had to replace the SAS cards because I needed the slots for the GPUs.
  • Corsair Force LE 2.5" 960GB SATA III TLC Internal Solid State Drive (SSD) CSSD-F960GBLEB. This replaces the two Samsung 840 EVO SSDs

 

I'm running Windows 10 in the two gaming VMs. KVM XML files attached. Installation of the OS with virtio driver support was challenging. Be sure to use the temporary disk method described in the wiki, and also apply the various post-install tuning steps.

 

Initially, I had a lot of trouble with audio crackling and abysmal gaming performance. I passed through the USB3 controllers, which seemed to help. (See research on DPC latency, latencymon, etc.) After lots of fiddling, I noticed that only one CPU config got rid of the audio problems and gave good enough performance. Lots of people advise using isolcpus in the kernel boot params, and computing the hyperthread-paired virtual CPUs to dedicate to the VM. Oddly, the only thing that worked for me is to pass half of each hyperthreaded pair to the VM. So since my pairs are 0 and 4, 1 and 5, 2 and 6, and 3 and 7, I pass vCPUs 1,2,3 to one VM, and vCPUs 5,6,7 to the other gaming VM.

 

For the GPUs to pass through, I had to set "pcie_acs_override=downstream" in the kernel params for this motherboard.

 

For the PVM VM, I pass vCPUs 4,5. I leave vCPU 0 for UNRAID and docker containers. For its OS disk, I previously was passing through an SSD. Since the new standard is to have a "domains" folder on the cache drive with .img files, I went with that. That's partly why I replaced the two smaller SSDs with one bigger one. I'm still passing through a 1 TB hard drive for the PVR files.

 

I may revisit this CPU pairing issue again later. Recently at work (and on Xen), we noticed that even when passing paired vCPUs to the VM, the OS wouldn't detect them as paired, and gaming performance would suffer. Maybe something similar is happening here? Also, I feel like there are CPU inefficiencies with Windows VM in KVM. When the VMs report no CPU usage, in the UNRAID host it reports 30% cpu usage. I fiddled with all kinds of things but couldn't resolve this issue.

 

With 2 gaming VMs, a PVR VM, and 7 docker containers (including one with 4 Minecraft servers), this machine is stretched to the limit. I investigated buying better hardware, but honestly I was disappointed that CPUs haven't advanced more since I did my original build. I could go server class, but I like this motherboard and case a lot.

 

Anyway, the current state is that we can play OverWatch on two VMs at the same time. We can't really have anything CPU-intensive running in the background, and occasionally the computers freeze for a few seconds. But generally it's a smooth experience.

gaming1.xml.txt

pvr.xml.txt

gaming2.xml.txt

Link to comment

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.

Guest
Reply to this topic...

×   Pasted as rich text.   Restore formatting

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.

×
Go to topic listing