Excelsior - Plugins, Dockers and VM's, Oh my

[bnevets27]

Excelsior is a Latin adjective meaning "higher".

 

See second post for initial purchase and plans. This server started life as a server from TAMS (popular dealer for second hand servers, master thread in over at avsforums) As time has gone on and unRAID has changed so has this build.

 

Current specs:

Chassis: AIC 24 Bay - Modified to house the following

Motherboard: Supermicro X7DWN+

Processors: Intel Xeon L5420 @ 2.50GHz X2

Heatsinks: Apple Mac Pro A1186 (w/drilled out mounting legs) X2

RAM: 40GB (2GBx12) + (4GBx4) Fully Buffered ECC DDR2 667 MHz

SATA Expansion Card(s): SAT2-MV8 Raid Controller Cards X2

IPMI Card w/KVM: Supermicro AOC-SIMSO+

Power Supply: SeaSonic X-1250

unRAID USB: Kingston MobileLite G3 w/8GB microSD running latest v6

HVM: Enabled

IOMMU: Enabled

 

 

Modifications:

 

Case:

This server of course came stock with 3 power supplies that are very loud. The AIC case is a bit more difficult to modify, compared to the other server TAMS was selling, the supermicro, due to the fact the power supplies are on the bottom. There are a few ways to go about getting a ATX power supply into to case but for various reasons I decided to "roll my own" solution.

 

 

I need to get the following to fit inside the case:

Enhanced Extended ATX motherboard

Apple Mac Pro heatsink

ATX power supply - SeaSonic x-1250

 

I also wanted it to be quiet, I knew it wouldn't be silent but wanted it to be tolerable to be in the same room and of course it needed to keep everything cool under load.

 

To accomplish the above I decided I only wanted to use 120mm fans and no CPU fan would be ideal. This of course is why I had purchased the apple heat sink, though I had no idea if it was going to be too big and if it would keep the CPU cool. I had looked up as much info as possible before hand but I didn't find anyone else that had tried to use it in this configuration.

 

First I removed the back plate, then the motherboard tray, and finally the power supplies, which left just the sides and hard drive backplane.

 

Thankfully AIC made everything very modular and everything is screwed together. This allowed me to remove the I/O port plate and PCI bracket piece from the rest of the backplate and reuse it.

 

After a couple different ideas and mock ups, I ended up with two 120mm fans. Which is what I wanted to do.

 

Mocking up

I started with an non modular PSU, but quickly realized I didn't want, or have room for, the extra cables.

 

I also cut down the motherboard tray so it would sit flat on the bottom and fit to one side. I did this so I didn't need to deal with stand offs.

 

Cut up backplate/IO bracket

 

Re-enforcing and re-attaching backplate to IO bracket

 

The heat sink:

Can't remember how I stumbled upon the heat sink but I don't think there is many passive heat sinks for socket 771. Anyhow, everything I could find about heat sink made it sound like it would at least fit the socket and should cool the chip. Though it is tall so I still wasn't sure if it would fit the 4U case. Fast forward to the installation of the heat sink and I found out apple, being typical apple, decided to use a non standard screw for attaching the heat sink to the motherboard. Well not the motherboard but the standoffs under the motherboard is what the heat sink attaches to. I decided to get some 1in 6-32 screws and drill out the heat sink to allow the bolts to pass through the M3 threads inside the legs of the heat sink.

 

 

Things still to do on the case:

The heat sinks are about a 1/4in too tall. But the cover does kind of fit, it just bends a bit. Haven't decided how to fit it but I will make it fit better.

Fan wall needs mount brackets

Various other final touches

 

PSU:

Made all custom cables except the main 24+4 motherboard connector. (Don't tell SeaSonic)

PSU modular connector to molex adapters

 

Backplane power cables, later shorten the connections between plugs

 

My cable on the left, stock right. Made the cable shorter and combined the 8pin and 4 pin MB power plugs into one cable/PSU plug

 

 

Motherboard:

I only have PCIe x8 slots but I want to use a GPU for a VM. So I cut the back out of one of the PCIe slots.

 

Big video card fits

 

 

Reused one of the squirrel cage fans (they are quite and have good air flow) over the PCI slots just because there was room. Its removable if it gets in the way.

 

I was short 2 long sata cables which is why there are two running over the top of the fans. There is still a bunch of little stuff to clean up. Will post finished pictures eventually.

 

 

 

Now for the plan:

 

I have an interest in a lot of the dockers that are available. I am also planning on running a VM and seeing how that can be utilized any may expand to more.

 

 

 

Plugins:

Installed:

Community Applications

Dynamix Plugins:

System Temp

System Stats

System Info

 

Preclear

IPMItool

Server Layout

Powerdown Package (2.18)

Enhanced Log Viewer

 

To install:

cache dirs

 

 

 

Dockers:

CouchPotato - linuxserver.io - Installed

Sabnzbd - linuxserver.io - Installed

SickRage - linuxserver.io - Installed

Plex - linuxserver.io - Installed

PlexPy - linuxserver.io - Installed

Hydra - linuxserver.io - Installed

Apache - linuxserver.io - Installed

Plex Request - aptalca - Installed

Dolphin - aptalca - Installed

 

ownCloud

synthing/BTSync/Sync - not sure which one yet

CrashPlan

Deluge - or any other BT client

 

One of these two:

Emby

Kodi - Headless w/mariadb

(I have been running xbmc headless with mysql on my current v5 unRAID server)

 

Dockers I may or may not keep, need to test them:

SNMP

WebSync - not going to use this probably, sad that this isn't as functional as it needs to be

darkstat

mochad

Krusader - use this or dolphin

Filebot

BrewPi

nzbget - never tried it, will try and see if it suits me better then sabnzbd

Sonarr - never tried it, will try and see if it suits me better then sickrage

-------------

BackupSQL - if i use kodi headless and mariadb

Amule

FlexGet

Headphones - w/musicbrainz docker?

JDownlaoder2

MovieGrabber

Calibre

Google Music Manager

H5AI

IRC Client

MakeMKV

HTPC Manager

Maraschino

Mosh

sonerezh

 

Configurations I still need to do:

Link current "production" server (which will become a backup/archive) to new server via direct gigabit connection

 

setup rsync scripts:

backup important files from main server

backup cache appdata and docker.img to array (caveat, I've copied the docker.img off the cache then back on and get the "please recreate the docker image (i'll paste the actual error here later))

backup usb flash to array

 

Script to print all file names to text file

 

Improve Apache configuration:

Add more users

Force user to log in to see index page?

Create nice index page with links to services

 

Move away from Apache to ngix?

 

run: unraid-tunables-tester.sh

 

Test UPS

 

Setup windows (10?) VM

 

 

 

 

Current hang ups:

 

 

 

 

Guides I used and various other info:

 

 

[bnevets27]

The following info is now deprecated with the introduction of v6 of unRAID. Updated specs and plans are in the first post.

 

I wanted a second server to start playing with and I was able to snag a server from TAMS. After seeing all this virtualization around the forums for awhile and seeing what flexibility it allows I was definitely interested in getting into it. Unfortunately the hardware that came with my TAMS server does not support VT-d. My hope was to replace the motherboard with a motherboard that supported VT-d but still was able to use all the same components.

 

Warning: All photos are taking with my phone in terrible lighting and in haste, sorry.

 

 

Here is the server specifications as received from TAMS:

Chassis: AIC 24 Bay

Motherboard: Supermicro X7DBE-X

Processor: Intel Xeon E5120 - 1.86Ghz Dual-Core

RAM: 8GB (4x2GB) Fully Buffered DDR2

SATA Expansion Card(s): 3 x SAT2-MV8 Raid Controller Cards

IPMI Card: Kira 100 SIMLC

Power Supplies: 3 x Emacs M1W-6500P

unRAID USB: Kingston MobileLite G3 w/8GB microSD

 

Upgrades I bought, all from ebay:

Motherboard: Supermicro X7DWN+

IPMI Card w/KVM: Supermicro AOC-SIMSO+

CPU: Intel Xeon L5420 - 2.50GHz Quad-Core

 

There is some evidence/talk about ESXi be successfully run on the X7DWN+ which is outlined in this post. Which is why I chose it, also because it could be found for cheap on ebay. My first worry was if it wouldn't fit into the case as the X7DBE-X is an extended ATX and the X7DWN+ is an Enhanced Extended ATX which is 1 inch wider. Luck would have it that the X7DWN+ does fit into the AIC case. One of the side fans and one of the standoffs has to be removed (this is really important as it would sit on the pins of the motherboard and cause a short). Most of the rest of the standoffs line up and even better the I/O panel was an exact fit.

 

X7DWN+ beside the stock X7DBE-X

 

Why the fan had to be removed

All wired up (AOC-SIMSO+ and L5420 not installed yet)

 

There is a few other things to keep in mind about the X7DWN+ there is only two PCI-X slots so only two of the three SAT2-MV8 cards can be used and therefore your limited to 18 drives using the existing components. The positive that is has three PCI-e slots so you can use newer controller cards (ie the M1015), the X7DBE-X has none. The X7DWN+ also has more memory slots and the IPMI w/KVM card is cheaper.

 

The server booted and ran unraid fine on both the X7DBE-X and the X7DWN+. I wasn't able to do any extensive testing as I don't have a lot of extra drives laying around. I had 2 data and 1 parity drives, ran parity checks a few times and nothing odd came up.

 

I purchased a pro key for my Kingston MobileLite G3, it took a little be of trial and error to get it to boot properly on the X7DWN+ and outlined in this thread.

 

As many people with the TAMS servers have mentioned this server is LOUD. My server is in the basement so it's less of an issue but it's something that should not be over looked. I currently only have one PSU plugged in which helps a little with noise, the warning buzzer has to be reset every time you power on the server if only running one PSU.  The two exhaust fans can actually be louder then the PSU fans. They can be throttled in the BIOS with preset options, I chose "workstation", but they are still loud. I think the PSU fans are thermally throttled and can be surprisingly quite when cold, unplug the rear fans and its actually very quite for a server but its not practical to run like this as it doesn't draw any air through the case. This isn't a big deal for me at the moment but I make make some modifications in the future to quite the server down.

 

Now the question is which virtualization software should I use, ESXi or Xen/KVM? I looks like Xen/KVM is getting a lot of attention on the forum as of late. But virtualization is a whole new world for me. There also seems to be a few XEN/KVM options (thanks to grumpybutfun) .

 

[GUIDE] Virtualizing unRAID in Xen / KVM on Arch Linux

[GUIDE] Virtualizing unRAID in KVM on openSUSE 13.1 (64Bit) <--- Completed 12/19

[GUIDE] Virtualizing unRAID in XenServer 6.2 - Updated to 5.0.4 New

 

Any advice on which software would be best suited for me? Current plans are just to install unraid and a linux distro to run all the programs currently running as pluggins. I would also like a few VM's to play around with on linux and windows. I also might have some interest in running XBMC on one of them, if that matters. Currently I have the following plugins installed:

APC UPS

MySQL

Plex Media Server

Subsonic

XBMC SaLUd

Couchpotato

Headphones

SABnzbd

Sickbeard

Transmission

 

I'm hoping to find out if hardware passthrough works correctly with the X7DWN+ as it looks like it would be a cheap and easy upgrade to get hardware passthrough on TAMS servers (Intel versions only of course).

 

 

[bnevets27]

Reserved for future use

[bnevets27]

Reserved for future use

[bobmagnuson]

I'm looking forward to the results of your build.  I've got the same TAM setup and was disappointed to see that VT-d wasn't supported with the motherboard/cpu combo I've got.

[bnevets27]

Followed this guide: [GUIDE] Virtualizing unRAID in KVM on openSUSE 13.1 (64Bit) <--- Completed 12/19

 

And can report hardware passthough works! I'll have to iron out some details and post some screen shots in the coming days. Then when I get a change I'm going to try out ESXi.

 

Only hiccups I found were:

1. During the openSUSE install: non-existent floppy drive enabled in BIOS causes hang while searching for partitions

2. Had to add this to the bootloader:

intel_iommu=on

3. Had to add this:

options kvm allow_unsafe_assigned_interrupts=1

to /etc/modprobe.d/kvm.conf (create this file, if it does not exist)

 

I was able to pass just the USB device itself through. I installed openSUSE on a 16GB usb stick.

 

 

 

[bobmagnuson]

Great News!!!!  Keep the info coming.

[bnevets27]

Well 2 years later I finally ripped the server apart to rebuild it. Currently testing and working on configuring all the different programs. It will then be used as my primary server and will retire my current one to a backup.