Jump to content
vl1969

My New Build using Supermicro Server from TAMs

42 posts in this topic Last Reply

Recommended Posts

this is my new build that I have been working on for last couple of weeks.

ever since I got a server from TAMs (as my subject denotes).

 

I did run into some issues now and then.

 

if you are interested do a search on "VL1969" to see the threads I used or here are some of the links to relevant discussions.

 

the main help for  setting up UnRaid with XenServer is here "http://lime-technology.com/forum/index.php?topic=28665.msg255295#msg255295"

 

thanks ddeeds for that.

http://lime-technology.com/forum/index.php?topic=18610.msg241540#msg241540

http://lime-technology.com/forum/index.php?topic=28627.msg254899#msg254899

 

this build is still in early stages as I had to post pone all the works for knee surgery.

but now it slowly back on track.

 

hope info here helps someone. and make it easier to build similar setup.

 

======New Build (In Progress)======================

With Respect I thank Ddeeds as I have my first UnRaid VM on XenServer

booted and running

 

well the Virtualization did not go as expected.

The hardware at hand has very limited support for virtualization

And I was only able to make the VM work under Ubuntu 12.4 L.. whatever Server and Xen 4.2

ESXi and XenServer option do not work.

And Ubuntu 13.4 and Xen 4.3 also have problems

 

====== Hardware ====================================

Supermicro SC846 -- 24 Bay (from TAMs)

Motherboard: H8DME-2

Procs: AMD Opteron Quad Core 2346HE @ 1.8GHz

Ram: 8 GB using 4x2GB

(MB is capable of 64GB per Processor using 8GB modules for a total of 128GB

  but real number (unless you do have an unlimited budget for this as 8GB modules are prohibitively expensive) is more like 32GB per Proc. using 4GB modules for 64GB total

 

IPMI Card: Kira 100

3 SAT2-MV8 PCI-X Raid cards

2 Ablecom PWS-902-IR Redundant PSU

Network : 2 on-board Gigabit ports

 

ON A SIDE NOTE : if you are planning to get an AMD based setup from TAMs and think you might update the CPUs

go for the dual core setup, save $50 bucks.

 

Main Changes and additions:

1. Updated BIOS to v3.5 

2. Replaced Original AMD Opteron Quad Core 2346HE @ 1.8GHz with 2 AMD Opteron Hex Core 2431 @ 2.4Ghz for total of 12 cores

3. Added RAM for the total of 24GB DDR2 PC-5300 @ 667mHz  4x2GB(original memory) + 4x4GB  still 8 empty slots

4. Added 1 Intel Pro Dual port PCI-e card for total of 4 ports

(add on card is to be used as a pass-through with pfSence or Untangle VM router/firewall)

 

5. some people do a PSU mod, gut the PSU  and replace the whole setup with standard ATX silent PSU to make this a totally silent setup.

but if your server is in the basement somewhere and you can tolerate some noise,  try what I did.

 

5a. Remove 1 PSU form the chasses, I just pulled one a quarter way out to disconnect it from the board.

if you are doing the PSU mod you lose the redundancy anyway so this is much simpler.

it is noisy but very tolerable with only one of the unit working. I can barely hear it from the basement.

5b. replaced original back wall (not the fan wall in the middle of the chassis) 80mm fans with quiet models (nothing special here just newer fans I found in local MicroCenter for $3 each)

the original server level fans are very loud.

FYI: the new fans are thinner but will still fit in the mounting cages just fine.

however you might not be able to use the connectors on the cage itself and will need to run the wires directly to the MB.

I have played with the cages and manage to connect the new fans to the provided sockets,

but it is a pain and I just did it for the why the heck not factor. nothing else.

 

    5c. removed original fan wall with 80mm fans(too loud) and replaced  it with 3 120mm fans (found it in local MicorCenter for $10 each )

the 120mm came with one side fan guard, but for safty I added second guard to other side, also in local MC for $2 each.

if you are not doing the PSU mod(replacing the original redandunt PSUs with silent ATX kind)

you do not even need the fan wall. 3 120mm fans stringed together with zipties fit nice and snugly in place of the original fan wall

just need to find a way of holding it there.

will post the pics later but, I used a mounting bracket from original fan wall on right side(looking on the case from the front)

thread it between the blade guard and the fan body and a zip tie through the screw holes on the other side.

I have a suspect that if you are doing the PSU mod  you can use the 140mm fans in the same way without fan wall.

 

5d.  since I was replacing CPUs I also got new passive HeatSinks coolers on ebay. this one are usually used in 1U chassis thus no fans.

    NOT GOOD!  this coolers are not enough thus I have spent 4 days trying to start the server only for it to crash with overheat error.

which also took me about 2 days to figure out (there are number of manuals on SM website for this MB just need to read through them)

the server would beep continually and shut down.

so, since all was already installed and done, just mount an 80mm fan to the cooler and all is well.

this also makes the setup quieter as CPU cooler fan it came with is also very loud.

have been running non-stop for 2 weeks already while I test the ESXi,Xen/XCP  XenServer installs.

and tried  fruitlessly in booting unRaid in VM on all of this platforms.

 

HDD:

at the moment I only have 1 - 1TB HDD attached to MB SATA port and mounted inside the chassis

(the case have a space for one HDD but it expected to be a PATA drive with 90 degree  power connector and flat PATA connector.

had to Gerry-rig the mounting cage sideway for SATA drive to work.)

with XenServer 6.2 loaded and configured on it.

with Ubuntu Server 12.04 LTS VM configured running Transmission for now.

I think I want to try booting the XenServer from USB though (have a verbatium toughstick @ 16GB) will see if it works out.

 

 

ALSO: apparently this case supports a slim CD/DVD drive accessible from the back of the case (right next to the PSU)

but you will need special SM mounting hardware if you want to do that.

 

 

I finally brock down and did the PSU mod.

The original PSUs are just way too loud.

Lucky for me(if you call it lucky) I had older PSU from my desktop laying around. It got replaced during troubleshooting of mystery BSD on my main desktop as possible suspect (it wasn’t, I had a bad stick of RAM in it.)

So I just swap my 760 watt PSU from my UnRaid server and install it into this new case.

Unfortunately this PSU is not modular so the cables are a mess. It works though,  and now I have some peace and quiet.(once I replace the fan on my NetGear switch that is  :-P  )

 

====== Software as Projected today. ====================================

Citrix XenServer 6.2 as Host

 

Guest VMs: (in order of importance as much as possible, see notes next to each for progress)

 

unRaid: 5.0rc16c in VM (this is what started all this madness :-) and yet I haven't even started the setup.)

if you are looking for help check this thread from [ddeeds]  out

"http://lime-technology.com/forum/index.php?topic=28665.0"

 

Ubuntu Server 12.04 LTS (installed and running)

SABNZBD

Sickbeard

CouchPotato

Transmission (this is up and running already)

 

Windows 7 or 8 for Transcoding movies into MKV (if I can find the way to do it in Linux as easy as I do it now in Windows this might change)

 

Linux Mint or Ubuntu Desktop (or both) for fun and learning.

 

FOG/PXE boot server

 

pfSence and/or Untangle Super router/firewall  to replace my cablevision NetGear router.

 

fullsystem.jpg.a3ba8e239d09de78f93412bde970e424.jpg

Fanwall1.jpg.66b638b045315d549e5934258be7ed94.jpg

fanwall2.jpg.e1271e966b9ded603b60686868ef4399.jpg

Share this post


Link to post

Really digging your setup, keep posting updates for us. You said the passive heat sinks were not enough to cool it?

Share this post


Link to post

Well I got a set of HS that said it was for use with 1U chassis.  It is a really low profile HS and I bet not for use with my cpu , probably for slower less powerful units. It is good hs. All Cooper and all but may not have enough umphpff for 2431 cpu. Works  if you ad fans though.  I just want to have less fans but setled for having quite fans instead.

 

Sent from my SGH-T889 using Tapatalk 4 Beta

 

 

Share this post


Link to post

I am so Pissed right now  >:(

 

here it is,  a perfect source of info I have been searching for for the last 3 weeks,

(and I am referring to the"Virtualizing unRAID in Xen / XenServer 6.2" by ddeeds)

and I have absolutely no time to sit down and try if it works. it's like an OCD, I can barely wait see if it works.

 

PS>> sorry just venting my frustration.

Share this post


Link to post

I have a total of 5 of these TAM machines (3 Intel, and 2 AMD), so I follow this with interest.  One of them is running bare metal unRaid at present and the other 4 are for future projects.  It would be good to be able to do something besides gutting them, replacing MB and CPU and memory to support virtualization.  Even if that was all I could do with them, they are still worth it just for parts.

 

Tell us the good and the bad of your project....

 

Share this post


Link to post

well tr0910 , I am not sure what chassis setup you have,

but you should try the Xen/XenServer on them.

 

I am still working out the kinks with passthrough stuff, but the hardware is still quiet good for  setting up Virtual Lab.

 

if you want a bare metal unRaid for Pro License setup (as in 24 drives  total) there is no reason to gut them

other than the fact of them being too noisy. which you can deal with too with a minimum modifications(well except for PSUs ).

 

so far I like the chassis very much and if I can setup my pci pass-though it would be even better :-)

but pass-though is needed only for power saving and to prolong the HDD life span.

for me currently it might not be the diving factor. YMMV

 

 

 

Share this post


Link to post

But you should try the Xen/XenServer on them.  So far I like the chassis very much and if I can setup my pci pass-though it would be even better :-)

but pass-though is needed only for power saving and to prolong the HDD life span. for me currently it might not be the diving factor. YMMV

 

Hardware pass-through is required to allow drives to spin down ??  That's the only limitation so far??  Performance is good otherwise??

 

What about for uses other than unRaid?

Share this post


Link to post

 

Hardware pass-through is required to allow drives to spin down ??  That's the only limitation so far??  Performance is good otherwise??

 

What about for uses other than unRaid?

 

I am not sure what your needs are.

 

my setup is for home. I started with simple unraid box but the want to get a good case.

also want to go Virtual route to have some dedicated configs for special needs.

like Torrent download to be done off main PC

SickBeard+coachpotato+snzdb to be done on  dedicated setup.

a PC image and backup server to save time when my PC breaks,

maybe an ftp server al-la dropbox so I can give my friends and family access to some of my files safely.

some VMs to play with new setups and development (I am a .NET dev so a nice stable  test env is nice to have.)

 

nothing overly optimistic :-)

 

 

 

 

 

well, if you want for unraid to be able to control the drive spin up/spin down

you need to pass through the controller into VM, as far as I know this is the only option.

 

 

 

I just recently setup the chassis in more or less stable config using XenServer 6.2  and still playing with it.

The XS setup went very smooth and pain free.

I used the IPMI from my main PC all the way. as in I attached ISO as a CD drive via IPMI (keep in mind that my ISOs are on the current unraid share) so it vent from Unraid-Shared to my PC to IPMI ,boot ed the server, it went into install no problem.

whole install took about 30 min total. most of which POST/BOOT and  was answering the questions. after that I left to get a coffee and came back to a working Server.

I have create Ubuntu 12.04 LSI server VM same way, attached ISO to IPMI

xenserver found ed and passed it to new VM as DVD drive and I was running in 15 min.

installed transmission in 2 min, and downloaded 400GB ISO

all very fast

 

I am not a Linux guy , and simply do not know how to test the performance of this setup.

other than , if it does what I want for it to do and don't give me any issues I am happy.

 

using ddeeds help, I had the unraid 5.0rc16c VM up and running in  5 min (no drives) but it booted, and gave me the webGUI with all the info. installed simplefutures or something plug-in

which updated the GUI and added some other futures by simply accessing the flash drive fro mthe webGUI and placing the files onto it. and rebooting VM.

it started very fast.

 

I am not sure how it works with RDM, but consensus on this board is that RDM is to confusing to be used with large number of drives. (at least that what I got from various post so far)

 

I am not doing anything serious yet because I want to make sure that my plan in moving my unraid to VM can be done in way I planed it. after that I will move onto next item on the list.

if not I will reassess  my plan.

Share this post


Link to post

Sorry ddeds, could not fully test it. As of now it is a no go.  But there is something I want to try before I give up.if all go well I might be able to try it tonight.

 

Sent from my SGH-T889 using Tapatalk 4 Beta

 

 

Share this post


Link to post

Yep, that what I was thinking to do.  Great minds think a like ;-)

 

Sent from my SGH-T889 using Tapatalk 4 Beta

 

 

Share this post


Link to post

OK I have tried everything so far nothing works.

I am attaching the files you asked for

 

the zip file contains

extlinux.conf  --> config file from XenServer

CommandOutput.txt  --> all CLI commands and output

lspciOut --> output of lspci command

and a screen print of the error from XenCenter

XenServerFiles.zip

Share this post


Link to post

The zio

P

 

 

 

 

 

The zip. File attched to previous post have all that in commandoutput.txt.

However, the xl pci list assignabke decices show the controller, do show the

 

Sent from my SGH-T889 using Tapatalk 4 Beta

 

 

I tried to boot from your image directly, this morning,

the server booted and I ca nsee both drives on my system

the 2TB attached to the SAS controller and 1TB attached to SATA port on MB

attached is the pic.

Clipboard01.jpg.8d387d1cd17eabdd5c069b68714cb8df.jpg

Share this post


Link to post

Ps to my last post,  I did not try the stock biulld directly. Sorry did nit think of it. Will do that asap.

 

Sent from my SGH-T889 using Tapatalk 4 Beta

 

 

Share this post


Link to post

ddeeds, after some light reading I need to ask,

when I create unraid  DOMU as per instruction, is it a PV guest or HVM guest.

I have found several references on the web that say that for HVM guest you need the full IOMMU support of the hardware (MB and CPU) for PCI passthrough

but not for PV guest. maybe that's it.

my MB is not fully supports IOMMU (I knew that but was houping that XEN will work anyway) so if the unraid VM is HVM it might be it.

 

 

Share this post


Link to post

I am quiet sure any Linux distro have drivers for it. as it is an old controller that is very common in the server market.

 

will do the test As soon as I can.

 

also maybe I can do this changes for unraid VM ?

I can remove the passthorugh with param remove command

(it boots fine if I do that. )

do the changes and than add the passthrough back again.

 

 

Share this post


Link to post

I am quiet sure any Linux distro have drivers for it. as it is an old controller that is very common in the server market.

 

will do the test As soon as I can.

 

also maybe I can do this changes for unraid VM ?

I can remove the passthorugh with param remove command

(it boots fine if I do that. )

do the changes and than add the passthrough back again.

 

If the test works in Ubuntu, we can edit the syslinux.cfg on your unRAID USB Stick and add the parameters above and see if it works. I am just not sure if the XenServer VM bootloader with pick them up or not. I know it does work in grub but not sure about syslinux. Worst case, we create a grub.cnf / menu.lst (grub config) that XenServer would use.

 

Got it,

 

I am thinking in just dropping the XenServer and do a ubuntu 12.04 server install and Xen 4.3. do you think it might make it more compartible.

from what I have read so far, it is possible to use XenCenter to admin that as

ubuntu have XAPI in kernel. am I wrong?

 

 

also what do you think in setting this up to boot from a usb stick :-)

I have a Verbatium TUFF stick  16Gb. with a n XCP 1.6 on it but it never booted properly.

 

I know, I know ,  one step at a time :-)

Share this post


Link to post

Ok, an update here.

I pass the devise to my ubuntu vm. And it crashes with the same error. So a wild guess here but looks like I am out of luck. I will give one more try by building out full ubuntu install with xen. And that it.

after that I am either parting out the whole thing and getting the new setup in the case or just build a VM server for everything and keep the current unraid as is.

 

I am not a Linux guy, so I am not sure if I can undertake configuring a full slackware unRaid build as Dom0 for xen, if it is even possible.

 

I bet we could find a few people over here who would not mined  having a full capable OS setup for unRaid where you can have a real VM infrastructure and not crippled VirtualBox Plugin..

using flash stick as a license dongle only.(Tom :  hint, hint  :-)  )

 

Sent from my SGH-T889 using Tapatalk 4 Beta

 

 

Share this post


Link to post

well I am stuck with this hardware for a while.

the replacement will cost a bundle.

new MB + new CPUs + RAM + SAS cards

 

I was so confident that it will work, I should have tried it first.

and in all other respect, sans virtualization capability for unRaid ,  the setup is very good.

I LOVE IPMI!

 

so I will play with ubuntu+xen setup and see if it works.

if not will try going full Slackware install, I only need to do this once I hope.

after that it will only be unraid recompile. I run unraid on betta for 2+ years.

 

when/if I go this rout I would need to ask you how to include the xen drivers.

 

do you think if I do the Slackware + Xen install first

and than folow the instructions on WiKi from Slackware as Dom0. will it include the drivers as needed?

 

 

 

 

 

Share this post


Link to post

Why not use all the parts you already have and replace only the motherboard?

well

#1. the CPUs are opteron Socket "F" that is the one where the pins are on the MB and CPU is flat, no pins ,so not any kind of MB will do same is for everything else as well...

 

#2. the SAS cards are PCI-X 

 

#3. the RAM is DDR2 not DDR3

 

I have already tried to find MB with IOMMU  that would work with everything  but so far no luck.

 

 

 

Share this post


Link to post

well like I said, I give a ubuntu+xen a try first

and maybe the full slackware install + xen

 

if first one does not work the second one should work for sure as in that case would not need to do a passthrough .

if not  I could just risk it all and just use the ubuntu + xen with NFS shares.

or do the Hyper-V with storage spaces.

Share this post


Link to post

For what it's worth...I'm running a TAMs server just like yours, (2) 6 core procs, 32GB ram and I'm running 2008 r2 data center server with hyper-v and it works perfectly with lots of vms, although I can't get unraid to work with 2008 hyper-v because it doesn't have the correct network driver.  I can get unraid to boot, I can pass hard drives to it directly but it will never see the network card because Microsoft's hyper-v driver isn't seen by unraid.  I haven't tried 2012 hyper-v though but I feel it will be the same way.  You can download it for free and try it if you want.

Share this post


Link to post

I also have the same server from TAM. I am using server 2012 core (hyper-v) I have unraid working as a VM. I am able to pass through drives to my windows 2008 R2 VM with no problems. I have not tried to pass through the hard drives to my unraid vm because I started with trying to pass through my USB device first. So far I have have not been able to find a way to pass my USB stick through, or in hyper-v term take it offline. I have read that there are some usb sticks that can be passed through but no one lists what brand or make they are. This mean you cant get the full version of unRaid.  :-[

 

ddeeds: I would be interested in your unRAID Hyper-V bzroot and bzimage

 

 

Share this post


Link to post

kunkler99 >> are you actually passing thorough drives or you just able to connect the drives to VM?

there is a difference.

the issue here is that if you simply connect(or RDM) the drives to unraid, it will work but you are loosing the whole power saving thing that that you get with unraid.

as in  the drives power down until needed, the only drive that spin-up  is the drive where your data actually resides not the whole array. 

etc.

 

that is why I was trying to do whole SAS controller  Passthrough so unraid have full control of controller functions.

 

also I am planning to add a cablecard tuner later and would like to do the same with it so I can share tuners  on several HTPCs.

not a big deal but nice to have option.

 

 

I think I could connect to the drives from unraid via RDM , I will try it when I get my setup back running.

I was/am trying to configure Xen/ubuntu setup and hitting a wall at network configuration.

as in after installing XEN  the network can not access the internet, so I can not continue with farther setup. I think I know what is the issue (USER ERROR :-) ) but was very busy at work and could not try the new setup.

 

 

 

Share this post


Link to post

As far as I know it does actually pass the drive through. I will test it tonight with unraid and let you know if the drives spin down.

Share this post


Link to post

Well I would love to report good new, but just like you I am unable to get yet another hyper adviser to work with this motherboard. I have tried ESXi (cant pass through the raid cards), I have tried XEN which is why I am on this thread, and now I cant get the network working for unraid in hyper-v. All I want is a virtual unraid without having to spend more money. :(

 

Good news is Unraid works great if that's all you want to do with this server. I just hate knowing I have a dual processor motherboard that is not being used to its full potential

Share this post


Link to post

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.

Guest
Reply to this topic...

×   Pasted as rich text.   Restore formatting

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.